Package: pl-compare
pl_compare: Compare and find the differences between two Polars DataFrames.Github-PyPi PageYou will find pl-compare useful if you find yourself writing various SQL/Dataframe operations to:Understand how well two tables ReconcileexampleFind the schema differences between two tablesexampleFind counts or examples of rows that exist in one table but not anotherexampleFind counts or examples of value differences between two tablesexampleAssert that two tables are exactly equal (such as for an automated test)exampleAssert that two tables have matching schemas, rows or column valuesexampleWith pl-compare you can:Get statistical summaries and/or examples and/or a boolean to indicate:Schema differencesRow differencesValue differencesEasily works for Pandas dataframes and other tabular data formats with conversion using Apache arrowView differences as a text reportGet differences as a Polars LazyFrame or DataFrameUse LazyFrames for larger than memory comparisonsSpecify the equality calculation that is used to dermine value differencesInstallationpipinstallpl_compareExamples (click to expand)Return booleans to check for schema, row and value differences>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],base_df,compare_df)>>>print("is_schemas_equal:",compare_result.is_schemas_equal())is_schemas_equal:False>>>print("is_rows_equal:",compare_result.is_rows_equal())is_rows_equal:False>>>print("is_values_equal:",compare_result.is_values_equal())is_values_equal:False>>>Schema differences summary and details>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],base_df,compare_df)>>>print("schemas_summary()")schemas_summary()>>>print(compare_result.schemas_summary())shape:(6,2)┌─────────────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞═════════════════════════════════╪═══════╡│Columnsinbase┆3││Columnsincompare┆4││Columnsinbaseandcompare┆3││Columnsonlyinbase┆0││Columnsonlyincompare┆1││Columnswithschemadifferences┆1│└─────────────────────────────────┴───────┘>>>print("schemas_sample()")schemas_sample()>>>print(compare_result.schemas_sample())shape:(2,3)┌──────────┬─────────────┬────────────────┐│column┆base_format┆compare_format││---┆---┆---││str┆str┆str│╞══════════╪═════════════╪════════════════╡│Example2┆Utf8┆Int64││Example3┆null┆Int64│└──────────┴─────────────┴────────────────┘>>>Row differences summary and details>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],base_df,compare_df)>>>print("rows_summary()")rows_summary()>>>print(compare_result.rows_summary())shape:(5,2)┌──────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞══════════════════════════╪═══════╡│Rowsinbase┆3││Rowsincompare┆3││Rowsonlyinbase┆1││Rowsonlyincompare┆1││Rowsinbaseandcompare┆2│└──────────────────────────┴───────┘>>>print("rows_sample()")rows_sample()>>>print(compare_result.rows_sample())shape:(2,3)┌────────────┬──────────┬─────────────────┐│ID┆variable┆value││---┆---┆---││str┆str┆str│╞════════════╪══════════╪═════════════════╡│12345678┆status┆inbaseonly││1234567810┆status┆incompareonly│└────────────┴──────────┴─────────────────┘>>>Value differences summary and details>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],base_df,compare_df)>>>print("values_summary()")values_summary()>>>print(compare_result.values_summary())shape:(2,3)┌─────────────────────────┬───────┬────────────┐│ValueDifferences┆Count┆Percentage││---┆---┆---││str┆i64┆f64│╞═════════════════════════╪═══════╪════════════╡│TotalValueDifferences┆1┆50.0││Example1┆1┆50.0│└─────────────────────────┴───────┴────────────┘>>>print("values_sample()")values_sample()>>>print(compare_result.values_sample())shape:(1,4)┌─────────┬──────────┬──────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆i64┆i64│╞═════════╪══════════╪══════╪═════════╡│1234567┆Example1┆6┆2│└─────────┴──────────┴──────┴─────────┘>>>Full report>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],base_df,compare_df)>>>compare_result.report()--------------------------------------------------------------------------------COMPARISONREPORT--------------------------------------------------------------------------------<BLANKLINE>SCHEMADIFFERENCES:shape:(6,2)┌─────────────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞═════════════════════════════════╪═══════╡│Columnsinbase┆3││Columnsincompare┆4││Columnsinbaseandcompare┆3││Columnsonlyinbase┆0││Columnsonlyincompare┆1││Columnswithschemadifferences┆1│└─────────────────────────────────┴───────┘shape:(2,3)┌──────────┬─────────────┬────────────────┐│column┆base_format┆compare_format││---┆---┆---││str┆str┆str│╞══════════╪═════════════╪════════════════╡│Example2┆Utf8┆Int64││Example3┆null┆Int64│└──────────┴─────────────┴────────────────┘--------------------------------------------------------------------------------<BLANKLINE>ROWDIFFERENCES:shape:(5,2)┌──────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞══════════════════════════╪═══════╡│Rowsinbase┆3││Rowsincompare┆3││Rowsonlyinbase┆1││Rowsonlyincompare┆1││Rowsinbaseandcompare┆2│└──────────────────────────┴───────┘shape:(2,3)┌────────────┬──────────┬─────────────────┐│ID┆variable┆value││---┆---┆---││str┆str┆str│╞════════════╪══════════╪═════════════════╡│12345678┆status┆inbaseonly││1234567810┆status┆incompareonly│└────────────┴──────────┴─────────────────┘--------------------------------------------------------------------------------<BLANKLINE>VALUEDIFFERENCES:shape:(2,3)┌─────────────────────────┬───────┬────────────┐│ValueDifferences┆Count┆Percentage││---┆---┆---││str┆i64┆f64│╞═════════════════════════╪═══════╪════════════╡│TotalValueDifferences┆1┆50.0││Example1┆1┆50.0│└─────────────────────────┴───────┴────────────┘shape:(1,4)┌─────────┬──────────┬──────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆i64┆i64│╞═════════╪══════════╪══════╪═════════╡│1234567┆Example1┆6┆2│└─────────┴──────────┴──────┴─────────┘--------------------------------------------------------------------------------EndofReport-------------------------------------------------------------------------------->>>Compare two pandas dataframes>>>importpolarsaspl>>>importpandasaspd# doctest: +SKIP>>>frompl_compareimportcompare>>>>>>base_df=pd.DataFrame(data=...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)# doctest: +SKIP>>>compare_df=pd.DataFrame(data=...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)# doctest: +SKIP>>>>>>compare_result=compare(["ID"],pl.from_pandas(base_df),pl.from_pandas(compare_df))# doctest: +SKIP>>>compare_result.report()# doctest: +SKIP--------------------------------------------------------------------------------COMPARISONREPORT--------------------------------------------------------------------------------SCHEMADIFFERENCES:shape:(6,2)┌─────────────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞═════════════════════════════════╪═══════╡│Columnsinbase┆3││Columnsincompare┆4││Columnsinbaseandcompare┆3││Columnsonlyinbase┆0││Columnsonlyincompare┆1││Columnswithschemadifferences┆1│└─────────────────────────────────┴───────┘shape:(2,3)┌──────────┬─────────────┬────────────────┐│column┆base_format┆compare_format││---┆---┆---││str┆str┆str│╞══════════╪═════════════╪════════════════╡│Example2┆Utf8┆Int64││Example3┆null┆Int64│└──────────┴─────────────┴────────────────┘--------------------------------------------------------------------------------ROWDIFFERENCES:shape:(5,2)┌──────────────────────────┬───────┐│Statistic┆Count││---┆---││str┆i64│╞══════════════════════════╪═══════╡│Rowsinbase┆3││Rowsincompare┆3││Rowsonlyinbase┆1││Rowsonlyincompare┆1││Rowsinbaseandcompare┆2│└──────────────────────────┴───────┘shape:(2,3)┌────────────┬──────────┬─────────────────┐│ID┆variable┆value││---┆---┆---││str┆str┆str│╞════════════╪══════════╪═════════════════╡│12345678┆status┆inbaseonly││1234567810┆status┆incompareonly│└────────────┴──────────┴─────────────────┘--------------------------------------------------------------------------------VALUEDIFFERENCES:shape:(2,3)┌─────────────────────────┬───────┬────────────┐│ValueDifferences┆Count┆Percentage││---┆---┆---││str┆i64┆f64│╞═════════════════════════╪═══════╪════════════╡│TotalValueDifferences┆1┆50.0││Example1┆1┆50.0│└─────────────────────────┴───────┴────────────┘shape:(1,4)┌─────────┬──────────┬──────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆i64┆i64│╞═════════╪══════════╪══════╪═════════╡│1234567┆Example1┆6┆2│└─────────┴──────────┴──────┴─────────┘--------------------------------------------------------------------------------EndofReport-------------------------------------------------------------------------------->>>Specify a threshold to control the granularity of the comparison for numeric columns.>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1.111,6.11,3.11],...}...)>>>>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1.114,6.14,3.12],...},...)>>>>>>print("With equality_resolution of 0.01")Withequality_resolutionof0.01>>>compare_result=compare(["ID"],base_df,compare_df,resolution=0.01)>>>print(compare_result.values_sample())shape:(1,4)┌─────────┬──────────┬──────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆f64┆f64│╞═════════╪══════════╪══════╪═════════╡│1234567┆Example1┆6.11┆6.14│└─────────┴──────────┴──────┴─────────┘>>>print("With no equality_resolution")Withnoequality_resolution>>>compare_result=compare(["ID"],base_df,compare_df)>>>print(compare_result.values_sample())shape:(2,4)┌─────────┬──────────┬───────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆f64┆f64│╞═════════╪══════════╪═══════╪═════════╡│123456┆Example1┆1.111┆1.114││1234567┆Example1┆6.11┆6.14│└─────────┴──────────┴───────┴─────────┘>>>Example using alias for base and compare dataframes.>>>importpolarsaspl>>>frompl_compareimportcompare>>>>>>base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":["1","2","3"],...}...)>>>compare_df=pl.DataFrame(...{..."ID":["123456","1234567","1234567810"],..."Example1":[1,2,3],..."Example2":[1,2,3],..."Example3":[1,2,3],...},...)>>>>>>compare_result=compare(["ID"],...base_df,...compare_df,...base_alias="before_change",...compare_alias="after_change")>>>>>>print("values_summary()")values_summary()>>>print(compare_result.schemas_sample())shape:(2,3)┌──────────┬──────────────────────┬─────────────────────┐│column┆before_change_format┆after_change_format││---┆---┆---││str┆str┆str│╞══════════╪══════════════════════╪═════════════════════╡│Example2┆Utf8┆Int64││Example3┆null┆Int64│└──────────┴──────────────────────┴─────────────────────┘>>>print("values_sample()")values_sample()>>>print(compare_result.values_sample())shape:(1,4)┌─────────┬──────────┬───────────────┬──────────────┐│ID┆variable┆before_change┆after_change││---┆---┆---┆---││str┆str┆i64┆i64│╞═════════╪══════════╪═══════════════╪══════════════╡│1234567┆Example1┆6┆2│└─────────┴──────────┴───────────────┴──────────────┘>>>Assert two frames are equal for a test>>>importpolarsaspl>>>importpytest>>>frompl_compare.compareimportcompare>>>>>>deftest_example():...base_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,3],..."Example2":[1,2,3],...}...)...compare_df=pl.DataFrame(...{..."ID":["123456","1234567","12345678"],..."Example1":[1,6,9],..."Example2":[1,2,3],...}...)...comparison=compare(["ID"],base_df,compare_df)...ifnotcomparison.is_equal():...raiseException(comparison.report())...>>>test_example()# doctest: +IGNORE_EXCEPTION_DETAILTraceback(mostrecentcalllast):File"<stdin>",line1,in<module>File"<stdin>",line18,intest_exampleException:--------------------------------------------------------------------------------COMPARISONREPORT--------------------------------------------------------------------------------NoSchemadifferencesfound.--------------------------------------------------------------------------------NoRowdifferencesfound(whenjoiningbythesuppliedid_columns).--------------------------------------------------------------------------------VALUEDIFFERENCES:shape:(3,3)┌─────────────────────────┬───────┬────────────┐│ValueDifferences┆Count┆Percentage││---┆---┆---││str┆i64┆f64│╞═════════════════════════╪═══════╪════════════╡│TotalValueDifferences┆1┆16.666667││Example1┆1┆33.333333││Example2┆0┆0.0│└─────────────────────────┴───────┴────────────┘shape:(1,4)┌──────────┬──────────┬──────┬─────────┐│ID┆variable┆base┆compare││---┆---┆---┆---││str┆str┆i64┆i64│╞══════════╪══════════╪══════╪═════════╡│12345678┆Example1┆3┆9│└──────────┴──────────┴──────┴─────────┘--------------------------------------------------------------------------------EndofReport-------------------------------------------------------------------------------->>>To DO:Linting (Ruff)Make into python packageAdd makefile for easy linting and testsStatistics should indicate which statistics are referencing columnsAdd all statistics frame to testsAdd schema differences to schema summaryMake row examples alternate between base only and compare only so that it is more readable.Add limit value to the examples.Updated value differences summary so that Statistic is something that makes sense.Publish package to pypiAdd difference criterion.Add licenseMake package easy to use (i.e. so you only have to import pl_compare and then you can us pl_compare)Add table name labels that can replace 'base' and 'compare'.Update code to use a config dataclass that can be passed between the class and functions.Write up docstringsWrite up readme (with code examples)Add parameter to hide column differences with 0 differences.Add flag to indicate if there are differences between the tables.Update report so that non differences are not displayed.Seperate out dev dependencies from library dependencies?Change 'threshold' to be equality resolution.strict MyPy type checkingRaise error and print examples if duplicates are present.Add total number of value differences to the value differences summary.Add percentage column so the value differences summary.Change id_columns to be named 'join_columns'Github actions for publishing[] Fix report output when tables are exactly equal.[] add a test that checks that abritrary join conditions work.[] Add a count of the number of rows that have any differences to the value differences summary.[] Simplify custom equality checks and add example.[] Test for large amounts of data[] Benchmark for different sizes of data.[] Github actions for testing[] Github actions for linting[] Investigate use for very large datasets 50GB-100GB. Can this be done using LazyFrames only?