Package: datadelta
The best Python package for comparing two dataframesExplore the docs »Table of ContentsAbout The ProjectGetting StartedDependenciesInstallationUsage ExamplesContributingExample HTML Report OutputLicenseContactAbout The ProjectDataDelta is a very useful Python package for easily comparing two pandas dataframes for use in data analysis, data engineering, and tracking table changes across time.DataDelta generates areportas both a Python dict and HTML file that summarizes the key changes between two dataframes through completing a series of tests (that can also be selected individually). The Python report is intended for use as part of a DevOps / DataOps pipeline for testing to ensure table changes are expected.Report Bug·Request Feature(back to top)Getting StartedDataDelta is easy to install through pip or feel free to clone locally to make changes.DependenciesDataDelta has very few dependencies:pandas:a fast, powerful, flexible and easy to use open source data analysis and manipulation tool- DataDelta is built on for comparing dataframesnumpy:The fundamental package for scientific computing with Python- used for transformations and calculationsjinja2:a fast, expressive, extensible templating engine- used to generate the HTML reportpytest(optional):a mature full-featured Python testing tool that helps you write better programs- used for testingInstallationInstall using Pip through PyPI:pipinstalldatadeltaORClone the repo locally:gitclonehttps://github.com/gibbsbravo/DataDelta.git(back to top)Usage ExamplesQuick starter code to get summary dataframe changes report:importpandasaspd
importdatadeltaasdeltaold_df=pd.read_csv('MainTestData_old_df.csv')# Add your old dataframe herenew_df=pd.read_csv('MainTestData_new_df.csv')# Add your new dataframe hereprimary_key='A'# Set the primary keycolumn_subset=None# Specify the subset of columns of interest or leave None to compare all columns# The consolidated_report dictionary will contain the summary changesconsolidated_report,record_changes_comparison_df=delta.create_consolidated_report(old_df,new_df,primary_key,column_subset)# This will create a report named datadelta_html_report.html in the current working directory containing the summary changesdelta.export_html_report(consolidated_report,record_changes_comparison_df,export_file_name='datadelta_html_report.html',overwrite_existing_file=False)Get dataframe summary:importpandasaspdimportdatadeltaasdeltanew_df=pd.read_csv('MainTestData_new_df.csv')# Add your new dataframe here# Returns a report summarizing the key attributes and values of a dataframesummary_report=delta.get_df_summary(input_df=new_df,primary_key=primary_key,column_subset=column_subset,max_cols=15)Get record count changes report:old_df=pd.read_csv('MainTestData_old_df.csv')# Add your old dataframe herenew_df=pd.read_csv('MainTestData_new_df.csv')# Add your new dataframe hereprimary_key='A'# Set the primary keycolumn_subset=None# Specify the subset of columns of interest or leave None to compare all columns# Returns a report summarizing any changes to the number of records (and composition) between two dataframesrecord_count_change_report=delta.check_record_count(old_df,new_df,primary_key)Other functions include:check_column_names: Returns a report summarizing any changes to column names between two dataframescheck_datatypes: Returns a report summarizing any columns with different datatypescheck_chg_in_values: Returns a report summarizing any records with changes in valuesget_records_in_both_tables: Returns the records found in both dataframesget_record_changes_comparison_df: Returns a dataframe comparing any records with changes in values by columnexport_html_report: Exports an html report of the differences between two dataframes(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)Example HTML Report Output(back to top)LicenseDistributed under the GNU General Public License v3 (GPLV3) License. SeeLICENSE.txtfor more information.(back to top)ContactAndrew Gibbs-Bravo -andrewgbravo@gmail.comProject Link:https://github.com/gibbsbravo/DataDelta(back to top)