Package: dir-content-diff
Directory Content DifferenceThis project provides simple tools to compare the content of a directory against a reference
directory.This is useful to check the results of a process that generate several files, like a luigi
workflow for example.InstallationThis package should be installed using pip:pipinstalldir-content-diffUsageThedir-content-diffpackage introduces a framework to compare two directories. A comparator
is associated to each file extension and then each file in the reference directory is compared to
the file with the same relative path in the compared directory. By default, a few comparators are
provided for usual files but others can be associated to new file extensions or can even replace
the default ones. The comparators should be able to report the differences between two files
accurately, reporting which elements are different among the data. When an extension has no
comparator associated, a default comparator is used which just compares the whole binary data of
the files, so it is not able to report which values are different.Compare two directoriesIf one wants to compare two directories with the following structures:└──reference_dir├──sub_dir_1|├──sub_file_1.a|└──sub_file_2.b└──file_1.c└──compared_dir├──sub_dir_1|├──sub_file_1.a|└──sub_file_2.b|└──sub_file_3.b└──file_1.cThese two directories can be compared with the following code:importdir_content_diffdir_content_diff.compare_trees("reference_dir","compared_dir")This code will return an empty dictionary because no difference was detected.Ifreference_dir/file_1.cis the following JSON-like file:{"a":1,"b":[1,2]}Andcompared_dir/file_1.cis the following JSON-like file:{"a":2,"b":[10,2,0]}The following code registers theJsonComparatorfor the file extension.cand compares the
two directories:importdir_content_diffdir_content_diff.register_comparator(".c",dir_content_diff.JsonComparator())dir_content_diff.compare_trees("reference_dir","compared_dir")The previous code will output the following dictionary:{'file_1.c':('The files\'reference_dir/file_1.c\'and\'compared_dir/file_1.c\'are different:\n''Added the value(s)\'{"2": 0}\'in the\'[b]\'key.\n''Changed the value of\'[a]\'from 1 to 2.\n''Changed the value of\'[b][0]\'from 1 to 10.')}It is also possible to check whether the two directories are equal or not with the following code:importdir_content_diffdir_content_diff.register_comparator(".c",dir_content_diff.JsonComparator())dir_content_diff.assert_equal_trees("reference_dir","compared_dir")Which will output the followingAssertionError:AssertionError:Thefiles'reference_dir/file_1.c'and'compared_dir/file_1.c'aredifferent:
Addedthevalue(s)'{"2": 0}'inthe'[b]'key.
Changedthevalueof'[a]'from1to2.
Changedthevalueof'[b][0]'from1to10.Finally, the comparators have parameters that can be passed either to be used for all files of a
given extension or only for a specific file:importdir_content_diff# Get the default comparatorscomparators=dir_content_diff.get_comparators()# Replace the comparators for JSON files to perform the comparison with a given tolerancecomparators[".json"]=dir_content_diff.JsonComparator(default_diff_kwargs={"tolerance":0.1})# Use a specific tolerance for the file ``sub_dir_1/sub_file_1.a``# In this case, the kwargs are used to compute the difference by default, except the following# specific kwargs: ``return_raw_diffs``, ``load_kwargs``, ``format_data_kwargs``, ``filter_kwargs``,# ``format_diff_kwargs``, ``sort_kwargs``, ``concat_kwargs`` and ``report_kwargs``.specific_args={"sub_dir_1/sub_file_1.a":{"tolerance":0.5}}dir_content_diff.assert_equal_trees("reference_dir","compared_dir",comparators=comparators,specific_args=specific_args,)Each comparator has different arguments that are detailed in the documentation.It's also possible to specify a arbitrary comparator for a specific file:specific_args={"sub_dir_1/sub_file_1.a":{"comparator":dir_content_diff.JsonComparator(),"tolerance":0.5,}}And last but not least, it's possible to use regular expressions to associate specific arguments to
a set of files:specific_args={"all files with *.a of *.b extensions":{"patterns":[r".*\.[a,b]$"],"comparator":dir_content_diff.BaseComparator(),}}Export formatted dataSome comparators have to format the data before comparing them. For example, if one wants to
compare data with file paths inside, it's likely that only a relative part of these paths are
relevant, not the entire absolute paths. To do this, a specific comparator can be defined with a
customformat_data()method which is automatically called after the data are loaded but before
the data are compared. It is then possible to export the data just after they have been formatted
for check purpose for example. To do this, theexport_formatted_filesargument of thedir_content_diff.compare_treesanddir_content_diff.assert_equal_treesfunctions can be set
toTrue. Thus all the files processed by a comparator with asave()method will be exported
to a new directory. This new directory is the same as the compared directory to which a suffix is
added. By default, the suffix is_FORMATTED, but it can be overridden by passing a non-empty
string to theexport_formatted_filesargument.Pytest pluginThis package can be used as a pytest plugin. Whenpytestis run anddir-content-diffis
installed, it is automatically detected and registered as a plugin. It is then possible to trigger
the export of formatted data with the followingpytestoption:--dcd-export-formatted-data.
It is also possible to define a custom suffix for the new directory with the following option:--dcd-export-suffix.Funding & AcknowledgmentThe development of this software was supported by funding to the Blue Brain Project, a research
center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH
Board of the Swiss Federal Institutes of Technology.For license and authors, seeLICENSE.txtandAUTHORS.mdrespectively.Copyright © 2021-2023 Blue Brain Project/EPFL