Repository: glotzerlab/signac

Issue:
signac diff
### Feature description

Add the functionality to call `signac diff` to compare two or more state points. This could also be extended to comparing files, job docs, etc. 

### Proposed solution

partially solved at #247 

### Additional context

Any input from @bdice @csadorf @vyasr @b-butler @mikemhenry is very welcome!


Discussion:
vyasr: Discussion on the appropriate scope and API of this PR is happening at #248. Once everything there is finalized we can update this PR.
codecov[bot]: # [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=h1) Report
> Merging [#247](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=desc) into [master](https://codecov.io/gh/glotzerlab/signac/commit/a5ca930d393b838a2e340034eddd57f053e71730?src=pr&el=desc) will **decrease** coverage by `0.34%`.
> The diff coverage is `40%`.

[![Impacted file tree graph](https://codecov.io/gh/glotzerlab/signac/pull/247/graphs/tree.svg?width=650&token=FghbjTyzgM&height=150&src=pr)](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #247      +/-   ##
==========================================
- Coverage   65.08%   64.73%   -0.35%     
==========================================
  Files          39       40       +1     
  Lines        5588     5615      +27     
==========================================
- Hits         3637     3635       -2     
- Misses       1951     1980      +29
```


| [Impacted Files](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [signac/\_\_main\_\_.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL19fbWFpbl9fLnB5) | `0% <0%> (ø)` | :arrow_up: |
| [signac/\_\_init\_\_.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL19faW5pdF9fLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [signac/diff.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2RpZmYucHk=) | `100% <100%> (ø)` | |
| [signac/common/config.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2NvbW1vbi9jb25maWcucHk=) | `65.74% <0%> (-4.63%)` | :arrow_down: |
| [signac/contrib/collection.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2NvbnRyaWIvY29sbGVjdGlvbi5weQ==) | `88.59% <0%> (-0.57%)` | :arrow_down: |
| [signac/contrib/import\_export.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2NvbnRyaWIvaW1wb3J0X2V4cG9ydC5weQ==) | `84.47% <0%> (-0.27%)` | :arrow_down: |
| [signac/contrib/job.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2NvbnRyaWIvam9iLnB5) | `90.47% <0%> (-0.17%)` | :arrow_down: |
| [signac/contrib/project.py](https://codecov.io/gh/glotzerlab/signac/pull/247/diff?src=pr&el=tree#diff-c2lnbmFjL2NvbnRyaWIvcHJvamVjdC5weQ==) | `90.96% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=footer). Last update [a5ca930...719498c](https://codecov.io/gh/glotzerlab/signac/pull/247?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

bdice: Reviewers @vyasr @cbkerr and author @atravitz: I made some final touch-ups to this PR, and I am satisfied with the feature's quality and completeness for merging to `master`. I will be making a release tomorrow afternoon, probably around 5pm. If you have any review comments, it would be great to hear those and incorporate any needed changes before then.
vyasr: Regarding the different use-cases outlined in #248, I think the existing API is sufficiently general to allow this. We would simply need to add an extra argument to the Python and CLI such as `diff_by` that would default to statepoint (which I think is the sensible default for diffing jobs). The output would be of the same form, except that if we chose to diff files the leaf nodes of the dict would probably have to be a list of files instead of a mapping of key-value pairs. Document and data diffing should be transparently supported using the same API and output structure.
atravitz: suggestions from @vyasr have been addressed

Patch:
diff --git a/changelog.txt b/changelog.txt
index 1e1c9d02..0af6dab0 100644
--- a/changelog.txt
+++ b/changelog.txt
@@ -25,6 +25,7 @@ Added
 
  - Official support for Python 3.8 (#258).
  - Add properties ``Project.id`` and ``Job.id`` (#250).
+ - Add ``signac.diff_jobs`` function to compare two or more state points (#248, #247).
  - Add function to initialize a sample data space for testing purposes (#215).
  - Add schema version to ensure compatibility and enable migrations in future package versions (#165, #253).
 
diff --git a/contributors.yaml b/contributors.yaml
index 2371267f..1a12652f 100644
--- a/contributors.yaml
+++ b/contributors.yaml
@@ -67,6 +67,11 @@ contributors:
     affiliation: "Boise State University"
     orcid: "https://orcid.org/0000-0001-9665-5420"
   -
+    family-names: Travitz
+    given-names: Alyssa
+    affiliation: "University of Michigan"
+    orcid: "https://orcid.org/0000-0001-5953-8807"
+
     family-names: Butler
     given-names: Brandon
     affiliation: "University of Michigan"
diff --git a/doc/cli-help.txt b/doc/cli-help.txt
index 2d476e51..fb2f44f0 100644
--- a/doc/cli-help.txt
+++ b/doc/cli-help.txt
@@ -1,12 +1,12 @@
 usage: signac [-h] [--debug] [--version] [-v] [-y]
-              {init,project,job,statepoint,document,rm,move,clone,index,find,view,schema,shell,sync,import,export,update-cache,config}
+              {init,project,job,statepoint,diff,document,rm,move,clone,index,find,view,schema,shell,sync,import,export,update-cache,config}
               ...
 
 signac aids in the management, access and analysis of large-scale
 computational investigations.
 
 positional arguments:
-  {init,project,job,statepoint,document,rm,move,clone,index,find,view,schema,shell,sync,import,export,update-cache,config}
+  {init,project,job,statepoint,diff,document,rm,move,clone,index,find,view,schema,shell,sync,import,export,update-cache,config}
 
 optional arguments:
   -h, --help            show this help message and exit
diff --git a/signac/__init__.py b/signac/__init__.py
index c76beb52..6dc17007 100644
--- a/signac/__init__.py
+++ b/signac/__init__.py
@@ -35,6 +35,7 @@ from .contrib import index
 from .contrib import RegexFileCrawler
 from .contrib import MasterCrawler
 from .contrib import SignacProjectCrawler
+from .diff import diff_jobs
 from .db import get_database
 from .core.jsondict import buffer_reads_writes as buffered
 from .core.jsondict import in_buffered_mode as is_buffered
@@ -50,6 +51,7 @@ from .version import __version__
 __all__ = ['__version__', 'contrib', 'db', 'errors', 'warnings', 'sync',
            'cite',
            'Project', 'TemporaryProject', 'get_project', 'init_project', 'get_job',
+           'diff_jobs',
            'get_database', 'fetch',
            'export_one', 'export', 'export_to_mirror',
            'Collection',
diff --git a/signac/__main__.py b/signac/__main__.py
index 0ca786b4..20aed70a 100644
--- a/signac/__main__.py
+++ b/signac/__main__.py
@@ -42,6 +42,7 @@ from .errors import SyncConflict
 from .errors import FileSyncConflict
 from .errors import DocumentSyncConflict
 from .errors import SchemaSyncConflict
+from .diff import diff_jobs
 
 try:
     from .common.host import get_client, get_database, get_credentials, make_uri
@@ -384,6 +385,19 @@ def main_find(args):
             raise
 
 
+def main_diff(args):
+    project = get_project()
+
+    jobs = find_with_filter_or_none(args)
+    jobs = (_open_job_by_id(project, job) for job in jobs) if jobs is not None else project
+
+    diff = diff_jobs(*jobs)
+
+    for jobid, sp in diff.items():
+        print(jobid)
+        pprint(sp)
+
+
 def main_view(args):
     project = get_project()
     project.create_linked_view(
@@ -1123,8 +1137,7 @@ def main():
         'job_id',
         nargs='*',
         type=str,
-        help="One or more job ids. The job corresponding to a job "
-             "id must be initialized.")
+        help="One or more job ids. The corresponding jobs must be initialized.")
     parser_statepoint.add_argument(
         '-p', '--pretty',
         type=int,
@@ -1145,6 +1158,40 @@ def main():
         help="Sort the state point keys for output.")
     parser_statepoint.set_defaults(func=main_statepoint)
 
+    parser_diff = subparsers.add_parser(
+        'diff',
+        description="Find the difference among job state points.")
+    parser_diff.add_argument(
+        'job_id',
+        nargs='*',
+        type=str,
+        help="One or more job ids. The corresponding jobs must be initialized.")
+    parser_diff.add_argument(
+        '-p', '--pretty',
+        type=int,
+        nargs='?',
+        const=3,
+        help="Print state point in pretty format. "
+             "An optional argument to this flag specifies the maximal "
+             "depth a state point is printed.")
+    parser_diff.add_argument(
+        '-i', '--indent',
+        type=int,
+        nargs='?',
+        const='2',
+        help="Specify the indentation of the JSON formatted state point.")
+    parser_diff.add_argument(
+        '-f', '--filter',
+        type=str,
+        nargs='+',
+        help="Limit the diff to jobs matching this state point filter.")
+    parser_diff.add_argument(
+        '-d', '--doc-filter',
+        type=str,
+        nargs='+',
+        help="Show documents of jobs matching this document filter.")
+    parser_diff.set_defaults(func=main_diff)
+
     parser_document = subparsers.add_parser(
         'document',
         description="Print the document(s) corresponding to one or "
@@ -1219,8 +1266,7 @@ def main():
         'job_id',
         nargs='+',
         type=str,
-        help="One or more job ids of jobs to move. The job corresponding to a "
-             "job id must be initialized.")
+        help="One or more job ids. The corresponding jobs must be initialized.")
     parser_move.set_defaults(func=main_move)
 
     parser_clone = subparsers.add_parser('clone')
@@ -1232,8 +1278,7 @@ def main():
         'job_id',
         nargs='+',
         type=str,
-        help="One or more job ids of jobs to clone. The job corresponding to a "
-             "job id must be initialized.")
+        help="One or more job ids. The corresponding jobs must be initialized.")
     parser_clone.set_defaults(func=main_clone)
 
     parser_index = subparsers.add_parser('index')
diff --git a/signac/diff.py b/signac/diff.py
new file mode 100644
index 00000000..4b608f1d
--- /dev/null
+++ b/signac/diff.py
@@ -0,0 +1,84 @@
+# Copyright (c) 2019 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+from .contrib.collection import _traverse_filter
+
+
+def _dotted_keys_to_nested_dicts(mapping):
+    """Converts dictionaries with dot-separated keys into nested dictionaries.
+
+    :param mapping: A mapping with dots in its keys, e.g. {'a.b': 'c'}
+    :returns: A mapping with nested keys, e.g. {'a': {'b': 'c'}}
+    :rtype: dict
+    """
+    result = {}
+
+    def make_nested_dict(d, keys):
+        item = d
+        for key in keys[:-1]:
+            if key not in item:
+                item[key] = {}
+            item = item[key]
+        return item
+
+    for dotted_key, value in mapping.items():
+        keys = dotted_key.split('.')
+        make_nested_dict(result, keys)[keys[-1]] = value
+
+    return result
+
+
+def diff_jobs(*jobs):
+    r"""Find differences among a list of jobs' state points.
+
+    The resulting diff is a dictionary where the keys are job ids and the
+    values are each job's state point minus the intersection of all provided
+    jobs' state points. The comparison is performed over the combined set
+    of keys and values.
+
+    Example:
+
+    .. code-block:: python
+
+        >>> import signac
+        >>> project = signac.init_project('project_name')
+        >>> job1 = project.open_job({'constant': 42, 'diff1': 0, 'diff2': 1}).init()
+        >>> job2 = project.open_job({'constant': 42, 'diff1': 1, 'diff2': 1}).init()
+        >>> job3 = project.open_job({'constant': 42, 'diff1': 2, 'diff2': 2}).init()
+        >>> print(job1)
+        c4af2b26f1fd256d70799ad3ce3bdad0
+        >>> print(job2)
+        b96b21fada698f8934d58359c72755c0
+        >>> print(job3)
+        e4289419d2b0e57e4852d44a09f167c0
+        >>> signac.diff_jobs(job1, job2, job3)
+        {'c4af2b26f1fd256d70799ad3ce3bdad0': {'diff2': 1, 'diff1': 0},
+        'b96b21fada698f8934d58359c72755c0': {'diff2': 1, 'diff1': 1},
+        'e4289419d2b0e57e4852d44a09f167c0': {'diff2': 2, 'diff1': 2}}
+        >>> signac.diff_jobs(*project)
+        {'c4af2b26f1fd256d70799ad3ce3bdad0': {'diff2': 1, 'diff1': 0},
+        'b96b21fada698f8934d58359c72755c0': {'diff2': 1, 'diff1': 1},
+        'e4289419d2b0e57e4852d44a09f167c0': {'diff2': 2, 'diff1': 2}}
+
+    :param \*jobs: The jobs whose state points will be diffed.
+    :type \*jobs: :py:class:`~.Job`
+    :returns: A dictionary where the keys are job ids and values are the unique
+        parts of that job's state point.
+    :rtype: dict
+    """
+    if len(jobs) == 0:
+        return {}
+
+    else:
+        sps = {}
+        for job in jobs:
+            sps[job] = set(_traverse_filter(job.sp()))
+
+        intersection = set.intersection(*sps.values())
+
+        diffs = {}
+        for job in jobs:
+            unique_sps = sps[job]-intersection
+            diffs[job.id] = _dotted_keys_to_nested_dicts(dict(unique_sps))
+
+        return diffs


Test patch:
diff --git a/tests/test_diff.py b/tests/test_diff.py
new file mode 100644
index 00000000..c5688a3e
--- /dev/null
+++ b/tests/test_diff.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2019 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import unittest
+import signac
+from tempfile import TemporaryDirectory
+
+
+class BaseDiffTest(unittest.TestCase):
+
+    project_class = signac.Project
+
+    def setUp(self):
+        self._tmp_dir = TemporaryDirectory(prefix='signac_')
+        self.addCleanup(self._tmp_dir.cleanup)
+        self.project = self.project_class.init_project(
+            name='diff_test_project',
+            root=self._tmp_dir.name)
+
+
+class DiffTest(BaseDiffTest):
+
+    def test_two_jobs(self):
+        job1 = self.project.open_job({'a': 0, 'b': 1})
+        job2 = self.project.open_job({'a': 0})
+        expected = {str(job1.id): {'b': 1}, str(job2.id): {}}
+        result = signac.diff_jobs(job1, job2)
+        self.assertEqual(expected, result, '{} is not {}'.format(result, expected))
+
+    def test_one_job(self):
+        job1 = self.project.open_job({'a': 0})
+        expected = {str(job1.id): {}}
+        result = signac.diff_jobs(job1)
+        self.assertEqual(expected, result, '{} is not {}'.format(result, expected))
+
+    def test_no_jobs(self):
+        self.assertTrue(signac.diff_jobs() == {})
+
+    def test_nested(self):
+        job1 = self.project.open_job({'a': 0, 'b': {'c': True, 'd': 11}})
+        job2 = self.project.open_job({'a': 0, 'b': {'c': True, 'd': 4}})
+        expected = {str(job1.id): {'b': {'d': 11}}, str(job2.id): {'b': {'d': 4}}}
+        result = signac.diff_jobs(job1, job2)
+        self.assertEqual(expected, result, '{} is not {}'.format(result, expected))
+
+    def test_same_job(self):
+        job1 = self.project.open_job({'a': 0, 'b': 1})
+        expected = {str(job1.id): {}}
+        result = signac.diff_jobs(job1, job1)
+        self.assertEqual(expected, result, '{} is not {}'.format(result, expected))
+
+
+if __name__ == '__main__':
+    unittest.main()