Repository: glotzerlab/signac (python)

Issue:
ipython notebook html display for jobs
**[Original report](https://bitbucket.org/glotzer/signac/issue/38) by Anonymous.**

----------------------------------------

Since ipython/jupyter notebook can display html output, I thought it might be nice to display jobs in a table (the same way pandas dataframes are displayed). So I copied an [online example](http://calebmadrigal.com/display-list-as-table-in-ipython-notebook/):

```
#!python

class JobTable(list):
    """ Overridden list class which takes a list of signac jobs 
        and renders an HTML table in IPython Notebook with the id
        and statepoint attributes. """
    
    def _repr_html_(self):
        """Return html table representing myself, the list of jobs.

        Operations
        ----------
        1. Determine what the attributes are.
        2. Add header row with name of each attribute.
        3. Add row to table for each job, with job id in first column.
        """
        attribs = set(sum((job.statepoint().keys() for job in self), []))
        html = ["<table>"]
        html.append("<td><b>{0}</b></td>".format('id'))
        for attrib in attribs:
            html.append("<td><b>{0}</b></td>".format(attrib))
            
        for job in self:
            html.append("<tr>")
            html.append("<td><b>{0}</b></td>".format(job.get_id()))
            
            for attrib in attribs:
                try:
                    html.append("<td>{0}</td>".format(job.statepoint()[attrib]))
                except KeyError:
                    html.append("<td></td>")
            
            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)
```

This can then be used:

```
#!python
from IPython.display import display
display(JobTable(project.find_jobs({'type':'pmf'})))
```

And an HTML table describing the jobs appears in the results cell (image attached). If this were implemented as part of signac, it would be cleaner to change `find_jobs` from a generator function to instead return an iterator class with the `_repr_html_` method. Do you the pretty notebook display is worth embedding into signac?


Pull request description:
Enable IPython display
## Description
Resolves #38 by adding IPython display capabilities for the `JobsCursor` class.

## Types of Changes
<!-- Please select all items that apply either now or after creating the pull request: -->
- [ ] Documentation update
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change<sup>1</sup>

This pull request is based on
- [ ] *master*, because it is a bug fix or an update to the documentation.
- [x] *develop*, because it introduces a new feature.

<sup>1</sup>The change breaks (or has the potential to break) existing functionality.

## Checklist:
<!-- Please select all items that apply either now or after creating the pull request. -->
<!-- If you are unsure about any of these items, do not hesitate to ask! -->
- [x] I am familiar with the [**Contributing Guidelines**](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md).
- [x] I agree with the terms of the [**Contributor Agreement**](https://github.com/glotzerlab/signac/blob/master/ContributorAgreement.md).
- [x] My name is on the [list of contributors](https://github.com/glotzerlab/signac/blob/master/contributors.txt).
- [x] My code follows the [code style guideline](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md#code-style) of this project.
- [x] The changes introduced by this pull request are covered by existing or newly introduced tests.
- [x] I have updated the [changelog](https://github.com/glotzerlab/signac/blob/master/changelog.txt).


Patch:
diff --git a/changelog.txt b/changelog.txt
index 87f0e3d7..3d706c26 100644
--- a/changelog.txt
+++ b/changelog.txt
@@ -30,6 +30,7 @@ Added
  - Enable `Collection` class to manage collections stored in gzip files.
  - Enable deleting of `JSONDict` keys through the attribute interface, e.g., `del job.doc.foo`.
  - The `Job.init()` method returns the job to allow one-line job creation and initialization.
+ - Add HTML representation of instances of `Project` and `JobsCursor` targeted at Jupyter Notebooks (requires pandas).
 
 Fixed
 +++++
diff --git a/signac/contrib/project.py b/signac/contrib/project.py
index 0485aa36..50460ad2 100644
--- a/signac/contrib/project.py
+++ b/signac/contrib/project.py
@@ -133,6 +133,8 @@ class Project(object):
     FN_CACHE = '.signac_sp_cache.json.gz'
     "The default filename for the state point cache file."
 
+    _use_pandas_for_html_repr = True  # toggle use of pandas for html repr
+
     def __init__(self, config=None):
         if config is None:
             config = load_config()
@@ -167,6 +169,9 @@ class Project(object):
                    type=self.__class__.__module__ + '.' + self.__class__.__name__,
                    id=self.get_id(), rd=self.root_directory(), wd=self.workspace())
 
+    def _repr_html_(self):
+        return repr(self) + self.find_jobs()._repr_html_jobs()
+
     def __eq__(self, other):
         return repr(self) == repr(other)
 
@@ -520,8 +525,10 @@ class Project(object):
     def find_jobs(self, filter=None, doc_filter=None, index=None):
         """Find all jobs in the project's workspace.
 
-        The optional filter arguments must be a Mapping of key-value
-        pairs and JSON serializable.
+        The optional filter arguments must be a Mapping of key-value pairs and
+        JSON serializable. The `filter` argument is used to search against job
+        statepoints, whereas the `doc_filter` argument compares against job
+        document keys.
 
         .. note::
             Providing a pre-calculated index may vastly increase the
@@ -532,6 +539,7 @@ class Project(object):
         :type filter: Mapping
         :param doc_filter: A mapping of key-value pairs that all
             indexed job documents are compared against.
+        :type doc_filter: Mapping
         :yields: Instances of :class:`~signac.contrib.job.Job`
         :raise TypeError: If the filters are not JSON serializable.
         :raises ValueError: If the filters are invalid.
@@ -625,6 +633,10 @@ class Project(object):
     def find_statepoints(self, filter=None, doc_filter=None, index=None, skip_errors=False):
         """Find all statepoints in the project's workspace.
 
+        This function is deprecated.
+
+        .. deprecated:: 0.9.0
+
         :param filter: If not None, only yield statepoints matching the filter.
         :type filter: mapping
         :param skip_errors: Show, but otherwise ignore errors while
@@ -840,12 +852,16 @@ class Project(object):
     def find_job_documents(self, filter=None):
         """Find all job documents in the project's workspace.
 
+        This function is deprecated.
+
         This method iterates through all jobs or all jobs matching
         the filter and yields each job's document as a dict.
         Each dict additionally contains a field 'statepoint',
         with the job's statepoint and a field '_id', which is
         the job's id.
 
+        .. deprecated:: 0.9.0
+
         :param filter: If not None,
             only find job documents matching filter.
         :type filter: mapping
@@ -1626,6 +1642,7 @@ class JobsCursor(object):
     """An iterator over a search query result, enabling simple iteration and
     grouping operations.
     """
+    _use_pandas_for_html_repr = True  # toggle use of pandas for html repr
 
     def __init__(self, project, filter, doc_filter):
         self._project = project
@@ -1827,6 +1844,38 @@ class JobsCursor(object):
             data={job._id: dict(_export_sp_and_doc(job)) for job in self},
             orient='index').infer_objects()
 
+    def __repr__(self):
+        return "{type}({{'project': '{project}', 'filter': '{filter}',"\
+               " 'docfilter': '{doc_filter}'}})".format(
+                   type=self.__class__.__module__ + '.' + self.__class__.__name__,
+                   project=self._project,
+                   filter=self._filter,
+                   doc_filter=self._doc_filter)
+
+    def _repr_html_jobs(self):
+        html = ''
+        len_self = len(self)
+        try:
+            if len_self > 100:
+                raise RuntimeError  # too large
+            if self._use_pandas_for_html_repr:
+                import pandas
+            else:
+                raise RuntimeError
+        except ImportError:
+            warnings.warn('Install pandas for a pretty representation of jobs.')
+            html += '<br/><strong>{}</strong> job(s) found'.format(len_self)
+        except RuntimeError:
+            html += '<br/><strong>{}</strong> job(s) found'.format(len_self)
+        else:
+            with pandas.option_context("display.max_rows", 20):
+                html += self.to_dataframe()._repr_html_()
+        return html
+
+    def _repr_html_(self):
+        """Returns an HTML representation of JobsCursor."""
+        return repr(self) + self._repr_html_jobs()
+
 
 def init_project(name, root=None, workspace=None, make_dir=True):
     """Initialize a project with the given name.


Test patch:
diff --git a/tests/test_project.py b/tests/test_project.py
index 352f0218..2b609aa7 100644
--- a/tests/test_project.py
+++ b/tests/test_project.py
@@ -28,6 +28,13 @@ if six.PY2:
 else:
     from tempfile import TemporaryDirectory
 
+try:
+    import pandas  # noqa
+except ImportError:
+    PANDAS = False
+else:
+    PANDAS = True
+
 
 # Make sure the jobs created for this test are unique.
 test_token = {'test_token': str(uuid.uuid4())}
@@ -1370,6 +1377,78 @@ class ProjectExportImportTest(BaseProjectTest):
             self.assertEqual(len(tmp_project), len(self.project))
 
 
+@unittest.skipIf(six.PY2, 'requires python 3')
+class ProjectRepresentationTest(BaseProjectTest):
+
+    valid_sp_values = [None, 0, 1, 0.0, 1.0, True, False, [0, 1, 2], [0, 1.0, False]]
+
+    num_few_jobs = 10
+    num_many_jobs = 200
+
+    def call_repr_methods(self):
+
+        with self.subTest(of='project'):
+            with self.subTest(type='str'):
+                str(self.project)
+            with self.subTest(type='repr'):
+                repr(self.project)
+            with self.subTest(type='html'):
+                for use_pandas in (True, False):
+                    type(self.project)._use_use_pandas_for_html_repr = use_pandas
+                    with self.subTest(use_pandas=use_pandas):
+                        if use_pandas and not PANDAS:
+                            raise unittest.SkipTest('requires use_pandas')
+                        self.project._repr_html_()
+
+        with self.subTest(of='JobsCursor'):
+            for filter_ in (None, ):
+                with self.subTest(filter=filter_):
+                    with self.subTest(type='str'):
+                        str(self.project.find_jobs(filter_))
+                    with self.subTest(type='repr'):
+                        repr(self.project.find_jobs(filter_))
+                    with self.subTest(type='html'):
+                        for use_pandas in (True, False):
+                            type(self.project)._use_use_pandas_for_html_repr = use_pandas
+                            with self.subTest(use_pandas=use_pandas):
+                                if use_pandas and not PANDAS:
+                                    raise unittest.SkipTest('requires use_pandas')
+                                self.project.find_jobs(filter_)._repr_html_()
+
+    def test_repr_no_jobs(self):
+        self.call_repr_methods()
+
+    def test_repr_few_jobs_homogeneous(self):
+        # Many jobs with many different state points
+        for i in range(self.num_few_jobs):
+            self.project.open_job(
+                {'{}_{}'.format(i, j): v
+                 for j, v in enumerate(self.valid_sp_values)}).init()
+        self.call_repr_methods()
+
+    def test_repr_many_jobs_homogeneous(self):
+        # Many jobs with many different state points
+        for i in range(self.num_many_jobs):
+            self.project.open_job(
+                {'{}_{}'.format(i, j): v
+                 for j, v in enumerate(self.valid_sp_values)}).init()
+        self.call_repr_methods()
+
+    def test_repr_few_jobs_heterogeneous(self):
+        # Many jobs with many different state points
+        for i in range(self.num_few_jobs):
+            for v in self.valid_sp_values:
+                self.project.open_job(dict(a=v)).init()
+        self.call_repr_methods()
+
+    def test_repr_many_jobs_heterogeneous(self):
+        # Many jobs with many different state points
+        for i in range(self.num_many_jobs):
+            for v in self.valid_sp_values:
+                self.project.open_job(dict(a=v)).init()
+        self.call_repr_methods()
+
+
 class LinkedViewProjectTest(BaseProjectTest):
 
     def test_create_linked_view(self):