Repository: mne-tools/mne-python

Issue:
Feature request: incremental updates to an `mne.Report`
This is something that came up when writing the huge analysis pipeline for the [conpy](https://github.com/wmvanvliet/conpy) project:

**I think the `Report` object of MNE-Python could be made even more useful if we could write incremental updates to it.**

As the analysis progresses, more and more figures are incrementally added to the report. Every time the report is saved, the `.html` file is updated. This would make it super easy to add visual checks to each step of the analysis. Easier than the current situation where we have to make one giant `make_report.py` file at the end of the pipeline that reads in all the intermediate results and plots them. 

Say I have three analysis scripts:

    01_filter_raw.py
    02_make_epochs.py
    03_grand_average.py

It would be cool if each of these scripts could add/update figures in a shared `Report` object, for example like this:

```python
# Somewhere in 01_filter_raw.py
report = mne.read_report('subject1_report.pickle')  # Loads existing or creates new
report.add_figs_to_section([raw.plot_psd()], ['PSD plot'], 'preprocessing')
report.pickle()  # since report.save() is already taken

# Somewhere in 02_make_epochs.py
report = mne.read_report('subject1_report.pickle')
report.add_figs_to_section([epochs.average.plot()], ['Evokeds'], 'preprocessing')
report.pickle()

# Somewhere in 03_grand_average.py
report = mne.read_report('subject1_report.pickle')
report.add_figs_to_section([grand_average.plot()], ['Grand average'], 'preprocessing')
report.pickle()
```

Even implement `__enter__()` and `__exit__()` blocks so we can do this:

```python
with mne.read_report('subject1_report.pickle') as report:
     report.add_figs_to_section(...)
# report is automatically saved when exiting the block
```

For this to be used as intended, there needs to be a way to overwrite existing figures.


Discussion:
can we already pickle a report object? if so it should quite easy to
implement

yes we can! I can quickly create a PR with the bulk of the desired functionality if there seems to be interest in this feature.
I don't think it's much code and I can understand the practical value. So
let's try I would say

From what I understand pickling is a bad way to get long-term persistence. I'd rather use `hdf5`, in theory it shouldn't add too many lines beyond what the `pickle` does.
true but I imagine the usecase suggests you don't need long term
persistence. Whatever works but KISS

Thinking even smaller scale, `pickle` seems like an unreliable way to transfer data across Python instances. Say for some reason (mayavi!) a user ran one script in Python 3 env to do preproc, then later a different one in Python 2 to viz source estimates, a `pickle` pipeline very likely will fail. These sorts of corner cases should be pretty easily avoided via `h5io` storage. It's more standard for us anyway so it's also more consistent.

The real question is what to call this. Maybe `Report.save(fname, output='html' | 'instance')`?
`Report.to_hdf5`?
`.to_*` makes it sound like it returns an object. We usually use `write_` or `save` for things that write to disk, and for methods it's always `save` of some sort -- so perhaps `save_hdf5`?
Since the saved report is an html, wouldn't it be possible to just read it and grep for the `<div>` elements, populate the `self.html` list that's internal to the `Report` object? It's probably more work than the `save_hdf5` option though ...
maybe use save and decide what to use based on file extension?

> maybe use save and decide what to use based on file extension?

that's a good idea!

Resolving patch:
diff --git a/mne/__init__.py b/mne/__init__.py
--- a/mne/__init__.py
+++ b/mne/__init__.py
@@ -87,7 +87,7 @@
 from .selection import read_selection
 from .dipole import read_dipole, Dipole, DipoleFixed, fit_dipole
 from .channels import equalize_channels, rename_channels, find_layout
-from .report import Report
+from .report import Report, open_report
 
 from . import beamformer
 from . import channels
diff --git a/mne/report.py b/mne/report.py
--- a/mne/report.py
+++ b/mne/report.py
@@ -15,7 +15,6 @@
 import time
 from glob import glob
 import warnings
-
 import numpy as np
 
 from . import read_evokeds, read_events, pick_types, read_cov
@@ -32,6 +31,7 @@
 from .externals.tempita import HTMLTemplate, Template
 from .externals.six import BytesIO
 from .externals.six import moves
+from .externals.h5io import read_hdf5, write_hdf5
 
 VALID_EXTENSIONS = ['raw.fif', 'raw.fif.gz', 'sss.fif', 'sss.fif.gz',
                     '-eve.fif', '-eve.fif.gz', '-cov.fif', '-cov.fif.gz',
@@ -316,6 +316,49 @@ def _update_html(html, report_fname, report_sectionlabel):
     return htmls, report_fnames, report_sectionlabels
 
 
+def open_report(fname, **params):
+    """Read a saved report or, if it doesn't exist yet, create a new one.
+
+    The returned report can be used as a context manager, in which case any
+    changes to the report are saved when exiting the context block.
+
+    Parameters
+    ----------
+    fname : str
+        The file containing the report, stored in the HDF5 format. If the file
+        does not exist yet, a new report is created that will be saved to the
+        specified file.
+    **params : list of parameters
+        When creating a new report, any named parameters other than ``fname``
+        are passed to the `__init__` function of the `Report` object. When
+        reading an existing report, the parameters are checked with the
+        loaded report and an exception is raised when they don't match.
+
+    Returns
+    -------
+    report : instance of Report
+        The report.
+    """
+    if op.exists(fname):
+        # Check **params with the loaded report
+        state = read_hdf5(fname, title='mnepython')
+        for param in params.keys():
+            if param not in state:
+                raise ValueError('The loaded report has no attribute %s' %
+                                 param)
+            if params[param] != state[param]:
+                raise ValueError("Attribute '%s' of loaded report does not "
+                                 "match the given parameter." % param)
+        report = Report()
+        report.__setstate__(state)
+    else:
+        report = Report(**params)
+    # Keep track of the filename in case the Report object is used as a context
+    # manager.
+    report._fname = fname
+    return report
+
+
 ###############################################################################
 # IMAGE FUNCTIONS
 
@@ -1354,17 +1397,67 @@ class construction.
                 warn('`subjects_dir` and `subject` not provided. Cannot '
                      'render MRI and -trans.fif(.gz) files.')
 
+    def _get_state_params(self):
+        """Obtain all fields that are in the state dictionary of this object.
+
+        Returns
+        -------
+        non_opt_params : list of str
+            All parameters that must be present in the state dictionary.
+        opt_params : list of str
+            All parameters that are optionally present in the state dictionary.
+        """
+        # Note: self._fname is not part of the state
+        return (['baseline', 'cov_fname', 'fnames', 'html', 'include',
+                 'image_format', 'info_fname', 'initial_id', 'raw_psd',
+                 '_sectionlabels', 'sections', '_sectionvars',
+                 '_sort_sections', 'subjects_dir', 'subject', 'title',
+                 'verbose'],
+                ['data_path', '_sort'])
+
+    def __getstate__(self):
+        """Get the state of the report as a dictionary."""
+        state = dict()
+        non_opt_params, opt_params = self._get_state_params()
+        for param in non_opt_params:
+            state[param] = getattr(self, param)
+        for param in opt_params:
+            if hasattr(self, param):
+                state[param] = getattr(self, param)
+        return state
+
+    def __setstate__(self, state):
+        """Set the state of the report."""
+        non_opt_params, opt_params = self._get_state_params()
+        for param in non_opt_params:
+            setattr(self, param, state[param])
+        for param in opt_params:
+            if param in state:
+                setattr(self, param, state[param])
+        return state
+
     def save(self, fname=None, open_browser=True, overwrite=False):
-        """Save html report and open it in browser.
+        """Save the report and optionally open it in browser.
 
         Parameters
         ----------
-        fname : str
-            File name of the report.
+        fname : str | None
+            File name of the report. If the file name ends in '.h5' or '.hdf5',
+            the report is saved in HDF5 format, so it can later be loaded again
+            with :func:`open_report`. If the file name ends in anything else,
+            the report is rendered to HTML. If ``None``, the report is saved to
+            'report.html' in the current working directory.
+            Defaults to ``None``.
         open_browser : bool
-            Open html browser after saving if True.
+            When saving to HTML, open the rendered HTML file browser after
+            saving if True. Defaults to True.
         overwrite : bool
-            If True, overwrite report if it already exists.
+            If True, overwrite report if it already exists. Defaults to False.
+
+        Returns
+        -------
+        fname : str
+            The file name to which the report was saved.
         """
         if fname is None:
             if not hasattr(self, 'data_path'):
@@ -1375,14 +1468,6 @@ def save(self, fname=None, open_browser=True, overwrite=False):
         else:
             fname = op.realpath(fname)
 
-        self._render_toc()
-
-        with warnings.catch_warnings(record=True):
-            warnings.simplefilter('ignore')
-            html = footer_template.substitute(date=time.strftime("%B %d, %Y"),
-                                              current_year=time.strftime("%Y"))
-        self.html.append(html)
-
         if not overwrite and op.isfile(fname):
             msg = ('Report already exists at location %s. '
                    'Overwrite it (y/[n])? '
@@ -1391,23 +1476,55 @@ def save(self, fname=None, open_browser=True, overwrite=False):
             if answer.lower() == 'y':
                 overwrite = True
 
+        _, ext = op.splitext(fname)
+        is_hdf5 = ext.lower() in ['.h5', '.hdf5']
+
         if overwrite or not op.isfile(fname):
             logger.info('Saving report to location %s' % fname)
-            fobj = codecs.open(fname, 'w', 'utf-8')
-            fobj.write(_fix_global_ids(u''.join(self.html)))
-            fobj.close()
-
-            # remove header, TOC and footer to allow more saves
-            self.html.pop(0)
-            self.html.pop(0)
-            self.html.pop()
 
-        if open_browser:
+            if is_hdf5:
+                write_hdf5(fname, self.__getstate__(), overwrite=overwrite,
+                           title='mnepython')
+            else:
+                self._render_toc()
+
+                # Annotate the HTML with a TOC and footer.
+                with warnings.catch_warnings(record=True):
+                    warnings.simplefilter('ignore')
+                    html = footer_template.substitute(
+                        date=time.strftime("%B %d, %Y"),
+                        current_year=time.strftime("%Y"))
+                self.html.append(html)
+
+                # Writing to disk may fail. However, we need to make sure that
+                # the TOC and footer are removed regardless, otherwise they
+                # will be duplicated when the user attempts to save again.
+                try:
+                    # Write HTML
+                    with codecs.open(fname, 'w', 'utf-8') as fobj:
+                        fobj.write(_fix_global_ids(u''.join(self.html)))
+                finally:
+                    self.html.pop(0)
+                    self.html.pop(0)
+                    self.html.pop()
+
+        if open_browser and not is_hdf5:
             import webbrowser
             webbrowser.open_new_tab('file://' + fname)
 
+        self.fname = fname
         return fname
 
+    def __enter__(self):
+        """Do nothing when entering the context block."""
+        return self
+
+    def __exit__(self, type, value, traceback):
+        """Save the report when leaving the context block."""
+        if self._fname is not None:
+            self.save(self._fname, open_browser=False, overwrite=True)
+        return self
+
     @verbose
     def _render_toc(self, verbose=None):
         """Render the Table of Contents."""


Test patch:
diff --git a/mne/tests/test_report.py b/mne/tests/test_report.py
--- a/mne/tests/test_report.py
+++ b/mne/tests/test_report.py
@@ -16,9 +16,9 @@
 from mne import Epochs, read_events, read_evokeds
 from mne.io import read_raw_fif
 from mne.datasets import testing
-from mne.report import Report
+from mne.report import Report, open_report
 from mne.utils import (_TempDir, requires_mayavi, requires_nibabel,
-                       run_tests_if_main, traits_test)
+                       run_tests_if_main, traits_test, requires_h5py)
 from mne.viz import plot_alignment
 
 import matplotlib
@@ -307,4 +307,34 @@ def test_validate_input():
     assert_equal(len(comments_new), len(items))
 
 
+@requires_h5py
+def test_open_report():
+    """Test the open_report function."""
+    tempdir = _TempDir()
+    hdf5 = op.join(tempdir, 'report.h5')
+    import matplotlib.pyplot as plt
+    fig = plt.plot([1, 2], [1, 2])[0].figure
+
+    # Test creating a new report through the open_report function
+    with open_report(hdf5, subjects_dir=subjects_dir) as report:
+        assert report.subjects_dir == subjects_dir
+        assert report._fname == hdf5
+        report.add_figs_to_section(figs=fig, captions=['evoked response'])
+    # Exiting the context block should have triggered saving to HDF5
+    assert op.exists(hdf5)
+
+    # Load the HDF5 version of the report and check equivalency
+    report2 = open_report(hdf5)
+    assert report2._fname == hdf5
+    assert report2.subjects_dir == report.subjects_dir
+    assert report2.html == report.html
+    assert report2.__getstate__() == report.__getstate__()
+    assert '_fname' not in report2.__getstate__()
+
+    # Check parameters when loading a report
+    pytest.raises(ValueError, open_report, hdf5, foo='bar')  # non-existing
+    pytest.raises(ValueError, open_report, hdf5, subjects_dir='foo')
+    open_report(hdf5, subjects_dir=subjects_dir)  # This should work
+
+
 run_tests_if_main()