Repository: sphinx-gallery/sphinx-gallery

Issue:
Feature Request: `reset_modules` to be applied after each or all examples
My understanding is that the functions supplied to `reset_modules` will run _before_ each example.  But if the final example modifies a module, then the modified modules will leak out into other parts of the documentation building process.  This is particularly troublesome when regenerating a single failed example that modifies a module, which I've run into when trying to implement `reset_modules` to address https://github.com/pyvista/pyvista/issues/1711.  Right now, I need to manually force the build process to process one extra example to successful reset the module, which works, but is cumbersome.

It would be helpful to also provide a `reset_modules_post` which would work the same way, but runs _after_ each example or after all examples are run.  This would guard against this edge case.  

A counterargument to this request would be that we should reset the module in each build step, e.g. docstring examples.  But, I think it would be beneficial to have consistency in the end state after `sphinx-gallery` is completed that does not depend on the order of running the examples.

Discussion:
larsoner: > This is a bit of magic for the user, but I like it. 

I also think of it as a way for us to add new information to callbacks in a backward-compatible way. I don't think the implementation is too painful and I don't anticipate any terrible maintenance from it, so why not :)
MatthewFlamm: Currently, we aren't restricting users to have a function signature that has exactly the same parameters defined, only that there are 2 parameters.  I've taken the same approach and have inspected for function with 3 parameters, otherwise fallback to the old behavior.

Looks like python 3.8 passed by 3.6 all failed due to some difference in inspect like you mentioned.  There could also be a difference in `unittest.mock`, but I haven't checked yet.  The only possible relevant difference I can see in the documentation between 3.6 and 3.7 is the below, but it shouldn't be applicable here.  https://docs.python.org/3/library/inspect.html#inspect.Signature.parameters

> Changed in version 3.7: Python only explicitly guaranteed that it preserved the declaration order of keyword-only parameters as of version 3.7, although in practice this order had always been preserved in Python 3.


larsoner: I would prefer to check for kwarg name rather than number, it's more explicit that people have when=None in their signatures than some_other_name and rely on that in the third position

I recall needing a shim until 3.7 for inspection, let's just bump...
MatthewFlamm: Are you suggesting enforcing that `when` is a parameter?  Or that `when` additionally has a default (note that we always provide a value, so I'm unclear on why we would enforce a signature with a default)?

In `def fun(a, b, when=None)`, all parameters are `POSITIONAL_OR_KEYWORD`, but `when` has a default.

 I would more strongly not recommend that we enforce `when` to be a keyword only argument as the function signature becomes more complex, e.g. `def fun(a, b, *, when=None):`.

I'm leaning even more towards the current PR state.  It seems overly restrictive to require users to use the same variable names with no clear benefit to me.  I think it will be surprising to users that they cannot use any name as this isn't common practice.  And that we would enforce the name for only 1 out of 3 parameters will lead to even more confusion.


MatthewFlamm: Sorry for all the pushes in a row, I know it can be a lot of notifications potentially.  I had to figure out how to run the local style check to match the CI, and then all the subtleties with the CI requirements.  I think I have it figured out now after poring over the different CI scripts, so it should be less going forward.

I could not find any examples of projects using `inspect` like this to require function signatures to have certain names.  The most prevalent examples I could find are for documentation purposes, for example to modify or style parameters that are shown in the documentation.   But, I'm OK in implementing it if it is this project's preference.  Either way, documentation is very important here, which still needs to be worked on.
larsoner: > I could not find any examples of projects using inspect like this to require function signatures to have certain names

We use a `_get_args` [in MNE-Python](https://github.com/mne-tools/mne-python/blob/main/mne/fixes.py#L77-L88) all the time to look to see for example if `matplotlib` or [`pyvista` has a specific argument](https://github.com/mne-tools/mne-python/blob/75a13167e3d89fae36aeda7da2f34cd00cd3e995/mne/viz/backends/_pyvista.py#L585) as it's sometimes easier than relying on a version number. It has worked well over the years.

>  Either way, documentation is very important here, which still needs to be worked on.

I think we just need to say that in the callback, feel free to add a `when` after the first two positional arguments (should work as any of `a, b, when`, `a, b, when=None`, or `a, b, *, when`) and then the callback will get the information. I don't think it's too complicated...?
MatthewFlamm: I think these examples introduce more flexibility to the users, which I would be in support of.  As I understand, one example checks number of arguments and then adjusts the code to work with varying numbers.  The other checks for a specific argument for specific functionality.  Here we are only limiting flexibility to users by requiring a name. 

One case where this naming requirement would add flexibility is if we allowed users to use any combination, order, number of parameter inputs like:

```
def mpl_reset(fname):
    ...
```

vs.

```
def mpl_reset(when, gallery_conf):
    ...
```

For these examples, it is required to know which parameter name is used and in what order. In the current PR state, it is determined by number of parameters only, we just restrict the third parameters name.

But, I agree that it isn't complex, and if it is wanted by this project, I am OK implementing.  In fact, it is already implemented.  `when` is now the required third parameter.  I am working on the documentation.

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 73f5f23..d165c45 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -83,7 +83,7 @@ jobs:
 
   deploy_dev:
     docker:
-      - image: circleci/python:3.6-stretch
+      - image: circleci/python:3.7-stretch
     steps:
       - checkout
       - add_ssh_keys:
@@ -95,7 +95,7 @@ jobs:
 
   deploy_stable:
     docker:
-      - image: circleci/python:3.6-stretch
+      - image: circleci/python:3.7-stretch
     steps:
       - checkout
       - add_ssh_keys:
diff --git a/README.rst b/README.rst
index 1703727..0b85dc1 100644
--- a/README.rst
+++ b/README.rst
@@ -46,6 +46,7 @@ Who uses Sphinx-Gallery
 * `Auto-sklearn <https://automl.github.io/auto-sklearn/master/examples/index.html>`_
 * `OpenML-Python <https://openml.github.io/openml-python/master/examples/index.html>`_
 * `TorchIO <https://torchio.readthedocs.io/auto_examples/index.html>`_
+* `Neuraxle <https://www.neuraxle.org/stable/examples/index.html>`_
 
 .. installation-begin-content
 
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 8a53a0e..c09becf 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -39,8 +39,8 @@ stages:
       vmImage: 'vs2017-win2016'
     strategy:
       matrix:
-        Python36:
-          python.version: '3.6'
+        Python37:
+          python.version: '3.7'
         Python38:
           python.version: '3.8'
 
@@ -49,7 +49,9 @@ stages:
         call "%CONDA%\Scripts\activate" base
         echo "##vso[task.setvariable variable=PATH]%CONDA%\Scripts;%PATH%"
       displayName: Add conda to PATH and activate conda base
-    - script: conda install python=%PYTHON_VERSION% ipython numpy seaborn statsmodels matplotlib sphinx pillow pytest pytest-cov sphinx_rtd_theme joblib plotly
+    - script: |
+        conda install python=%PYTHON_VERSION% ipython numpy seaborn statsmodels matplotlib sphinx pillow pytest pytest-cov sphinx_rtd_theme joblib "plotly>=4.0"
+        conda list
       displayName: Setup conda environment
     - script: python setup.py develop
       displayName: Install sphinx-gallery
@@ -82,17 +84,17 @@ stages:
       DISPLAY: ':99'
     strategy:
       matrix:
-        ubuntu_python36:
+        ubuntu_python37:
           DISTRIB: 'ubuntu'
-          PYTHON_VERSION: '3.6'
+          PYTHON_VERSION: '3.7'
           SPHINXOPTS: ''
-        conda_python36_sphinx183:
+        conda_python37_sphinx183:
           DISTRIB: 'conda'
-          PYTHON_VERSION: '3.6'
+          PYTHON_VERSION: '3.7'
           SPHINX_VERSION: '1.8.3'
-        conda_python36_localeC:
+        conda_python37_localeC:
           DISTRIB: 'conda'
-          PYTHON_VERSION: '3.6'
+          PYTHON_VERSION: '3.7'
           LOCALE: 'C'
           BAD_MTIME: '1'  # not clear why, maybe some specific Sphinx version?
         conda_python38_sphinxDev:
diff --git a/continuous_integration/azure/install.sh b/continuous_integration/azure/install.sh
index 8599c04..ae63322 100644
--- a/continuous_integration/azure/install.sh
+++ b/continuous_integration/azure/install.sh
@@ -15,10 +15,10 @@ if [ "$DISTRIB" == "conda" ]; then
     echo "##vso[task.prependpath]$CONDA/bin"
     export PATH=${CONDA}/bin:${PATH}
     CONDA_TO_INSTALL="$@"
-    CONDA_TO_INSTALL="$CONDA_TO_INSTALL python=$PYTHON_VERSION pip numpy setuptools matplotlib pillow pytest pytest-cov coverage seaborn statsmodels plotly joblib flake8 wheel"
+    CONDA_TO_INSTALL="$CONDA_TO_INSTALL python=$PYTHON_VERSION pip numpy setuptools matplotlib pillow pytest pytest-cov coverage seaborn statsmodels 'plotly>=4.0' joblib flake8 wheel"
     PIP_DEPENDENCIES="$@"
     PIP_DEPENDENCIES="$PIP_DEPENDENCIES sphinx_rtd_theme check-manifest"
-    if [ "$PYTHON_VERSION" != "3.6" -o "$LOCALE" != "C" ]; then
+    if [ "$PYTHON_VERSION" != "3.7" -o "$LOCALE" != "C" ]; then
         PIP_DEPENDENCIES="${PIP_DEPENDENCIES} memory_profiler vtk<=9.0.1 traits<6.3.0 https://github.com/enthought/mayavi/zipball/master ipython pypandoc"
     fi
     if [ "$SPHINX_VERSION" == "" ]; then
diff --git a/doc/advanced.rst b/doc/advanced.rst
index de160fc..09603cf 100644
--- a/doc/advanced.rst
+++ b/doc/advanced.rst
@@ -356,12 +356,12 @@ it with Sphinx-Gallery. You can:
 Resetting before each example
 =============================
 
-Sphinx-Gallery supports 'resetting' via a prolog that is run before each
-example script is executed. This is used natively in Sphinx-Gallery to 'reset'
+Sphinx-Gallery supports 'resetting' function(s) that are run before and/or after
+each example script is executed. This is used natively in Sphinx-Gallery to 'reset'
 the behavior of the visualization packages ``matplotlib`` and ``seaborn``
 (:ref:`reset_modules`). However, this functionality could be used to reset other
 libraries, modify the resetting behavior for a natively-reset library or run
-an arbitrary custom function at the start of each script.
+an arbitrary custom function at the start and/or end of each script.
 
 This is done by adding a custom function to the resetting tuple defined in
 ``conf.py``. The function should take two variables: a dictionary called
@@ -393,6 +393,22 @@ function (see :ref:`reset_modules`).
           examples themselves is discouraged due to the inconsistency
           that results between the rendered examples and local outputs.
 
+If the custom function needs to be aware of whether it is being run before or after
+an example, a function signature with three parameters can be used, where the third parameter
+is required to be named ``when``::
+
+    def reset_mpl(gallery_conf, fname, when):
+
+        import matplotlib as mpl
+        mpl.rcParams['lines.linewidth'] = 2
+        if when == 'after' and fname=='dashed_lines':
+            mpl.rcParams['lines.linestyle'] = '-'
+
+The value passed into ``when`` can be ``before`` or ``after``.
+If ``reset_modules_order`` in the :ref:`configuration <reset_modules_order>` is set to ``before`` or
+``after``, ``when`` will always be the same value.  This function signature
+is only useful when used in conjuction with ``reset_modules_order`` set to ``both``.
+
 Altering Sphinx-Gallery CSS
 ===========================
 
diff --git a/doc/conf.py b/doc/conf.py
index 6a21936..93b56e1 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -332,10 +332,11 @@ else:
 
 # Set plotly renderer to capture _repr_html_ for sphinx-gallery
 try:
-    import plotly.io as pio
-    pio.renderers.default = 'sphinx_gallery'
+    import plotly.io.renderers
 except ImportError:
     pass
+else:
+    plotly.io.renderers.default = 'sphinx_gallery'
 
 min_reported_time = 0
 if 'SOURCE_DATE_EPOCH' in os.environ:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 77e225f..451f6b2 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -36,6 +36,7 @@ file:
 - ``compress_images`` (:ref:`compress_images`)
 - ``image_srcset`` (:ref:`image_srcset`)
 - ``reset_modules`` (:ref:`reset_modules`)
+- ``reset_modules_order`` (:ref:`reset_modules_order`)
 - ``abort_on_example_error`` (:ref:`abort_on_first`)
 - ``only_warn_on_example_error`` (:ref:`warning_on_error`)
 - ``expected_failing_examples`` (:ref:`dont_fail_exit`)
@@ -388,6 +389,9 @@ If you do not use ``intersphinx``, then you should add entries that
 point to the directory containing ``searchindex.js``, such as
 ``'matplotlib': 'https://matplotlib.org'``.
 
+If you wish to do the same for ordinary RST documentation,
+see :ref:`plain_rst`.
+
 .. _references_to_examples:
 
 Add mini-galleries for API documentation
@@ -1287,7 +1291,7 @@ so they will not be overwritten.
 .. note::
     This configuration **only** works when the example is set to not execute
     (i.e., the ``plot_gallery`` is False, the example is in `ignore_pattern`
-    or the example is not in `filename_pattern - see
+    or the example is not in ``filename_pattern`` - see
     :ref:`filename/ignore patterns <build_pattern>`). This means that you will
     not need to remove any ``sphinx_gallery_dummy_images`` lines in your
     examples when you switch to building your gallery with execution.
@@ -1317,6 +1321,26 @@ this tuple in order to define resetting behavior for other visualization librari
 
 To do so, follow the instructions in :ref:`custom_reset`.
 
+.. _reset_modules_order:
+
+Order of resetting modules
+==========================
+
+By default, Sphinx-Gallery will reset modules before each example is run.
+The choices for ``reset_modules_order`` are ``before`` (default), ``after``, and
+``both``. If the last example run in Sphinx-Gallery modifies a module, it is
+recommended to use ``after`` or ``both`` to avoid leaking out a modified module to
+other parts of the Sphinx build process.  For example, set ``reset_modules_order``
+to ``both`` in the configuration::
+
+    sphinx_gallery_conf = {
+        ...
+        'reset_modules_order': 'both',
+    }
+
+Custom functions can be constructed to have custom functionality depending on
+whether they are called before or after the examples.  See :ref:`custom_reset`
+for more information.
 
 Dealing with failing Gallery example scripts
 ============================================
diff --git a/doc/projects_list.rst b/doc/projects_list.rst
index bd49476..5c21dce 100644
--- a/doc/projects_list.rst
+++ b/doc/projects_list.rst
@@ -28,3 +28,4 @@ Here is a list of projects using `sphinx-gallery`.
 * `Auto-sklearn <https://automl.github.io/auto-sklearn/master/examples/index.html>`_
 * `OpenML-Python <https://openml.github.io/openml-python/master/examples/index.html>`_
 * `TorchIO <https://torchio.readthedocs.io/auto_examples/index.html>`_
+* `Neuraxle <https://www.neuraxle.org/stable/examples/index.html>`_
diff --git a/doc/syntax.rst b/doc/syntax.rst
index 73f956c..0039675 100644
--- a/doc/syntax.rst
+++ b/doc/syntax.rst
@@ -136,3 +136,15 @@ functionality::
 For a clear example refer to the rendered example
 :ref:`sphx_glr_tutorials_plot_parse.py` and compare it to the generated
 :download:`original python script <tutorials/plot_parse.py>`
+
+.. _plain_rst:
+
+Plain RST examples
+==================
+
+Sphinx-Gallery generates examples from Python scripts,
+so examples written in plain RST files are not supported.
+If you're looking to generate hyperlinks for functions (linking to their
+corresponding online documentation) in code blocks of ordinary RST
+documentation, you might find
+`sphinx-codeautolink <https://sphinx-codeautolink.rtfd.io>`_ helpful.
diff --git a/setup.py b/setup.py
index f222797..0e7a481 100644
--- a/setup.py
+++ b/setup.py
@@ -49,7 +49,7 @@ setup(
     author="Óscar Nájera",
     author_email='najera.oscar@gmail.com',
     install_requires=install_requires,
-    python_requires='>=3.6',
+    python_requires='>=3.7',
     license='3-clause BSD',
     classifiers=['Intended Audience :: Developers',
                  'Development Status :: 4 - Beta',
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index 3f9448c..aff560f 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -81,6 +81,7 @@ DEFAULT_GALLERY_CONF = {
     'image_scrapers': ('matplotlib',),
     'compress_images': (),
     'reset_modules': ('matplotlib', 'seaborn'),
+    'reset_modules_order': 'before',
     'first_notebook_cell': '%matplotlib inline',
     'last_notebook_cell': None,
     'notebook_images': False,
@@ -287,6 +288,14 @@ def _complete_gallery_conf(sphinx_gallery_conf, src_dir, plot_gallery,
                               % (resetter,))
     gallery_conf['reset_modules'] = tuple(resetters)
 
+    if not isinstance(gallery_conf['reset_modules_order'], str):
+        raise ConfigError('reset_modules_order must be a str, '
+                          'got %r' % gallery_conf['reset_modules_order'])
+    if gallery_conf['reset_modules_order'] not in ['before', 'after', 'both']:
+        raise ConfigError("reset_modules_order must be in"
+                          "['before', 'after', 'both'], "
+                          'got %r' % gallery_conf['reset_modules_order'])
+
     lang = lang if lang in ('python', 'python3', 'default') else 'python'
     gallery_conf['lang'] = lang
     del resetters
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index 8b42810..cac39b1 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -925,8 +925,11 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf,
         'src_file': src_file,
         'target_file': target_file}
 
-    if executable:
-        clean_modules(gallery_conf, fname)
+    if (
+        executable
+        and gallery_conf['reset_modules_order'] in ['before', 'both']
+    ):
+        clean_modules(gallery_conf, fname, 'before')
     output_blocks, time_elapsed = execute_script(script_blocks,
                                                  script_vars,
                                                  gallery_conf)
@@ -995,6 +998,9 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf,
     _write_backreferences(backrefs, seen_backrefs, gallery_conf, target_dir,
                           fname, intro, title)
 
+    if executable and gallery_conf['reset_modules_order'] in ['after', 'both']:
+        clean_modules(gallery_conf, fname, 'after')
+
     return intro, title, (time_elapsed, memory_used)
 
 
diff --git a/sphinx_gallery/load_style.py b/sphinx_gallery/load_style.py
index b600f0a..6ceccd7 100644
--- a/sphinx_gallery/load_style.py
+++ b/sphinx_gallery/load_style.py
@@ -6,6 +6,7 @@ This should not be used at the same time as sphinx_gallery.gen_gallery.
 
 """
 from . import __version__, glr_path_static
+from .directives import ImageSg, imagesg_addnode
 
 
 def config_inited(app, config):
@@ -18,6 +19,8 @@ def config_inited(app, config):
 def setup(app):
     app.require_sphinx('1.8')
     app.connect('config-inited', config_inited)
+    app.add_directive("image-sg", ImageSg)
+    imagesg_addnode(app)
     return {
         'parallel_read_safe': True,
         'version': __version__,
diff --git a/sphinx_gallery/scrapers.py b/sphinx_gallery/scrapers.py
index 1ee8dca..33eae3a 100644
--- a/sphinx_gallery/scrapers.py
+++ b/sphinx_gallery/scrapers.py
@@ -13,6 +13,7 @@ images are injected as rst ``image-sg`` directives into the ``.rst``
 file generated for each example script.
 """
 
+import inspect
 import os
 import sys
 import re
@@ -564,8 +565,8 @@ _reset_dict = {
 }
 
 
-def clean_modules(gallery_conf, fname):
-    """Remove, unload, or reset modules after running each example.
+def clean_modules(gallery_conf, fname, when):
+    """Remove, unload, or reset modules.
 
     After a script is executed it can load a variety of settings that one
     does not want to influence in other examples in the gallery.
@@ -577,6 +578,21 @@ def clean_modules(gallery_conf, fname):
     fname : str or None
         The example being run. Will be None when this is called entering
         a directory of examples to be built.
+    when : str
+        Whether this module is run before or after examples.
+
+        This parameter is only forwarded when the callables accept 3
+        parameters.
     """
     for reset_module in gallery_conf['reset_modules']:
-        reset_module(gallery_conf, fname)
+
+        sig = inspect.signature(reset_module)
+        if len(sig.parameters) == 3:
+            third_param = list(sig.parameters.keys())[2]
+            if third_param != 'when':
+                raise ValueError(f"3rd parameter in {reset_module.__name__} "
+                                 "function signature must be 'when', "
+                                 f"got {third_param}")
+            reset_module(gallery_conf, fname, when=when)
+        else:
+            reset_module(gallery_conf, fname)


Test patch:
diff --git a/sphinx_gallery/tests/test_full.py b/sphinx_gallery/tests/test_full.py
index 60da4b5..3ba93c7 100644
--- a/sphinx_gallery/tests/test_full.py
+++ b/sphinx_gallery/tests/test_full.py
@@ -694,7 +694,9 @@ def test_error_messages(sphinx_app, name, want):
     src_dir = sphinx_app.srcdir
     example_rst = op.join(src_dir, 'auto_examples', name + '.rst')
     with codecs.open(example_rst, 'r', 'utf-8') as fid:
-        rst = fid.read().replace('\n', ' ')
+        rst = fid.read()
+    print(rst)
+    rst = rst.replace('\n', ' ')
     assert re.match(want, rst) is not None
 
 
diff --git a/sphinx_gallery/tests/test_gen_gallery.py b/sphinx_gallery/tests/test_gen_gallery.py
index 16ec492..705f1c5 100644
--- a/sphinx_gallery/tests/test_gen_gallery.py
+++ b/sphinx_gallery/tests/test_gen_gallery.py
@@ -75,6 +75,23 @@ def test_bad_reset(sphinx_app_wrapper, err_class, err_match):
         sphinx_app_wrapper.create_sphinx_app()
 
 
+@pytest.mark.parametrize('err_class, err_match', [
+    pytest.param(ConfigError, 'reset_modules_order must be a str',
+                 id='Resetter unknown',
+                 marks=pytest.mark.conf_file(
+                     content=("sphinx_gallery_conf="
+                              "{'reset_modules_order': 1,}"))),
+    pytest.param(ConfigError, "reset_modules_order must be in",
+                 id='reset_modules_order not valid',
+                 marks=pytest.mark.conf_file(
+                     content=("sphinx_gallery_conf="
+                              "{'reset_modules_order': 'invalid',}"))),
+])
+def test_bad_reset_modules_order(sphinx_app_wrapper, err_class, err_match):
+    with pytest.raises(err_class, match=err_match):
+        sphinx_app_wrapper.create_sphinx_app()
+
+
 @pytest.mark.parametrize('err_class, err_match', [
     pytest.param(ConfigError, 'Unknown css', id='CSS str error',
                  marks=pytest.mark.conf_file(
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index a3dcc06..d26f581 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -13,6 +13,7 @@ import tempfile
 import re
 import os
 import shutil
+from unittest import mock
 import zipfile
 import codeop
 
@@ -823,6 +824,65 @@ def test_ignore_repr_types(gallery_conf, req_mpl, req_pil, script_vars):
     assert _clean_output(output) == ''
 
 
+@pytest.mark.parametrize(
+    ('order', 'call_count'), [('before', 1), ('after', 1), ('both', 2)]
+)
+def test_reset_module_order_2_param(gallery_conf, order, call_count, req_pil):
+    """Test that reset module with 2 parameters."""
+
+    def cleanup_2_param(gallery_conf, fname):
+        pass
+
+    mock_reset_module = mock.create_autospec(cleanup_2_param)
+    gallery_conf['reset_modules'] = (mock_reset_module,)
+    gallery_conf['reset_modules_order'] = order
+    _generate_rst(gallery_conf, 'plot_test.py', CONTENT)
+    assert mock_reset_module.call_count == call_count
+
+
+@pytest.mark.parametrize(
+    ('order', 'call_count', 'expected_call_order'),
+    [
+        ('before', 1, ('before',)),
+        ('after', 1, ('after',)),
+        ('both', 2, ('before', 'after'))
+    ]
+)
+def test_reset_module_order_3_param(gallery_conf, order, call_count,
+                                    expected_call_order, req_pil):
+    """Test reset module with 3 parameters."""
+
+    def cleanup_3_param(gallery_conf, fname, when):
+        pass
+
+    mock_reset_module = mock.create_autospec(cleanup_3_param)
+    gallery_conf['reset_modules'] = (mock_reset_module,)
+    gallery_conf['reset_modules_order'] = order
+    _generate_rst(gallery_conf, 'plot_test.py', CONTENT)
+    assert mock_reset_module.call_count == call_count
+
+    expected_calls = [
+        mock.call(mock.ANY, mock.ANY, order) for order in expected_call_order
+    ]
+    mock_reset_module.assert_has_calls(expected_calls)
+
+
+def test_reset_module_order_3_param_invalid_when(gallery_conf):
+    """Test reset module with unknown 3rd parameter."""
+
+    def cleanup_3_param(gallery_conf, fname, invalid):
+        pass
+
+    mock_reset_module = mock.create_autospec(cleanup_3_param)
+    gallery_conf['reset_modules'] = (mock_reset_module,)
+    gallery_conf['reset_modules_order'] = 'before'
+    with pytest.raises(ValueError,
+                       match=("3rd parameter in cleanup_3_param "
+                              "function signature must be 'when'")):
+        _generate_rst(gallery_conf, 'plot_test.py', CONTENT)
+    assert mock_reset_module.call_count == 0
+
+
 class TestLoggingTee:
     def setup(self):
         self.src_filename = 'source file name'
diff --git a/sphinx_gallery/tests/tinybuild/conf.py b/sphinx_gallery/tests/tinybuild/conf.py
index 0477147..cf18c90 100644
--- a/sphinx_gallery/tests/tinybuild/conf.py
+++ b/sphinx_gallery/tests/tinybuild/conf.py
@@ -1,5 +1,4 @@
 import os.path as op
-import numpy as np
 from sphinx_gallery.scrapers import matplotlib_scraper
 from sphinx_gallery.sorting import FileNameSortKey
 
@@ -28,21 +27,28 @@ class ResetArgv:
             return []
 
 
+def _raise(*args, **kwargs):
+    import matplotlib.pyplot as plt
+    plt.close('all')
+    raise ValueError('zero-size array to reduction operation minimum which '
+                     'has no identity')
+
+
 class MockScrapeProblem:
 
     def __init__(self):
-        from matplotlib.figure import Figure
-        self._orig_mpl = Figure.savefig
+        from matplotlib.colors import colorConverter
+        self._orig = colorConverter.to_rgba
 
     def __repr__(self):
         return "MockScrapeProblem"
 
     def __call__(self, gallery_conf, fname):
-        from matplotlib.figure import Figure
+        from matplotlib.colors import colorConverter
         if 'scraper_broken' in fname:
-            Figure.savefig = lambda *args, **kwargs: np.min([])
+            colorConverter.to_rgba = _raise
         else:
-            Figure.savefig = self._orig_mpl
+            colorConverter.to_rgba = self._orig
 
 
 extensions = [