Repository: sphinx-gallery/sphinx-gallery

Issue:
Add a mode that fails only for rst warnings and does not run examples
Idea by @jeromedockes

Have a mode where the examples are built without running the code and the links to images are not inserted. This mode would be used to detect errors in the restructured text without waiting for examples to run, which can take a long time. It is important in such a mode that the links to images are not inserted, because when the examples are not run, the images are not present, and hence they lead to sphinx warnings that drown the important warnings.

@jeromedockes, did I explain well your ideas? @lucyleeow : does this make sense?

Discussion:
GaelVaroquaux: That's a pretty big deal!! Thanks a lot Lucy!

@jeromedockes @NicolasGensollen, do you want to have a look to see if
this is going in the direction that could help nilearn.


lucyleeow: Thinking about it more, it may be useful for the dummy images to be generated ONLY if the example is not executable (e.g., plot_gallery is False). This way you would not have to remove the config once you are done doing local testing. Though, in the current design, accidentally leaving the config should not cause problems as the images are added after any generated plots, you would just have additional images.

Are there cases when the dummy image generating functionality would be useful when the examples are executable?

Note making it only work when not executable would make testing much more difficult.
jeromedockes: 
> Though, in the current design, accidentally leaving the config should not cause problems as the images are added after any generated plots, you would just have additional images.

for the case of nilearn at least I don't think having some extra images
would be a problem, as long as the true images are the ones that show up
in the gallery when they are available.
but having to change this configuration parameter depending on whether
the examples are executed would not be a problem either.

thanks for working on this! let us know when it is ready to try it on
the nilearn example gallery

GaelVaroquaux: > Thinking about it more, it may be useful for the dummy images to be generated ONLY if the example is not executable

+1


lucyleeow: @larsoner is it possible to run tinybuild with different config (e.g., plot_gallery: False)? Alternatively we could create another tinierbuild that runs with plot_gallery False, but I worry about maintenance.
larsoner: From the command line you can always change config values using something like `-D sphinx_gallery_conf.plot_gallery=False`, there is presumably some way to do something equivalent from our app testing infrastructure, too (temporarily setting `sys.argv`, maybe?). This would be my preferred solution.

Another way would be to `monkeypatch` our own `_complete_gallery_conf` to always set `plot_gallery=False`.

Another option (ugliest IMO) is to `shutil.copytree` to a `tmpdir`, then modify `conf.py`, then make the sphinx app.
lucyleeow: @larsoner I added a new file `test_full_noexec.py` and in the sphinx app here added `confoverrides` argument.

Was not sure what best to do about temp dir:
```
temp_dir = (tmpdir_factory.getbasetemp() / 'root_nonexec').strpath
```

I had to name it something else, as 'root' was already created by `test_full.py`. 
larsoner: Personally I don't use `tmpdir_factory` in pytest but just the `tmpdir` fixture, which IIRC uses a different tempdir per test function so you shouldn't have collisions. But what you did sounds reasonable, too
lucyleeow: > Personally I don't use `tmpdir_factory` in pytest but just the `tmpdir` fixture, which IIRC uses a different tempdir per test function so you shouldn't have collisions. But what you did sounds reasonable, too

How about I amend to using `tmpdir` (instead of tmpdir_factory - maybe we don't even need this fun) in both `test_full.py` and `test_full_noexec.py` ?
lucyleeow: > How about I amend to using tmpdir (instead of tmpdir_factory - maybe we don't even need this fun) in both test_full.py and test_full_noexec.py ?

So this does not work because `tmpdir` is a function scope fixture, and it can't be used in `sphinx_app` which is module scoped.
lucyleeow: This is ready for review if you want to take a look at it again.

Can fix errors once #830 and #831 are merged in.
larsoner: There is a merge conflict now

Patch:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 06193b1..58080fb 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -55,6 +55,7 @@ Some options can also be set or overridden on a file-by-file basis:
 - ``# sphinx_gallery_line_numbers`` (:ref:`adding_line_numbers`)
 - ``# sphinx_gallery_thumbnail_number`` (:ref:`choosing_thumbnail`)
 - ``# sphinx_gallery_thumbnail_path`` (:ref:`providing_thumbnail`)
+- ``# sphinx_gallery_dummy_images`` (:ref:`dummy_images`)
 
 Some options can be set on a per-code-block basis in a file:
 
@@ -1246,6 +1247,51 @@ further deferred, if desired).  The following produces only one plot::
   plt.plot([2, 2])
   plt.show()
 
+.. _dummy_images:
+
+Generating dummy images
+=======================
+
+For quick visualization of your gallery, especially during the writing process,
+Sphinx-Gallery allows you to build your gallery without executing the
+code (see :ref:`without_execution` and
+:ref:`filename/ignore patterns <build_pattern>`). This however,
+can cause warnings about missing image files if you have manually written
+links to automatically generated images. To prevent these warnings you can
+tell Sphinx-Gallery to create a number of dummy images for an example.
+
+For example, you may have an example ('my_example.py') that generates 2 figures,
+which you then reference manually elsewhere, e.g.,:
+
+.. code-block:: rst
+
+    Below is a great figure:
+
+    .. figure:: ../auto_examples/images/sphx_glr_my_example_001.png
+
+    Here is another one:
+
+    .. figure:: ../auto_examples/images/sphx_glr_my_example_002.png
+
+To prevent missing image file warnings when building without executing, you
+can add the following to the example file::
+
+    # sphinx_gallery_dummy_images=2
+
+This will cause Sphinx-Gallery to generate 2 dummy images with the same
+naming convention and stored in the same location as images that would be
+generated when building with execution. No dummy images will be generated
+if there are existing images (e.g., from a previous run of the build),
+so they will not be overwritten.
+
+.. note::
+    This configuration **only** works when the example is set to not execute
+    (i.e., the ``plot_gallery`` is False, the example is in `ignore_pattern`
+    or the example is not in `filename_pattern - see
+    :ref:`filename/ignore patterns <build_pattern>`). This means that you will
+    not need to remove any ``sphinx_gallery_dummy_images`` lines in your
+    examples when you switch to building your gallery with execution.
+
 .. _reset_modules:
 
 Resetting modules
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index 4309f58..3e2389c 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -865,6 +865,21 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf,
 
     logger.debug("%s ran in : %.2g seconds\n", src_file, time_elapsed)
 
+    # Create dummy images
+    if not executable:
+        dummy_image = file_conf.get('dummy_images', None)
+        if dummy_image is not None:
+            if type(dummy_image) is not int:
+                raise ExtensionError(
+                    'sphinx_gallery_dummy_images setting is not a number, '
+                    'got %r' % (dummy_image,))
+
+            image_path_iterator = script_vars['image_path_iterator']
+            stock_img = os.path.join(glr_path_static(), 'no_image.png')
+            for _, path in zip(range(dummy_image), image_path_iterator):
+                if not os.path.isfile(path):
+                    copyfile(stock_img, path)
+
     if gallery_conf['remove_config_comments']:
         script_blocks = [
             (label, remove_config_comments(content), line_number)


Test patch:
diff --git a/sphinx_gallery/tests/test_full.py b/sphinx_gallery/tests/test_full.py
index 878fd13..60da4b5 100644
--- a/sphinx_gallery/tests/test_full.py
+++ b/sphinx_gallery/tests/test_full.py
@@ -897,3 +897,14 @@ def test_defer_figures(sphinx_app):
     # be only one image, not two, in the output.
     assert '../_images/sphx_glr_plot_defer_figures_001.png' in html
     assert '../_images/sphx_glr_plot_defer_figures_002.png' not in html
+
+
+def test_no_dummy_image(sphinx_app):
+    """Test that sphinx_gallery_dummy_images are NOT created (when executable
+    is True)."""
+    img1 = op.join(sphinx_app.srcdir, 'auto_examples', 'images',
+                   'sphx_glr_plot_repr_001.png')
+    img2 = op.join(sphinx_app.srcdir, 'auto_examples', 'images',
+                   'sphx_glr_plot_repr_002.png')
+    assert not op.isfile(img1)
+    assert not op.isfile(img2)
diff --git a/sphinx_gallery/tests/test_full_noexec.py b/sphinx_gallery/tests/test_full_noexec.py
new file mode 100644
index 0000000..28b7f73
--- /dev/null
+++ b/sphinx_gallery/tests/test_full_noexec.py
@@ -0,0 +1,52 @@
+# -*- coding: utf-8 -*-
+# License: 3-clause BSD
+"""
+Test the SG pipeline using Sphinx and tinybuild
+"""
+from io import StringIO
+import os.path as op
+import shutil
+
+from sphinx.application import Sphinx
+from sphinx.util.docutils import docutils_namespace
+
+import pytest
+
+
+@pytest.fixture(scope='module')
+def sphinx_app(tmpdir_factory, req_mpl, req_pil):
+
+    temp_dir = (tmpdir_factory.getbasetemp() / 'root_nonexec').strpath
+    src_dir = op.join(op.dirname(__file__), 'tinybuild')
+
+    def ignore(src, names):
+        return ('_build', 'gen_modules', 'auto_examples')
+
+    shutil.copytree(src_dir, temp_dir, ignore=ignore)
+    # For testing iteration, you can get similar behavior just doing `make`
+    # inside the tinybuild directory
+    src_dir = temp_dir
+    conf_dir = temp_dir
+    out_dir = op.join(temp_dir, '_build', 'html')
+    toctrees_dir = op.join(temp_dir, '_build', 'toctrees')
+    # Avoid warnings about re-registration, see:
+    # https://github.com/sphinx-doc/sphinx/issues/5038
+    confoverrides = {'sphinx_gallery_conf.plot_gallery': 0, }
+    with docutils_namespace():
+        app = Sphinx(src_dir, conf_dir, out_dir, toctrees_dir,
+                     buildername='html', confoverrides=confoverrides,
+                     status=StringIO(), warning=StringIO())
+        # need to build within the context manager
+        # for automodule and backrefs to work
+        app.build(False, [])
+    return app
+
+
+def test_dummy_image(sphinx_app):
+    """Test that sphinx_gallery_dummy_images are created."""
+    img1 = op.join(sphinx_app.srcdir, 'auto_examples', 'images',
+                   'sphx_glr_plot_repr_001.png')
+    img2 = op.join(sphinx_app.srcdir, 'auto_examples', 'images',
+                   'sphx_glr_plot_repr_002.png')
+    assert op.isfile(img1)
+    assert op.isfile(img2)
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index 72d8571..3ce90fb 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -454,14 +454,22 @@ def test_remove_config_comments(gallery_conf, req_pil):
     assert '# sphinx_gallery_defer_figures' not in rst
 
 
+def test_dummy_image_error(gallery_conf, req_pil):
+    """Test correct error is raised if int not provided to
+    sphinx_gallery_dummy_images."""
+    content_image = CONTENT + ["# sphinx_gallery_dummy_images=False", ]
+    msg = "sphinx_gallery_dummy_images setting is not a number"
+    with pytest.raises(ExtensionError, match=msg):
+        _generate_rst(gallery_conf, 'test.py', content_image)
+
+
 def test_final_empty_block(gallery_conf, req_pil):
     """Test empty final block is removed. Empty final block can occur after
     sole config comment is removed from final block."""
-    CONTENT.extend(
-        ('# %%', '', '# sphinx_gallery_line_numbers = True')
-    )
+    content_block = \
+        CONTENT + ['# %%', '', '# sphinx_gallery_line_numbers = True']
     gallery_conf['remove_config_comments'] = True
-    rst = _generate_rst(gallery_conf, 'test.py', CONTENT)
+    rst = _generate_rst(gallery_conf, 'test.py', content_block)
     want = "RuntimeWarning)\n\n\n.. rst-class:: sphx-glr-timing"
     assert want in rst
 
diff --git a/sphinx_gallery/tests/tinybuild/examples/plot_repr.py b/sphinx_gallery/tests/tinybuild/examples/plot_repr.py
index f6893af..0837e9f 100644
--- a/sphinx_gallery/tests/tinybuild/examples/plot_repr.py
+++ b/sphinx_gallery/tests/tinybuild/examples/plot_repr.py
@@ -1,10 +1,11 @@
 """
 Repr test
 =========
-Test repr.
+Test repr and the sphinx_gallery_dummy_images config.
 """
 
 
+# sphinx_gallery_dummy_images=2
 class A:
     def _repr_html_(self):
         return '<p><b>This should print<b></p>'