Repository: sphinx-gallery/sphinx-gallery

Issue:
Delay output capturing to a further code block
Imagine you have a piece of code that would produce some output. Yet, you do not want to show that output (because it's too long or too distracting). Is that possible? 

In a more concrete example, I wish to create a matplotlib plot in several steps, having explanations as text in between the code. However I only want the final plot to show up. As an example:

```
"""
Header
================

Create a plot by importing matplotlib 
and creating a figure with axes
"""

import matplotlib.pyplot as plt
fig, ax = plt.subplots()

####################
# Now we want to add a plot, we do so using 
# :meth: `plot <matplotlib.pyplot.plot>`
# 

ax.plot( .... )
ax.other_stuff()

####################
# Finally we can show the plot
# 

plt.show()
```

Of course you would only want to have the complete plot being shown at the time you call `show`. 

In a jupyter notebook you would probably use `%%capture` in those intermediate cells and last state `fig` to show the plot. But how do you do it here?






Discussion:
larsoner: I have definitely wanted something like this before. It looks like there is a failure on Sphinx dev but this might be Sphinx's fault...

It would be good to add a test for this somewhere, though. Maybe in `test_gen_rst.py`?
ayshih: Okay, documentation and tests added, and all checks are green aside from Sphinx dev
lucyleeow: Will take a look this weekend, thanks @larsoner 
lucyleeow: Did we want to do something about the sphinx dev fail? Is it possible to mark xfail if sphinx is a certain version?
larsoner: > Did we want to do something about the sphinx dev fail? 

Someone should probably look in a separate PR to see why it fails. For example maybe they added another class to the CSS, so we just need to update. Or they removed one and it's not backward compatible and we need to open an issue with them.

It sounds like you might be volunteering to look @lucyleeow ? :)
lucyleeow: > It sounds like you might be volunteering to look @lucyleeow ? :)

It's on my todo!

Patch:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 6c074ad..3e20aeb 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -55,8 +55,12 @@ Some options can also be set or overridden on a file-by-file basis:
 - ``# sphinx_gallery_thumbnail_number`` (:ref:`choosing_thumbnail`)
 - ``# sphinx_gallery_thumbnail_path`` (:ref:`providing_thumbnail`)
 
-See also :ref:`removing_config_comments` to hide these comments from the
-rendered examples.
+Some options can be set on a per-code-block basis in a file:
+
+- ``# sphinx_gallery_defer_figures`` (:ref:`defer_figures`)
+
+See also :ref:`removing_config_comments` to hide config comments in files from
+the rendered examples.
 
 Some options can be set during the build execution step, e.g. using a Makefile:
 
@@ -647,8 +651,8 @@ setting::
 Removing config comments
 ========================
 
-Some configurations can be done on a file-by-file basis by adding a special
-comment with the pattern :samp:`# sphinx_gallery_{config} = {value}` to the
+Some configurations can be specified within a file by adding a special
+comment with the pattern :samp:`# sphinx_gallery_{config} [= {value}]` to the
 example source files. By default, the source files are parsed as is and thus
 the comment will appear in the example.
 
@@ -660,8 +664,8 @@ To remove the comment from the rendered example set the option::
     }
 
 This only removes configuration comments from code blocks, not from text
-blocks. However, note that technically, configuration comments will work when
-put in either code blocks or text blocks.
+blocks. However, note that technically, file-level configuration comments will
+work when put in either code blocks or text blocks.
 
 .. _own_notebook_cell:
 
@@ -1094,7 +1098,6 @@ optimize less but speed up the build time you could do::
 
 See ``$ optipng --help`` for a complete list of options.
 
-
 .. _image_scrapers:
 
 Image scrapers
@@ -1158,6 +1161,48 @@ useful for general use (e.g., a custom scraper for a plotting library)
 feel free to add it to the list above (see discussion
 `here <https://github.com/sphinx-gallery/sphinx-gallery/issues/441#issuecomment-493782430>`__)!
 
+.. _defer_figures:
+
+Using multiple code blocks to create a single figure
+====================================================
+
+By default, images are scraped following each code block in an example.  Thus,
+the following produces two plots, with one plot per code block::
+
+  # %%
+  # This first code block produces a plot with two lines
+
+  import matplotlib.pyplot as plt
+  plt.plot([1, 0])
+  plt.plot([0, 1])
+
+  # %%
+  # This second code block produces a plot with one line
+
+  plt.plot([2, 2])
+  plt.show()
+
+However, sometimes it can be useful to use multiple code blocks to create a
+single figure, particularly if the figure takes a large number commands that
+would benefit from being interleaved with text blocks.  The optional flag
+``sphinx_gallery_defer_figures`` can be inserted as a comment anywhere in a code
+block to defer the scraping of images to the next code block (where it can be
+further deferred, if desired).  The following produces only one plot::
+
+  # %%
+  # This first code block does not produce any plot
+
+  import matplotlib.pyplot as plt
+  plt.plot([1, 0])
+  plt.plot([0, 1])
+  # sphinx_gallery_defer_figures
+
+  # %%
+  # This second code block produces a plot with three lines
+
+  plt.plot([2, 2])
+  plt.show()
+
 .. _reset_modules:
 
 Resetting modules
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index e5966ae..c65fe46 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -559,7 +559,12 @@ def execute_code_block(compiler, block, example_globals,
 
     sys_path = copy.deepcopy(sys.path)
     sys.path.append(os.getcwd())
-    need_save_figures = True
+
+    # Save figures unless there is a `sphinx_gallery_defer_figures` flag
+    match = re.search(r'^[\ \t]*#\s*sphinx_gallery_defer_figures[\ \t]*\n?',
+                      bcontent, re.MULTILINE)
+    need_save_figures = match is None
+
     try:
         dont_inherit = 1
         if sys.version_info >= (3, 8):
@@ -600,8 +605,11 @@ def execute_code_block(compiler, block, example_globals,
         script_vars['memory_delta'].append(mem_max)
         # This should be inside the try block, e.g., in case of a savefig error
         logging_tee.restore_std()
-        need_save_figures = False
-        images_rst = save_figures(block, script_vars, gallery_conf)
+        if need_save_figures:
+            need_save_figures = False
+            images_rst = save_figures(block, script_vars, gallery_conf)
+        else:
+            images_rst = u''
     except Exception:
         logging_tee.restore_std()
         except_rst = handle_exception(sys.exc_info(), src_file, script_vars,
@@ -842,6 +850,14 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf,
         'src_file': src_file,
         'target_file': target_file}
 
+    if executable:
+        clean_modules(gallery_conf, fname)
+    output_blocks, time_elapsed = execute_script(script_blocks,
+                                                 script_vars,
+                                                 gallery_conf)
+
+    logger.debug("%s ran in : %.2g seconds\n", src_file, time_elapsed)
+
     if gallery_conf['remove_config_comments']:
         script_blocks = [
             (label, remove_config_comments(content), line_number)
@@ -852,14 +868,7 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf,
     # are removed
     if script_blocks[-1][1].isspace():
         script_blocks = script_blocks[:-1]
-
-    if executable:
-        clean_modules(gallery_conf, fname)
-    output_blocks, time_elapsed = execute_script(script_blocks,
-                                                 script_vars,
-                                                 gallery_conf)
-
-    logger.debug("%s ran in : %.2g seconds\n", src_file, time_elapsed)
+        output_blocks = output_blocks[:-1]
 
     example_rst = rst_blocks(script_blocks, output_blocks,
                              file_conf, gallery_conf)
diff --git a/sphinx_gallery/py_source_parser.py b/sphinx_gallery/py_source_parser.py
index f5faa53..29b7012 100644
--- a/sphinx_gallery/py_source_parser.py
+++ b/sphinx_gallery/py_source_parser.py
@@ -40,7 +40,7 @@ Example script with invalid Python syntax
 #
 #     b = 2
 INFILE_CONFIG_PATTERN = re.compile(
-    r"^[\ \t]*#\s*sphinx_gallery_([A-Za-z0-9_]+)\s*=\s*(.+)[\ \t]*\n?",
+    r"^[\ \t]*#\s*sphinx_gallery_([A-Za-z0-9_]+)(\s*=\s*(.+))?[\ \t]*\n?",
     re.MULTILINE)
 
 
@@ -137,7 +137,9 @@ def extract_file_config(content):
     file_conf = {}
     for match in re.finditer(INFILE_CONFIG_PATTERN, content):
         name = match.group(1)
-        value = match.group(2)
+        value = match.group(3)
+        if value is None:  # a flag rather than a config setting
+            continue
         try:
             value = ast.literal_eval(value)
         except (SyntaxError, ValueError):


Test patch:
diff --git a/sphinx_gallery/tests/test_full.py b/sphinx_gallery/tests/test_full.py
index fc94464..a20dc45 100644
--- a/sphinx_gallery/tests/test_full.py
+++ b/sphinx_gallery/tests/test_full.py
@@ -23,7 +23,7 @@ from sphinx_gallery.utils import (_get_image, scale_image, _has_optipng,
 
 import pytest
 
-N_TOT = 12
+N_TOT = 13
 
 N_FAILING = 2
 N_GOOD = N_TOT - N_FAILING
@@ -908,3 +908,17 @@ def test_binder_logo_exists(sphinx_app):
     assert 'binder_badge_logo' in img_fname  # can have numbers appended
     assert op.isfile(img_fname)
     assert 'https://mybinder.org/v2/gh/sphinx-gallery/sphinx-gallery.github.io/master?urlpath=lab/tree/notebooks/auto_examples/plot_svg.ipynb' in html  # noqa: E501
+
+
+def test_defer_figures(sphinx_app):
+    """Test the deferring of figures."""
+    root = op.join(sphinx_app.outdir, 'auto_examples')
+    fname = op.join(root, 'plot_defer_figures.html')
+    with codecs.open(fname, 'r', 'utf-8') as fid:
+        html = fid.read()
+
+    # The example has two code blocks with plotting commands, but the first
+    # block has the flag ``sphinx_gallery_defer_figures``.  Thus, there should
+    # be only one image, not two, in the output.
+    assert '../_images/sphx_glr_plot_defer_figures_001.png' in html
+    assert '../_images/sphx_glr_plot_defer_figures_002.png' not in html
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index 443faff..72d8571 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -37,7 +37,8 @@ CONTENT = [
     'And this is a second paragraph',
     '"""',
     '',
-    '# sphinx_gallery_thumbnail_number = 1'
+    '# sphinx_gallery_thumbnail_number = 1',
+    '# sphinx_gallery_defer_figures',
     '# and now comes the module code',
     'import logging',
     'import sys',
@@ -445,10 +446,12 @@ def test_remove_config_comments(gallery_conf, req_pil):
     """Test the gallery_conf['remove_config_comments'] setting."""
     rst = _generate_rst(gallery_conf, 'test.py', CONTENT)
     assert '# sphinx_gallery_thumbnail_number = 1' in rst
+    assert '# sphinx_gallery_defer_figures' in rst
 
     gallery_conf['remove_config_comments'] = True
     rst = _generate_rst(gallery_conf, 'test.py', CONTENT)
     assert '# sphinx_gallery_thumbnail_number = 1' not in rst
+    assert '# sphinx_gallery_defer_figures' not in rst
 
 
 def test_final_empty_block(gallery_conf, req_pil):
diff --git a/sphinx_gallery/tests/test_py_source_parser.py b/sphinx_gallery/tests/test_py_source_parser.py
index 8c3da35..8ad04de 100644
--- a/sphinx_gallery/tests/test_py_source_parser.py
+++ b/sphinx_gallery/tests/test_py_source_parser.py
@@ -45,7 +45,10 @@ def test_get_docstring_and_rest(unicode_sample, tmpdir, monkeypatch):
      {'line_numbers': True}),
     ("#sphinx_gallery_thumbnail_number\n=\n5",
      {'thumbnail_number': 5}),
-    ('#sphinx_gallery_thumbnail_number=1foo', None),
+    ("#sphinx_gallery_thumbnail_number=1foo",
+     None),
+    ("# sphinx_gallery_defer_figures",
+     {}),
 ])
 def test_extract_file_config(content, file_conf, log_collector):
     if file_conf is None:
@@ -74,6 +77,8 @@ def test_extract_file_config(content, file_conf, log_collector):
      "a = 1\n\n\nb = 1"),
     ("# comment\n# sphinx_gallery_line_numbers = True\n# commment 2",
      "# comment\n# commment 2"),
+    ("# sphinx_gallery_defer_figures",
+     ""),
 ])
 def test_remove_config_comments(contents, result):
     assert sg.remove_config_comments(contents) == result
diff --git a/sphinx_gallery/tests/tinybuild/examples/plot_defer_figures.py b/sphinx_gallery/tests/tinybuild/examples/plot_defer_figures.py
new file mode 100644
index 0000000..fe418da
--- /dev/null
+++ b/sphinx_gallery/tests/tinybuild/examples/plot_defer_figures.py
@@ -0,0 +1,21 @@
+"""
+Test plot deferring
+===================
+
+This tests the ``sphinx_gallery_defer_figures`` flag.
+"""
+
+import matplotlib.pyplot as plt
+
+# %%
+# This code block should produce no plot.
+
+plt.plot([0, 1])
+plt.plot([1, 0])
+# sphinx_gallery_defer_figures
+
+# %%
+# This code block should produce a plot with three lines.
+
+plt.plot([2, 2])
+plt.show()