Repository: sphinx-gallery/sphinx-gallery

Issue:
Add a mode which "skips" an example if it fails
We have the situation where some of our examples are dependant on remote servers or other things which are a little flakey. For our tagged releases we want these to pass, but for our regular builds on our development branch, it would be better if they rendered the same way as if they were skipped.

Would people be interested in a config option which didn't cause sphinx to exit with a non-zero code if an example errored, and just renders the output of that example with no output at all?

Discussion:
Cadair: I tested this out with sunpy here: https://github.com/sunpy/sunpy/pull/4993 note how the Azure build_docs job fails because the gallery fails to build, but RTD is green so you can see the preview and some of the examples show "broken" with the errors.
Cadair: Thanks for the review @larsoner.

I added two tests, one where it asserts the error is logged and the other that asserts it is raised if the equivalent of `-W` is set. I didn't think the `False` behaviour needed an explicit test as it's already covered by existing tests? I am not sure the `-W` test is really needed either, given you could probably argue that's just testing sphinx. :shrug: 

Patch:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index ea51f97..df4728d 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -36,6 +36,7 @@ file:
 - ``compress_images`` (:ref:`compress_images`)
 - ``reset_modules`` (:ref:`reset_modules`)
 - ``abort_on_example_error`` (:ref:`abort_on_first`)
+- ``only_warn_on_example_error`` (:ref:`warning_on_error`)
 - ``expected_failing_examples`` (:ref:`dont_fail_exit`)
 - ``min_reported_time`` (:ref:`min_reported_time`)
 - ``show_memory`` (:ref:`show_memory`)
@@ -1233,8 +1234,8 @@ the ``sphinx-build`` command.
 
 .. _dont_fail_exit:
 
-Don't fail the build on exit
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Don't fail the build if specific examples error
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 It might be the case that you want to keep the gallery even with
 failed examples. Thus you can configure Sphinx-Gallery to allow
@@ -1254,6 +1255,18 @@ the example script.
 .. note:: If an example is expected to fail, Sphinx-Gallery will error if
           the example runs without error.
 
+.. _warning_on_error:
+
+Never fail the build on error
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Sphinx-Gallery can be configured to only log warnings when examples fail. This means that sphinx will only exit with a non-zero exit code if the ``-W`` flag is passed to ``sphinx-build``. This can be enabled by setting::
+
+    sphinx_gallery_conf = {
+        ...
+        'only_warn_on_example_error': True
+    }
+
 
 .. _setting_thumbnail_size:
 
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index d6dd19b..5a30913 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -69,6 +69,7 @@ DEFAULT_GALLERY_CONF = {
     'plot_gallery': 'True',
     'download_all_examples': True,
     'abort_on_example_error': False,
+    'only_warn_on_example_error': False,
     'failing_examples': {},
     'passing_examples': [],
     'stale_examples': [],  # ones that did not need to be run due to md5sum
@@ -712,10 +713,13 @@ def summarize_failing_examples(app, exception):
                 color='brown')
 
     if fail_msgs:
-        raise ExtensionError(
-            "Here is a summary of the problems encountered "
-            "when running the examples\n\n" + "\n".join(fail_msgs) +
-            "\n" + "-" * 79)
+        fail_message = ("Here is a summary of the problems encountered "
+                        "when running the examples\n\n" +
+                        "\n".join(fail_msgs) + "\n" + "-" * 79)
+        if gallery_conf['only_warn_on_example_error']:
+            logger.warning(fail_message)
+        else:
+            raise ExtensionError(fail_message)
 
 
 def collect_gallery_files(examples_dirs, gallery_conf):


Test patch:
diff --git a/sphinx_gallery/tests/test_gen_gallery.py b/sphinx_gallery/tests/test_gen_gallery.py
index b8576bf..16ec492 100644
--- a/sphinx_gallery/tests/test_gen_gallery.py
+++ b/sphinx_gallery/tests/test_gen_gallery.py
@@ -4,16 +4,14 @@
 r"""
 Test Sphinx-Gallery
 """
-
-from __future__ import (division, absolute_import, print_function,
-                        unicode_literals)
 import codecs
 import os
 import re
+from pathlib import Path
 
 import pytest
 
-from sphinx.errors import ConfigError, ExtensionError
+from sphinx.errors import ConfigError, ExtensionError, SphinxWarning
 from sphinx_gallery.gen_gallery import (
     check_duplicate_filenames, check_spaces_in_filenames,
     collect_gallery_files, write_computation_times, _complete_gallery_conf)
@@ -321,6 +319,45 @@ def test_expected_failing_examples_were_executed(sphinx_app_wrapper):
     sphinx_app_wrapper.build_sphinx_app()
 
 
+@pytest.mark.conf_file(content="""
+sphinx_gallery_conf = {
+    'examples_dirs': 'src',
+    'gallery_dirs': 'ex',
+    'only_warn_on_example_error': True,
+}""")
+def test_only_warn_on_example_error(sphinx_app_wrapper):
+    """
+    Test behaviour of only_warn_on_example_error flag.
+    """
+    example_dir = Path(sphinx_app_wrapper.srcdir) / 'src'
+    with codecs.open(example_dir / 'plot_3.py', 'a', encoding='utf-8') as fid:
+        fid.write('raise ValueError')
+    sphinx_app = sphinx_app_wrapper.build_sphinx_app()
+
+    build_warn = sphinx_app._warning.getvalue()
+    assert 'plot_3.py failed to execute correctly' in build_warn
+    assert 'WARNING: Here is a summary of the problems' in build_warn
+
+
+@pytest.mark.conf_file(content="""
+sphinx_gallery_conf = {
+    'examples_dirs': 'src',
+    'gallery_dirs': 'ex',
+    'only_warn_on_example_error': True,
+}""")
+def test_only_warn_on_example_error_sphinx_warning(sphinx_app_wrapper):
+    """
+    Test behaviour of only_warn_on_example_error flag.
+    """
+    sphinx_app_wrapper.kwargs['warningiserror'] = True
+    example_dir = Path(sphinx_app_wrapper.srcdir) / 'src'
+    with codecs.open(example_dir / 'plot_3.py', 'a', encoding='utf-8') as fid:
+        fid.write('raise ValueError')
+    with pytest.raises(SphinxWarning) as excinfo:
+        sphinx_app_wrapper.build_sphinx_app()
+    assert "plot_3.py failed to execute" in str(excinfo.value)
+
+
 @pytest.mark.conf_file(content="""
 sphinx_gallery_conf = {
     'examples_dirs': 'src',