Repository: sphinx-gallery/sphinx-gallery

Issue:
Stop example execution as soon as one cell has an error
In a large majority of cases, an error when executing one cell, is going to cause further cells to have errors. I think it just adds unnecessary noise:

This is the root cause of the problem:
```
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 3, in <module>
  File "/home/ubuntu/nilearn/nilearn/datasets/func.py", line 481, in fetch_adhd
    resume=resume, verbose=verbose)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 728, in _fetch_files
    raise IOError('Fetching aborted: ' + abort)
IOError: Fetching aborted: CRC check failed 0x55f58d0f != 0xd6aafce5L
```

And this is all the additional stuff you get from trying to execute cells after the first failure:
```
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 6, in <module>
NameError: name 'func_filenames' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 4, in <module>
NameError: name 'components_img' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 4, in <module>
NameError: name 'components_img' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 3, in <module>
  File "/home/ubuntu/nilearn/nilearn/datasets/func.py", line 481, in fetch_adhd
    resume=resume, verbose=verbose)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 695, in _fetch_files
    overwrite=overwrite)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 523, in _fetch_file
    data = url_opener.open(request)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused| Error while fetching file adhd40_0010064.tgz; dataset fetching aborted.>
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 7, in <module>
NameError: name 'func_filenames' is not defined

```

Discussion:
Titan-C: More of a design question. Now that there is non­-zero exit code on example building, does it still make sense to have a non-building example in the gallery, which is build then by travis and will always fail?
GaelVaroquaux: > More of a design question. Now that there is non­-zero exit code on
> example building, does it still make sense to have a non-building
> example in the gallery, which is build then by travis and will always
> fail?

I think that it is a question left to the user of whether to fail or not
the build.

Titan-C: This is working quite well, I needs a more systematic test suite I believe.
I have written some documentation to explain my defaults on this one

It should also
Fix #105
Fix #94 because first image is no longer overwritten

Titan-C: Travis is unhappy for the Mayavi test case. I can't figure out why it does not work, I had solved that problem with the pyface forced update. And If I redo in a clean virtual environment it works, somehow in travis pyface does not get updated
lesteve: @Titan-C you need to rebase on master unfortunately ! This is probably due to #110 which was merged.
lesteve: @Titan-C still need a rebase.
Titan-C: @lesteve Now is working

Patch:
diff --git a/.travis.yml b/.travis.yml
index 943cfde..a1b0a30 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -56,4 +56,5 @@ script:
     - if [ "$DISTRIB" == "ubuntu" ]; then python setup.py nosetests; fi
     - if [ "$DISTRIB" == "conda" ]; then nosetests; fi
     - cd doc
+    - make html-noplot
     - make html
diff --git a/CHANGES.rst b/CHANGES.rst
index 1675f21..1b7b378 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -6,11 +6,19 @@ git master
 
 New features
 ''''''''''''
-
+* Summary of failing examples with traceback at the end of the sphinx
+  build. By default the build exits with a 1 exit code if an example
+  has failed. A list of examples that are expected to fail can be
+  defined in `conf.py` and exit the build with 0
+  exit code. Alternatively it is possible to exit the build as soon as
+  one example has failed.
 * Print aggregated and sorted list of computation times of all examples
   in the console during the build.
 * For examples that create multiple figures, set the thumbnail image.
-
+* The ``plot_gallery`` and ``abort_on_example_error`` options can now
+  be specified in ``sphinx_gallery_conf``. The build option (``-D``
+  flag passed to ``sphinx-build``) takes precedence over the
+  ``sphinx_gallery_conf`` option.
 
 v0.1.2
 ------
@@ -41,7 +49,7 @@ Example scripts are now available for download as IPython Notebooks
 `#75 <https://github.com/sphinx-gallery/sphinx-gallery/pull/75>`_
 
 New features
-------------
+''''''''''''
 
 * Configurable filename pattern to select which example scripts are
   executed while building the Gallery
diff --git a/README.rst b/README.rst
index 9109581..b5f146e 100644
--- a/README.rst
+++ b/README.rst
@@ -26,7 +26,6 @@ Who uses Sphinx-Gallery
 * `GIMLi <http://www.pygimli.org/_examples_auto/index.html>`_
 * `Nestle <http://kbarbary.github.io/nestle/examples/index.html>`_
 * `pyRiemann <http://pythonhosted.org/pyriemann/auto_examples/index.html>`_
-* `scikit-image <http://scikit-image.org/docs/dev/auto_examples/>`_
 
 
 Getting the package
diff --git a/doc/advanced_configuration.rst b/doc/advanced_configuration.rst
index d129d6b..fc01992 100644
--- a/doc/advanced_configuration.rst
+++ b/doc/advanced_configuration.rst
@@ -55,11 +55,11 @@ you would do:
     }
 
 Here, one should escape the dot ``'\.'`` as otherwise python `regular expressions`_ matches any character. Nevertheless, as
-one is targetting a specific file, it is most certainly going to match the dot in the filename.
+one is targeting a specific file, it is most certainly going to match the dot in the filename.
 
 Similarly, to build only examples in a specific directory, you can do:
 
-.. code-blocK:: python
+.. code-block:: python
 
     sphinx_gallery_conf = {
         'filename_pattern' : '/directory/plot_'
@@ -251,7 +251,13 @@ your ``Makefile`` with::
         @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
 Remember that for ``Makefile`` white space is significant and the indentation are tabs
-and not spaces
+and not spaces.
+
+Alternatively, you can add the ``plot_gallery`` option to the
+``sphinx_gallery_conf`` dictionary inside your ``conf.py``
+configuration file to have it as a default. The highest precedence is
+always given to the `-D` flag of the ``sphinx-build`` command.
+
 
 Dealing with failing Gallery example scripts
 ============================================
@@ -267,7 +273,17 @@ failing code block. Refer to example
 :ref:`sphx_glr_auto_examples_plot_raise.py` to view the default
 behavior.
 
-An extra functionality of Sphinx-Gallery is the early fail option. In
+The build is also failed exiting with code 1 and giving you a summary
+of the failed examples with their respective traceback. This way you
+are aware of failing examples right after the build and can find them
+easily.
+
+There are some additional options at your hand to deal with broken examples.
+
+Abort build on first fail
+-------------------------
+
+Sphinx-Gallery provides the early fail option. In
 this mode the gallery build process breaks as soon as an exception
 occurs in the execution of the examples scripts. To activate this
 behavior you need to pass a flag at the build process. It can be done
@@ -279,6 +295,34 @@ by including in your ``Makefile``::
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
 Remember that for ``Makefile`` white space is significant and the indentation are tabs
-and not spaces
+and not spaces.
+
+Alternatively, you can add the ``abort_on_example_error`` option to
+the ``sphinx_gallery_conf`` dictionary inside your ``conf.py``
+configuration file to have it as a default. The highest precedence is
+always given to the `-D` flag of the ``sphinx-build`` command.
+
+
+Don't fail the build on exit
+----------------------------
+
+It might be the case that you want to keep the gallery even with
+failed examples. Thus you can configure Sphinx-Gallery to allow
+certain examples to fail and still exit with a 0 exit code. For this
+you need to list all the examples you want to allow to fail during
+build. Change your `conf.py` accordingly:
+
+
+.. code-block:: python
+
+    sphinx_gallery_conf = {
+        ...
+	'expected_failing_examples': ['../examples/plot_raise.py']
+    }
+
+Here you list the examples you allow to fail during the build process,
+keep in mind to specify the full relative path from your `conf.py` to
+the example script.
+
 
 .. _regular expressions: https://docs.python.org/2/library/re.html
diff --git a/doc/conf.py b/doc/conf.py
index 788b37e..04aad13 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -32,6 +32,7 @@ import sphinx_gallery
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx.ext.napoleon',
     'sphinx.ext.autosummary',
     'sphinx.ext.doctest',
     'sphinx.ext.intersphinx',
@@ -114,6 +115,7 @@ pygments_style = 'sphinx'
 # a list of builtin themes.
 html_theme = 'default'
 
+
 def setup(app):
     app.add_stylesheet('theme_override.css')
 
@@ -199,22 +201,22 @@ htmlhelp_basename = 'Sphinx-Gallerydoc'
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
+    # The paper size ('letterpaper' or 'a4paper').
+    #'papersize': 'letterpaper',
 
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
+    # The font size ('10pt', '11pt' or '12pt').
+    #'pointsize': '10pt',
 
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
+    # Additional stuff for the LaTeX preamble.
+    #'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-  ('index', 'Sphinx-Gallery.tex', u'Sphinx-Gallery Documentation',
-   u'Óscar Nájera', 'manual'),
+    ('index', 'Sphinx-Gallery.tex', u'Sphinx-Gallery Documentation',
+     u'Óscar Nájera', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -257,9 +259,9 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  ('index', 'Sphinx-Gallery', u'Sphinx-Gallery Documentation',
-   u'Óscar Nájera', 'Sphinx-Gallery', 'One line description of project.',
-   'Miscellaneous'),
+    ('index', 'Sphinx-Gallery', u'Sphinx-Gallery Documentation',
+     u'Óscar Nájera', 'Sphinx-Gallery', 'One line description of project.',
+     'Miscellaneous'),
 ]
 
 # Documents to append as an appendix to all manuals.
@@ -304,4 +306,5 @@ sphinx_gallery_conf = {
     'examples_dirs': examples_dirs,
     'gallery_dirs': gallery_dirs,
     'find_mayavi_figures': find_mayavi_figures,
-    }
+    'expected_failing_examples': ['../examples/plot_raise.py']
+}
diff --git a/doc/index.rst b/doc/index.rst
index 83ef58e..789e22d 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -55,8 +55,6 @@ Features
   with the :ref:`notebook_examples_syntax`
 
 
-Sphinx-Gallery Show: :ref:`examples-index`
-------------------------------------------
 
 Contents:
 ---------
@@ -73,6 +71,10 @@ Contents:
    auto_mayavi_examples/index
    changes
 
+Sphinx-Gallery Show: :ref:`examples-index`
+''''''''''''''''''''''''''''''''''''''''''
+
+
 
 Indices and tables
 ==================
diff --git a/examples/plot_quantum.py b/examples/plot_quantum.py
index 527a641..18300eb 100644
--- a/examples/plot_quantum.py
+++ b/examples/plot_quantum.py
@@ -60,7 +60,7 @@ mu = np.linspace(0, 3, 800)
 for b in [10, 20, 30]:
     n = 2 * (np.exp(b * (mu - 1)) + np.exp(b * (2 * mu - 3))) / \
         (1 + np.exp(b * (mu - 1)) * (2 + np.exp(b * (mu - 2))))
-    plt.plot(mu, n, label=r"$\beta={}$".format(b))
+    plt.plot(mu, n, label=r"$\beta={0}$".format(b))
 plt.xlabel(r'$\mu$ ($\epsilon=1$, $U=1$)')
 plt.ylabel(r'$\langle N \rangle=\langle n_\uparrow \rangle+\langle n_\downarrow\rangle$')
 plt.legend(loc=0)
diff --git a/examples/plot_raise.py b/examples/plot_raise.py
index 88a05de..24e0152 100644
--- a/examples/plot_raise.py
+++ b/examples/plot_raise.py
@@ -10,14 +10,6 @@ image stamp. Thus allowing easy identification in the gallery display.
 You also get the python traceback of the failed code block
 """
 
-iae
-
-###############################################################################
-# Sphinx gallery as it executes scripts by block will continue
-# evaluating the script after exceptions, but there is no warranty
-# figure ordering will continue to match block's code. Anyway when the
-# script is broken, you should try to fix it first.
-
 # Code source: Óscar Nájera
 # License: BSD 3 clause
 
@@ -26,8 +18,18 @@ import matplotlib.pyplot as plt
 
 plt.pcolormesh(np.random.randn(100, 100))
 
+###############################################################################
+# This next block will raise a NameError
+
+iae
+
+###############################################################################
+# Sphinx gallery will stop executing the remaining code blocks after
+# the exception has occurred in the example script. Nevertheless the
+# html will still render all the example annotated text and
+# code blocks, but no output will be shown.
 
 ###############################################################################
-# Here is another error raising Block
+# Here is another error raising block but will not be executed
 
 plt.plot('Strings are not a valid argument for the plot function')
diff --git a/sphinx_gallery/_static/gallery.css b/sphinx_gallery/_static/gallery.css
index f4f71ed..86cf0c4 100644
--- a/sphinx_gallery/_static/gallery.css
+++ b/sphinx_gallery/_static/gallery.css
@@ -1,8 +1,8 @@
 /*
-Sphinx-Gallery has compatible CSS to fix default sphinx themes
+Sphinx-Gallery is has compatible CSS to fix default sphinx themes
 Tested for Sphinx 1.3.1 for all themes: default, alabaster, sphinxdoc,
 scrolls, agogo, traditional, nature, haiku, pyramid
-Tested for Read the Docs theme 0.1.7 */
+Tested for Read the docs theme 0.1.7 */
 .sphx-glr-thumbcontainer {
   background: #fff;
   border: solid #fff 1px;
@@ -132,5 +132,6 @@ p.sphx-glr-signature a.reference.external {
   -webkit-border-radius: 5px;
   border-radius: 5px;
   padding: 3px;
-  font-size: 75%;
+  position: absolute;
+  right: 15px;
 }
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index 9b742cf..7e59381 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -12,12 +12,27 @@ when building the documentation.
 
 
 from __future__ import division, print_function, absolute_import
+import copy
 import re
 import os
 from . import glr_path_static
 from .gen_rst import generate_dir_rst, SPHX_GLR_SIG
 from .docs_resolv import embed_code_links
 
+DEFAULT_GALLERY_CONF = {
+    'filename_pattern': re.escape(os.sep) + 'plot',
+    'examples_dirs': os.path.join('..', 'examples'),
+    'gallery_dirs': 'auto_examples',
+    'mod_example_dir': os.path.join('modules', 'generated'),
+    'doc_module': (),
+    'reference_url': {},
+    # build options
+    'plot_gallery': True,
+    'abort_on_example_error': False,
+    'failing_examples': {},
+    'expected_failing_examples': set(),
+}
+
 
 def clean_gallery_out(build_dir):
     """Deletes images under the sphx_glr namespace in the build directory"""
@@ -55,6 +70,7 @@ def generate_gallery_rst(app):
     except TypeError:
         plot_gallery = bool(app.builder.config.plot_gallery)
 
+    gallery_conf = copy.deepcopy(DEFAULT_GALLERY_CONF)
     gallery_conf.update(app.config.sphinx_gallery_conf)
     gallery_conf.update(plot_gallery=plot_gallery)
     gallery_conf.update(
@@ -143,21 +159,68 @@ def touch_empty_backreferences(app, what, name, obj, options, lines):
         open(examples_path, 'w').close()
 
 
-gallery_conf = {
-    'filename_pattern': re.escape(os.sep) + 'plot',
-    'examples_dirs': '../examples',
-    'gallery_dirs': 'auto_examples',
-    'mod_example_dir': os.path.join('modules', 'generated'),
-    'doc_module': (),
-    'reference_url': {},
-}
+def sumarize_failing_examples(app, exception):
+    """Collects the list of falling examples during build and prints them with the traceback
+
+    Raises ValueError if there where failing examples
+    """
+    if exception is not None:
+        return
+
+    # Under no-plot Examples are not run so nothing to summarize
+    if not app.config.sphinx_gallery_conf['plot_gallery']:
+        return
+
+    gallery_conf = app.config.sphinx_gallery_conf
+    failing_examples = set(gallery_conf['failing_examples'])
+    expected_failing_examples = set(gallery_conf['expected_failing_examples'])
+
+    examples_expected_to_fail = failing_examples.intersection(
+        expected_failing_examples)
+    expected_fail_msg = []
+    if examples_expected_to_fail:
+        expected_fail_msg.append("Examples failing as expected:")
+        for fail_example in examples_expected_to_fail:
+            expected_fail_msg.append(fail_example + ' failed leaving traceback:\n' +
+                                     gallery_conf['failing_examples'][fail_example] + '\n')
+        print("\n".join(expected_fail_msg))
+
+    examples_not_expected_to_fail = failing_examples.difference(
+        expected_failing_examples)
+    fail_msgs = []
+    if examples_not_expected_to_fail:
+        fail_msgs.append("Unexpected failing examples:")
+        for fail_example in examples_not_expected_to_fail:
+            fail_msgs.append(fail_example + ' failed leaving traceback:\n' +
+                             gallery_conf['failing_examples'][fail_example] + '\n')
+
+    examples_not_expected_to_pass = expected_failing_examples.difference(
+        failing_examples)
+    if examples_not_expected_to_pass:
+        fail_msgs.append("Examples expected to fail, but not failling:\n" +
+                         "Please remove this examples from\n" +
+                         "sphinx_gallery_conf['expected_failing_examples']\n" +
+                         "in your conf.py file"
+                         "\n".join(examples_not_expected_to_pass))
+
+    if fail_msgs:
+        raise ValueError("Here is a summary of the problems encountered when "
+                         "running the examples\n\n" + "\n".join(fail_msgs) +
+                         "\n" + "-" * 79)
+
+
+def get_default_config_value(key):
+    def default_getter(conf):
+        return conf['sphinx_gallery_conf'].get(key, DEFAULT_GALLERY_CONF[key])
+    return default_getter
 
 
 def setup(app):
     """Setup sphinx-gallery sphinx extension"""
-    app.add_config_value('plot_gallery', True, 'html')
-    app.add_config_value('abort_on_example_error', False, 'html')
-    app.add_config_value('sphinx_gallery_conf', gallery_conf, 'html')
+    app.add_config_value('sphinx_gallery_conf', DEFAULT_GALLERY_CONF, 'html')
+    for key in ['plot_gallery', 'abort_on_example_error']:
+        app.add_config_value(key, get_default_config_value(key), 'html')
+
     app.add_stylesheet('gallery.css')
 
     if 'sphinx.ext.autodoc' in app._extensions:
@@ -165,6 +228,7 @@ def setup(app):
 
     app.connect('builder-inited', generate_gallery_rst)
 
+    app.connect('build-finished', sumarize_failing_examples)
     app.connect('build-finished', embed_code_links)
 
 
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index b6779ad..7385662 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -96,6 +96,7 @@ class Tee(object):
 
 class MixedEncodingStringIO(StringIO):
     """Helper when both ASCII and unicode strings will be written"""
+
     def write(self, data):
         if not isinstance(data, unicode):
             data = data.decode('utf-8')
@@ -143,7 +144,7 @@ CODE_OUTPUT = u""".. rst-class:: sphx-glr-script-out
 
 SPHX_GLR_SIG = """\n.. rst-class:: sphx-glr-signature
 
-    `Generated by Sphinx-Gallery <http://sphinx-gallery.readthedocs.io>`_\n"""
+    `Generated by Sphinx-Gallery <http://sphinx-gallery.readthedocs.org>`_\n"""
 
 
 def get_docstring_and_rest(filename):
@@ -239,7 +240,8 @@ def extract_thumbnail_number(text):
     """ Pull out the thumbnail image number specified in the docstring. """
 
     # check whether the user has specified a specific thumbnail image
-    pattr = re.compile("^\s*#\s*sphinx_gallery_thumbnail_number\s*=\s*([0-9]+)\s*$", flags=re.MULTILINE)
+    pattr = re.compile(
+        r"^\s*#\s*sphinx_gallery_thumbnail_number\s*=\s*([0-9]+)\s*$", flags=re.MULTILINE)
     match = pattr.search(text)
 
     if match is None:
@@ -250,6 +252,7 @@ def extract_thumbnail_number(text):
 
     return thumbnail_number
 
+
 def extract_intro(filename):
     """ Extract the first paragraph of module-level docstring. max:95 char"""
 
@@ -284,34 +287,19 @@ def get_md5sum(src_file):
     return src_md5
 
 
-def check_md5sum_change(src_file):
-    """Returns True if src_file has a different md5sum"""
+def md5sum_is_current(src_file):
+    """Returns True if src_file has the same md5 hash as the one stored on disk"""
 
     src_md5 = get_md5sum(src_file)
 
     src_md5_file = src_file + '.md5'
-    src_file_changed = True
     if os.path.exists(src_md5_file):
         with open(src_md5_file, 'r') as file_checksum:
             ref_md5 = file_checksum.read()
-        if src_md5 == ref_md5:
-            src_file_changed = False
-
-    if src_file_changed:
-        with open(src_md5_file, 'w') as file_checksum:
-            file_checksum.write(src_md5)
-
-    return src_file_changed
-
-
-def _plots_are_current(src_file, image_path):
-    """Test existence of image file and no change in md5sum of
-    example"""
 
-    has_image = os.path.exists(image_path)
-    src_file_changed = check_md5sum_change(src_file)
+        return src_md5 == ref_md5
 
-    return has_image and not src_file_changed
+    return False
 
 
 def save_figures(image_path, fig_count, gallery_conf):
@@ -323,10 +311,15 @@ def save_figures(image_path, fig_count, gallery_conf):
         Path where plots are saved (format string which accepts figure number)
     fig_count : int
         Previous figure number count. Figure number add from this number
+    gallery_conf : dict
+        Contains the configuration of Sphinx-Gallery
 
     Returns
     -------
-    list of strings containing the full path to each figure
+    figure_list : list of str
+        strings containing the full path to each figure
+    images_rst : str
+        rst code to embed the images in the document
     """
     figure_list = []
 
@@ -362,7 +355,18 @@ def save_figures(image_path, fig_count, gallery_conf):
             figure_list.append(current_fig)
         mlab.close(all=True)
 
-    return figure_list
+    # Depending on whether we have one or more figures, we're using a
+    # horizontal list or a single rst call to 'image'.
+    images_rst = ""
+    if len(figure_list) == 1:
+        figure_name = figure_list[0]
+        images_rst = SINGLE_IMAGE % figure_name.lstrip('/')
+    elif len(figure_list) > 1:
+        images_rst = HLIST_HEADER
+        for figure_name in figure_list:
+            images_rst += HLIST_IMAGE_TEMPLATE % figure_name.lstrip('/')
+
+    return figure_list, images_rst
 
 
 def scale_image(in_fname, out_fname, max_width, max_height):
@@ -410,17 +414,28 @@ def scale_image(in_fname, out_fname, max_width, max_height):
                           generated images')
 
 
-def save_thumbnail(thumbnail_image_path, base_image_name, gallery_conf):
+def save_thumbnail(image_path_template, src_file, gallery_conf):
     """Save the thumbnail image"""
+    # read specification of the figure to display as thumbnail from main text
+    _, content = get_docstring_and_rest(src_file)
+    thumbnail_number = extract_thumbnail_number(content)
+    thumbnail_image_path = image_path_template.format(thumbnail_number)
+
     thumb_dir = os.path.join(os.path.dirname(thumbnail_image_path), 'thumb')
     if not os.path.exists(thumb_dir):
         os.makedirs(thumb_dir)
 
+    base_image_name = os.path.splitext(os.path.basename(src_file))[0]
     thumb_file = os.path.join(thumb_dir,
                               'sphx_glr_%s_thumb.png' % base_image_name)
 
-    if os.path.exists(thumbnail_image_path):
+    if src_file in gallery_conf['failing_examples']:
+        broken_img = os.path.join(glr_path_static(), 'broken_example.png')
+        scale_image(broken_img, thumb_file, 200, 140)
+
+    elif os.path.exists(thumbnail_image_path):
         scale_image(thumbnail_image_path, thumb_file, 400, 280)
+
     elif not os.path.exists(thumb_file):
         # create something to replace the thumbnail
         default_thumb_file = os.path.join(glr_path_static(), 'no_image.png')
@@ -475,19 +490,21 @@ def generate_dir_rst(src_dir, target_dir, gallery_conf, seen_backrefs):
     return fhindex, computation_times
 
 
-def execute_script(code_block, example_globals, image_path, fig_count,
-                   src_file, gallery_conf):
+def execute_code_block(code_block, example_globals,
+                       block_vars, gallery_conf):
     """Executes the code block of the example file"""
     time_elapsed = 0
     stdout = ''
 
-    # We need to execute the code
-    print('plotting code blocks in %s' % src_file)
+    # If example is not suitable to run, skip executing its blocks
+    if not block_vars['execute_script']:
+        return stdout, time_elapsed
 
     plt.close('all')
     cwd = os.getcwd()
     # Redirect output to stdout and
     orig_stdout = sys.stdout
+    src_file = block_vars['src_file']
 
     try:
         # First cd in the original example dir, so that any file
@@ -510,62 +527,62 @@ def execute_script(code_block, example_globals, image_path, fig_count,
         if my_stdout:
             stdout = CODE_OUTPUT.format(indent(my_stdout, u' ' * 4))
         os.chdir(cwd)
-        figure_list = save_figures(image_path, fig_count, gallery_conf)
-
-        # Depending on whether we have one or more figures, we're using a
-        # horizontal list or a single rst call to 'image'.
-        image_list = ""
-        if len(figure_list) == 1:
-            figure_name = figure_list[0]
-            image_list = SINGLE_IMAGE % figure_name.lstrip('/')
-        elif len(figure_list) > 1:
-            image_list = HLIST_HEADER
-            for figure_name in figure_list:
-                image_list += HLIST_IMAGE_TEMPLATE % figure_name.lstrip('/')
+        fig_list, images_rst = save_figures(
+            block_vars['image_path'], block_vars['fig_count'], gallery_conf)
+        fig_num = len(fig_list)
 
     except Exception:
         formatted_exception = traceback.format_exc()
 
-        sys.stdout = orig_stdout  # need this here so these lines don't bomb
-        print(80 * '_')
-        print('%s is not compiling:' % src_file)
-        print(formatted_exception)
-        print(80 * '_')
-
-        figure_list = []
-        image_list = codestr2rst(formatted_exception, lang='pytb')
+        fail_example_warning = 80 * '_' + '\n' + \
+            '%s failed to execute correctly:' % src_file + \
+            formatted_exception + 80 * '_' + '\n'
+        warnings.warn(fail_example_warning)
 
-        # Overrides the output thumbnail in the gallery for easy identification
-        broken_img = os.path.join(glr_path_static(), 'broken_example.png')
-        shutil.copyfile(broken_img, os.path.join(cwd, image_path.format(1)))
-        fig_count += 1  # raise count to avoid overwriting image
+        fig_num = 0
+        images_rst = codestr2rst(formatted_exception, lang='pytb')
 
         # Breaks build on first example error
         # XXX This check can break during testing e.g. if you uncomment the
         # `raise RuntimeError` by the `my_stdout` call, maybe use `.get()`?
         if gallery_conf['abort_on_example_error']:
             raise
+        # Stores failing file
+        gallery_conf['failing_examples'][src_file] = formatted_exception
+        block_vars['execute_script'] = False
 
     finally:
         os.chdir(cwd)
         sys.stdout = orig_stdout
 
     print(" - time elapsed : %.2g sec" % time_elapsed)
-    code_output = u"\n{0}\n\n{1}\n\n".format(image_list, stdout)
+    code_output = u"\n{0}\n\n{1}\n\n".format(images_rst, stdout)
+    block_vars['fig_count'] += fig_num
 
-    return code_output, time_elapsed, fig_count + len(figure_list)
+    return code_output, time_elapsed
 
 
 def generate_file_rst(fname, target_dir, src_dir, gallery_conf):
-    """ Generate the rst file for a given example.
+    """Generate the rst file for a given example.
 
-        Returns the amout of code (in characters) of the corresponding
-        files.
+    Returns
+    -------
+    amount_of_code : int
+        character count of the corresponding python script in file
+    time_elapsed : float
+        seconds required to run the script
     """
 
     src_file = os.path.join(src_dir, fname)
     example_file = os.path.join(target_dir, fname)
     shutil.copyfile(src_file, example_file)
+    script_blocks = split_code_and_text_blocks(src_file)
+    amount_of_code = sum([len(bcontent)
+                          for blabel, bcontent in script_blocks
+                          if blabel == 'code'])
+
+    if md5sum_is_current(example_file):
+        return amount_of_code, 0
 
     image_dir = os.path.join(target_dir, 'images')
     if not os.path.exists(image_dir):
@@ -575,80 +592,63 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf):
     image_fname = 'sphx_glr_' + base_image_name + '_{0:03}.png'
     image_path_template = os.path.join(image_dir, image_fname)
 
-    script_blocks = split_code_and_text_blocks(example_file)
-
-    # read specification of the figure to display as thumbnail from main text
-    _, content = get_docstring_and_rest(example_file)
-    thumbnail_number = extract_thumbnail_number(content)
-
-    amount_of_code = sum([len(bcontent)
-                          for blabel, bcontent in script_blocks
-                          if blabel == 'code'])
-
-    first_image_path = image_path_template.format(1)
-    if _plots_are_current(example_file, first_image_path):
-        return amount_of_code, 0
-
-    time_elapsed = 0
-
     ref_fname = example_file.replace(os.path.sep, '_')
     example_rst = """\n\n.. _sphx_glr_{0}:\n\n""".format(ref_fname)
     example_nb = Notebook(fname, target_dir)
 
     filename_pattern = gallery_conf.get('filename_pattern')
-    if re.search(filename_pattern, src_file) and gallery_conf['plot_gallery']:
-        example_globals = {
-            # A lot of examples contains 'print(__doc__)' for example in
-            # scikit-learn so that running the example prints some useful
-            # information. Because the docstring has been separated from
-            # the code blocks in sphinx-gallery, __doc__ is actually
-            # __builtin__.__doc__ in the execution context and we do not
-            # want to print it
-            '__doc__': '',
-            # Examples may contain if __name__ == '__main__' guards
-            # for in example scikit-learn if the example uses multiprocessing
-            '__name__': '__main__'}
-
-        fig_count = 0
-        # A simple example has two blocks: one for the
-        # example introduction/explanation and one for the code
-        is_example_notebook_like = len(script_blocks) > 2
-        for blabel, bcontent in script_blocks:
-            if blabel == 'code':
-                code_output, rtime, fig_count = execute_script(bcontent,
-                                                               example_globals,
-                                                               image_path_template,
-                                                               fig_count,
-                                                               src_file,
-                                                               gallery_conf)
-
-                time_elapsed += rtime
-                example_nb.add_code_cell(bcontent)
-
-                if is_example_notebook_like:
-                    example_rst += codestr2rst(bcontent) + '\n'
-                    example_rst += code_output
-                else:
-                    example_rst += code_output
-                    if 'sphx-glr-script-out' in code_output:
-                        # Add some vertical space after output
-                        example_rst += "\n\n|\n\n"
-                    example_rst += codestr2rst(bcontent) + '\n'
-
-            else:
-                example_rst += text2string(bcontent) + '\n'
-                example_nb.add_markdown_cell(text2string(bcontent))
-    else:
-        for blabel, bcontent in script_blocks:
-            if blabel == 'code':
+    execute_script = re.search(filename_pattern, src_file) and gallery_conf[
+        'plot_gallery']
+    example_globals = {
+        # A lot of examples contains 'print(__doc__)' for example in
+        # scikit-learn so that running the example prints some useful
+        # information. Because the docstring has been separated from
+        # the code blocks in sphinx-gallery, __doc__ is actually
+        # __builtin__.__doc__ in the execution context and we do not
+        # want to print it
+        '__doc__': '',
+        # Examples may contain if __name__ == '__main__' guards
+        # for in example scikit-learn if the example uses multiprocessing
+        '__name__': '__main__',
+    }
+
+    # A simple example has two blocks: one for the
+    # example introduction/explanation and one for the code
+    is_example_notebook_like = len(script_blocks) > 2
+    time_elapsed = 0
+    block_vars = {'execute_script': execute_script, 'fig_count': 0,
+                  'image_path': image_path_template, 'src_file': src_file}
+    for blabel, bcontent in script_blocks:
+        if blabel == 'code':
+            code_output, rtime = execute_code_block(bcontent,
+                                                    example_globals,
+                                                    block_vars,
+                                                    gallery_conf)
+
+            time_elapsed += rtime
+            example_nb.add_code_cell(bcontent)
+
+            if is_example_notebook_like:
                 example_rst += codestr2rst(bcontent) + '\n'
-                example_nb.add_code_cell(bcontent)
+                example_rst += code_output
             else:
-                example_rst += bcontent + '\n'
-                example_nb.add_markdown_cell(text2string(bcontent))
+                example_rst += code_output
+                if 'sphx-glr-script-out' in code_output:
+                    # Add some vertical space after output
+                    example_rst += "\n\n|\n\n"
+                example_rst += codestr2rst(bcontent) + '\n'
 
-    thumbnail_image_path = image_path_template.format(thumbnail_number)
-    save_thumbnail(thumbnail_image_path, base_image_name, gallery_conf)
+        else:
+            example_rst += text2string(bcontent) + '\n'
+            example_nb.add_markdown_cell(text2string(bcontent))
+
+    # Writes md5 checksum if example has build correctly
+    # not failed and was initially meant to run(no-plot shall not cache md5sum)
+    if block_vars['execute_script']:
+        with open(example_file + '.md5', 'w') as file_checksum:
+            file_checksum.write(get_md5sum(example_file))
+
+    save_thumbnail(image_path_template, src_file, gallery_conf)
 
     time_m, time_s = divmod(time_elapsed, 60)
     example_nb.save_file()
diff --git a/sphinx_gallery/notebook.py b/sphinx_gallery/notebook.py
index 121e945..c0ee5cd 100644
--- a/sphinx_gallery/notebook.py
+++ b/sphinx_gallery/notebook.py
@@ -4,7 +4,7 @@ r"""
 Parser for Jupyter notebooks
 ============================
 
-Class that holds the Jupyter notebook information
+Class that holds the Ipython notebook information
 
 """
 # Author: Óscar Nájera
@@ -48,7 +48,7 @@ def ipy_notebook_skeleton():
 
 def rst2md(text):
     """Converts the RST text from the examples docstrigs and comments
-    into markdown text for the Jupyter notebooks"""
+    into markdown text for the IPython notebooks"""
 
     top_heading = re.compile(r'^=+$\s^([\w\s-]+)^=+$', flags=re.M)
     text = re.sub(top_heading, r'# \1', text)
@@ -64,7 +64,7 @@ def rst2md(text):
 
 
 class Notebook(object):
-    """Jupyter notebook object
+    """Ipython notebook object
 
     Constructs the file cell-by-cell and writes it at the end"""
 


Test patch:
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index 787ed79..177bb7c 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -8,6 +8,7 @@ from __future__ import (division, absolute_import, print_function,
                         unicode_literals)
 import ast
 import codecs
+import copy
 import json
 import tempfile
 import re
@@ -17,6 +18,7 @@ import shutil
 from nose.tools import assert_equal, assert_false, assert_true
 
 import sphinx_gallery.gen_rst as sg
+from sphinx_gallery import gen_gallery
 from sphinx_gallery import notebook
 import matplotlib.pylab as plt  # Import gen_rst first to enable 'Agg' backend.
 
@@ -119,28 +121,57 @@ def test_md5sums():
         file_md5 = sg.get_md5sum(f.name)
         # verify correct md5sum
         assert_equal('ea8a570e9f3afc0a7c3f2a17a48b8047', file_md5)
-        # True because is a new file
-        assert_true(sg.check_md5sum_change(f.name))
-        # False because file has not changed since last check
-        assert_false(sg.check_md5sum_change(f.name))
-
+        # False because is a new file
+        assert_false(sg.md5sum_is_current(f.name))
+        # Write md5sum to file to check is current
+        with open(f.name + '.md5', 'w') as file_checksum:
+            file_checksum.write(file_md5)
+        assert_true(sg.md5sum_is_current(f.name))
     os.remove(f.name + '.md5')
 
 
+def build_test_configuration(**kwargs):
+    """Sets up a test sphinx-gallery configuration"""
+
+    gallery_conf = copy.deepcopy(gen_gallery.DEFAULT_GALLERY_CONF)
+    gallery_conf.update(examples_dir=tempfile.mkdtemp(),
+                        gallery_dir=tempfile.mkdtemp())
+    gallery_conf.update(kwargs)
+
+    return gallery_conf
+
+
+def test_fail_example():
+    """Test that failing examples are only executed until failing block"""
+
+    gallery_conf = build_test_configuration(filename_pattern='raise.py')
+
+    failing_code = CONTENT + ['#' * 79,
+                              'First_test_fail', '#' * 79, 'second_fail']
+
+    with codecs.open(os.path.join(gallery_conf['examples_dir'], 'raise.py'),
+                     mode='w', encoding='utf-8') as f:
+        f.write('\n'.join(failing_code))
+
+    sg.generate_file_rst('raise.py', gallery_conf['gallery_dir'],
+                         gallery_conf['examples_dir'], gallery_conf)
+
+    # read rst file and check if it contains traceback output
+
+    with codecs.open(os.path.join(gallery_conf['gallery_dir'], 'raise.rst'),
+                     mode='r', encoding='utf-8') as f:
+        ex_failing_blocks = f.read().count('pytb')
+        if ex_failing_blocks == 0:
+            raise ValueError('Did not run into errors in bad code')
+        elif ex_failing_blocks > 1:
+            raise ValueError('Did not stop executing script after error')
+
+
 def test_pattern_matching():
     """Test if only examples matching pattern are executed"""
-    examples_dir = tempfile.mkdtemp()
-    gallery_dir = tempfile.mkdtemp()
-
-    gallery_conf = {
-        'filename_pattern': re.escape(os.sep) + 'plot_0',
-        'examples_dirs': examples_dir,
-        'gallery_dirs': gallery_dir,
-        'plot_gallery': True,
-        'mod_example_dir': 'modules/generated',
-        'doc_module': (),
-        'reference_url': {},
-    }
+
+    gallery_conf = build_test_configuration(
+        filename_pattern=re.escape(os.sep) + 'plot_0')
 
     code_output = ('\n Out::\n'
                    '\n'
@@ -151,18 +182,19 @@ def test_pattern_matching():
     # create three files in tempdir (only one matches the pattern)
     fnames = ['plot_0.py', 'plot_1.py', 'plot_2.py']
     for fname in fnames:
-        with codecs.open(os.path.join(examples_dir, fname), mode='w',
-                         encoding='utf-8') as f:
+        with codecs.open(os.path.join(gallery_conf['examples_dir'], fname),
+                         mode='w', encoding='utf-8') as f:
             f.write('\n'.join(CONTENT))
         # generate rst file
-        sg.generate_file_rst(fname, gallery_dir, examples_dir, gallery_conf)
+        sg.generate_file_rst(fname, gallery_conf['gallery_dir'],
+                             gallery_conf['examples_dir'], gallery_conf)
         # read rst file and check if it contains code output
         rst_fname = os.path.splitext(fname)[0] + '.rst'
-        with codecs.open(os.path.join(gallery_dir, rst_fname),
+        with codecs.open(os.path.join(gallery_conf['gallery_dir'], rst_fname),
                          mode='r', encoding='utf-8') as f:
             rst = f.read()
         if re.search(gallery_conf['filename_pattern'],
-                     os.path.join(gallery_dir, rst_fname)):
+                     os.path.join(gallery_conf['gallery_dir'], rst_fname)):
             assert_true(code_output in rst)
         else:
             assert_false(code_output in rst)
@@ -185,6 +217,7 @@ def test_ipy_notebook():
         f.flush()
         assert_equal(json.load(f), example_nb.work_notebook)
 
+
 def test_thumbnail_number():
     # which plot to show as the thumbnail image
     for test_str in ['# sphinx_gallery_thumbnail_number= 2',
@@ -199,6 +232,7 @@ def test_thumbnail_number():
             thumbnail_number = sg.extract_thumbnail_number(content)
         assert_equal(thumbnail_number, 2)
 
+
 def test_save_figures():
     """Test file naming when saving figures. Requires mayavi."""
     try:
@@ -212,16 +246,21 @@ def test_save_figures():
     mlab.test_plot3d()
     plt.plot(1, 1)
     fname_template = os.path.join(examples_dir, 'image{0}.png')
-    fig_list = sg.save_figures(fname_template, 0, gallery_conf)
+    fig_list, _ = sg.save_figures(fname_template, 0, gallery_conf)
     assert_equal(len(fig_list), 2)
     assert fig_list[0].endswith('image1.png')
     assert fig_list[1].endswith('image2.png')
 
     mlab.test_plot3d()
     plt.plot(1, 1)
-    fig_list = sg.save_figures(fname_template, 2, gallery_conf)
+    fig_list, _ = sg.save_figures(fname_template, 2, gallery_conf)
     assert_equal(len(fig_list), 2)
     assert fig_list[0].endswith('image3.png')
     assert fig_list[1].endswith('image4.png')
 
     shutil.rmtree(examples_dir)
+
+# TODO: test that broken thumbnail does appear when needed
+# TODO: test that examples are not executed twice
+# TODO: test that examples are executed after a no-plot and produce
+#       the correct image in the thumbnail