Repository: sphinx-gallery/sphinx-gallery

Issue:
Support for Plotly
Nothing is much clear about the support of sphinx gallery for plotly. It is clear that it just supports matplotlib, seaborn and mayavi. Perhaps more clearity on is it possible to use sphinx gallery for showing plots on plotly? 

Discussion:
larsoner: @emmanuelle @banesullivan let me know if these docs look good to you:

https://880-25860190-gh.circle-artifacts.com/0/rtd_html/configuration.html#image-scrapers
https://880-25860190-gh.circle-artifacts.com/0/rtd_html/advanced.html#integrate-custom-scrapers-with-sphinx-gallery

Once you've implemented the private interface in the modules in the external list, it would be nice to say they can be used by just doing `'image_scrapers': ['pyvista']` or `['plotly']`. So please feel free to make a PR to SG to add a note about this once it's implemented.
codecov-io: # [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=h1) Report
> Merging [#494](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=desc) into [master](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/commit/8c630d0bbf6b858847162ef5b43376a4dc1796ce?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `92.3%`.

[![Impacted file tree graph](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494/graphs/tree.svg?width=650&token=K4eytvhwU5&height=150&src=pr)](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #494      +/-   ##
==========================================
- Coverage   96.34%   96.31%   -0.04%     
==========================================
  Files          29       29              
  Lines        2681     2711      +30     
==========================================
+ Hits         2583     2611      +28     
- Misses         98      100       +2
```


| [Impacted Files](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sphinx\_gallery/scrapers.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvc2NyYXBlcnMucHk=) | `97.43% <ø> (ø)` | :arrow_up: |
| [sphinx\_gallery/gen\_gallery.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvZ2VuX2dhbGxlcnkucHk=) | `92.3% <100%> (+0.55%)` | :arrow_up: |
| [sphinx\_gallery/gen\_rst.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvZ2VuX3JzdC5weQ==) | `97.7% <50%> (-0.56%)` | :arrow_down: |
| [sphinx\_gallery/tests/test\_scrapers.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvdGVzdHMvdGVzdF9zY3JhcGVycy5weQ==) | `98.56% <95.65%> (-0.61%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=footer). Last update [8c630d0...4882cc3](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/494?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

banesullivan: Looks good to me! I just added `_get_sg_image_scraper` to the master branch of PyVista so this will be live on our next release!
larsoner: Cool. @choldgraf you okay with the updated docs and functionality? Here is what `pyvista` did to make it work (assuming @banesullivan tested it locally and it did indeed work):

https://github.com/pyvista/pyvista/blob/master/pyvista/__init__.py#L34
https://github.com/pyvista/pyvista/blob/master/pyvista/sphinx_gallery.py#L10-L18
banesullivan: > (assuming @banesullivan tested it locally and it did indeed work)

Tested in d474c94
larsoner: I meant with the nicer syntax `'image_scrapers': ['pyvista']` (while on this branch)
emmanuelle: Thanks @larsoner ! Looks good to me.
banesullivan: > I meant with the nicer syntax 'image_scrapers': ['pyvista'] (while on this branch)

Tested and working on a separate branch shown in my last linked commit ^^^ https://travis-ci.org/pyvista/pyvista/builds/534954544

Patch:
diff --git a/doc/advanced.rst b/doc/advanced.rst
index db6b69e..0e912be 100644
--- a/doc/advanced.rst
+++ b/doc/advanced.rst
@@ -7,6 +7,10 @@ Advanced usage
 This page contains more advanced topics in case you want to understand how
 to use Sphinx-Gallery more deeply.
 
+.. contents:: **Contents**
+    :local:
+    :depth: 2
+
 Extend your Makefile for Sphinx-Gallery
 =======================================
 
@@ -62,8 +66,8 @@ and its thumbnail is ``sphx_glr_plot_gallery_version_thumb.png``
 
 .. _custom_scraper:
 
-Writing a custom image scraper
-==============================
+Write a custom image scraper
+============================
 
 .. warning:: The API for custom scrapers is currently experimental.
 
@@ -210,19 +214,39 @@ output formats. To use SVG, you can do::
 You can also use different formats on a per-image basis, but this requires
 writing a customized scraper class or function.
 
-Contributing scrapers back to Sphinx-gallery
---------------------------------------------
+Integrate custom scrapers with Sphinx-gallery
+---------------------------------------------
+
+Sphinx-gallery plans to internally maintain only two scrapers: matplotlib and
+mayavi. If you have extended or fixed bugs with these scrapers, we welcome PRs
+to improve them!
+
+On the other hand, if you have developed a custom scraper for a different
+plotting library that would be useful to the broader community, we encourage
+you to get it working with Sphinx-gallery and then maintain it externally
+(probably in the package that it scrapes), and then integrate and advertise
+it with Sphinx-gallery. You can:
+
+1. Contribute it to the list of externally supported scrapers located in
+   :ref:`reset_modules`.
+2. Optional: add a custom hook to your module root to simplify scraper use.
+   Taking PyVista as an example, adding ``pyvista._get_sg_image_scraper()``
+   that returns the ``callable`` scraper to be used by Sphinx-gallery allows
+   PyVista users to just use strings as they already can for
+   ``'matplotlib'`` and ``'mayavi'``::
+
+       sphinx_gallery_conf = {
+           ...
+           'image_scrapers': ('pyvista',)
+       }
 
-If you've developed a custom scraper for Sphinx-gallery that would be useful
-to the broader community, we encourage you to contribute it to the list of
-natively-supported scrapers located in
-`the scrapers module <https://github.com/sphinx-gallery/sphinx-gallery/blob/master/sphinx_gallery/scrapers.py>`_.
-We welcome PRs!
+   Sphinx-gallery will look for this custom function and call it to get the
+   PyVista image scraper to use before running any examples.
 
 .. _custom_reset:
 
-Defining resetting behavior for custom visualization libraries
---------------------------------------------------------------
+Define resetting behavior (e.g., for custom libraries)
+======================================================
 
 Sphinx-gallery natively supports resetting ``matplotlib`` and ``seaborn``.
 However, if you'd like to support resetting for other libraries (or would like
diff --git a/doc/configuration.rst b/doc/configuration.rst
index fdae0a3..d024880 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -702,55 +702,38 @@ Image scrapers
 Image scrapers are plugins that allow Sphinx-gallery to detect images produced
 during execution of your examples, and then embed them into documentation.
 Scrapers can be activated by appending scraper names to the ``image_scrapers``
-field of your Sphinx-gallery configuration (see below). There are currently
-two native image scrapers in Sphinx-gallery (Matplotlib and Mayavi).
+field of your Sphinx-gallery configuration:
 
-By default, Sphinx-gallery will only detect new :mod:`matplotlib.pyplot`
-figures. This behavior is equivalent to the default of::
+- By default, Sphinx-gallery will only detect new :mod:`matplotlib.pyplot`
+  figures. This behavior is equivalent to the default of::
 
     sphinx_gallery_conf = {
         ...
         'image_scrapers': ('matplotlib',),
     }
 
-Built-in support is also provided for finding :mod:`Mayavi <mayavi.mlab>`
-figures. Enable this feature with the following configuration::
+- Built-in support is also provided for finding :mod:`Mayavi <mayavi.mlab>`
+  figures. Enable this feature with the following configuration::
 
    sphinx_gallery_conf = {
        ...
        'image_scrapers': ('matplotlib', 'mayavi'),
    }
 
-.. note:: The parameter ``find_mayavi_figures`` which can also be used to
-          extract Mayavi figures is **deprecated** in version 0.2+,
-          and will be removed in a future release.
+- Some external packages maintain their own Sphinx-Gallery image
+  scrapers. If you are using these projects, then it is possible to
+  leverage their scrapers with Sphinx-Gallery. To do so, view their
+  documentation on how to integrate with Sphinx-gallery via the
+  ``'image_scrapers'`` configuration value:
 
+  * `PyVista <https://github.com/pyvista/pyvista>`_
+  * `PyGMT <https://github.com/GenericMappingTools/pygmt>`_
 
-External scrapers
-^^^^^^^^^^^^^^^^^
-
-Several external packages maintain their own Sphinx-Gallery image 
-scrapers. If you are using these projects, then it is possible to 
-leverage their scrapers with Sphinx-Gallery. These packages include:
-
-* `PyVista <https://github.com/pyvista/pyvista>`_
-* `PyGMT <https://github.com/GenericMappingTools/pygmt>`_
-
-
-Custom scrapers
-^^^^^^^^^^^^^^^
-
-It is possible to write custom scrapers for images generated by packages
-outside of those supported natively in Sphinx-gallery. This is accomplished
-by writing your own Python function to define how to detect and retrieve
-images produced by an arbitrary package. For instructions on how to accomplish
-this, see :ref:`custom_scraper`.
-
-.. note:: If you've developed a custom scraper for Sphinx-gallery that would
-          be useful to the broader community, we encourage you to contribute
-          it to the list of natively-supported scrapers located in
-          `the scrapers module <https://github.com/sphinx-gallery/sphinx-gallery/blob/master/sphinx_gallery/scrapers.py>`_.
-          We welcome PRs!
+- It is possible to write custom scrapers for images generated by packages
+  outside of those supported natively in Sphinx-gallery. This is accomplished
+  by writing your own Python function to define how to detect and retrieve
+  images produced by an arbitrary package. For instructions, see
+  :ref:`custom_scraper`.
 
 .. _reset_modules:
 
diff --git a/examples/plot_unicode_everywhere.py b/examples/plot_unicode_everywhere.py
index 26158c5..a739c6c 100644
--- a/examples/plot_unicode_everywhere.py
+++ b/examples/plot_unicode_everywhere.py
@@ -25,6 +25,7 @@ s = np.random.rand(*x.shape) * 800 + 500
 plt.scatter(x, y, s, marker=r'$\oint$')
 x = np.random.randn(60) * 7 - 4
 y = np.random.randn(60) * 3 - 2
+s = s[:x.size]
 plt.scatter(x, y, s, alpha=0.5, c='g', marker=r'$\clubsuit$')
 plt.xlabel('⇒')
 plt.ylabel('⇒')
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index 1a4ab02..42cc3d3 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -14,6 +14,7 @@ from __future__ import division, print_function, absolute_import
 import codecs
 import copy
 from datetime import timedelta, datetime
+from importlib import import_module
 import re
 import os
 from xml.sax.saxutils import quoteattr, escape
@@ -145,10 +146,19 @@ def _complete_gallery_conf(sphinx_gallery_conf, src_dir, plot_gallery,
     scrapers = list(scrapers)
     for si, scraper in enumerate(scrapers):
         if isinstance(scraper, basestring):
-            if scraper not in _scraper_dict:
-                raise ValueError('Unknown image scraper named %r' % (scraper,))
-            scrapers[si] = _scraper_dict[scraper]
-        elif not callable(scraper):
+            if scraper in _scraper_dict:
+                scraper = _scraper_dict[scraper]
+            else:
+                orig_scraper = scraper
+                try:
+                    scraper = import_module(scraper)
+                    scraper = getattr(scraper, '_get_sg_image_scraper')
+                    scraper = scraper()
+                except Exception as exp:
+                    raise ValueError('Unknown image scraper %r, got:\n%s'
+                                     % (orig_scraper, exp))
+            scrapers[si] = scraper
+        if not callable(scraper):
             raise ValueError('Scraper %r was not callable' % (scraper,))
     gallery_conf['image_scrapers'] = tuple(scrapers)
     del scrapers
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index 80160dd..1493b21 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -396,7 +396,10 @@ def _memory_usage(func, gallery_conf):
         assert callable(func)
         mem, out = memory_usage(func, max_usage=True, retval=True,
                                 multiprocess=True)
-        mem = mem[0]
+        try:
+            mem = mem[0]  # old MP always returned a list
+        except TypeError:  # 'float' object is not subscriptable
+            pass
     else:
         out = func()
         mem = 0
diff --git a/sphinx_gallery/scrapers.py b/sphinx_gallery/scrapers.py
index b0840b3..d43147b 100644
--- a/sphinx_gallery/scrapers.py
+++ b/sphinx_gallery/scrapers.py
@@ -6,6 +6,9 @@ Scrapers for embedding images
 =============================
 
 Collect images that have been produced by code blocks.
+
+The only scrapers we support are Matplotlib and Mayavi, others should
+live in modules that will support them (e.g., PyVista, Plotly).
 """
 
 import os


Test patch:
diff --git a/sphinx_gallery/tests/test_scrapers.py b/sphinx_gallery/tests/test_scrapers.py
index 1d2f212..5028c4d 100644
--- a/sphinx_gallery/tests/test_scrapers.py
+++ b/sphinx_gallery/tests/test_scrapers.py
@@ -4,6 +4,7 @@ import pytest
 import numpy as np
 from PIL import Image
 
+import sphinx_gallery
 from sphinx_gallery.gen_gallery import _complete_gallery_conf
 from sphinx_gallery.scrapers import (figure_rst, mayavi_scraper, SINGLE_IMAGE,
                                      matplotlib_scraper, ImagePathIterator,
@@ -30,7 +31,7 @@ class matplotlib_svg_scraper():
 
 @pytest.mark.parametrize('ext', ('png', 'svg'))
 def test_save_matplotlib_figures(gallery_conf, ext):
-    """Test matplotlib figure save"""
+    """Test matplotlib figure save."""
     if ext == 'svg':
         gallery_conf['image_scrapers'] = (matplotlib_svg_scraper(),)
     import matplotlib.pyplot as plt  # nest these so that Agg can be set
@@ -115,16 +116,30 @@ def test_save_mayavi_figures(gallery_conf):
         pixels = np.asarray(img.convert("RGB"))
     assert (pixels == [255, 245, 240]).all()
 
+
+def _custom_func(x, y, z):
+    return ''
+
+
+def test_custom_scraper(gallery_conf, monkeypatch):
+    """Test custom scrapers."""
     # custom finders
-    gallery_conf.update(image_scrapers=[lambda x, y, z: ''])
-    image_rst = save_figures(block, block_vars, gallery_conf)
-    assert len(image_path_iterator) == 3
+    with monkeypatch.context() as m:
+        m.setattr(sphinx_gallery, '_get_sg_image_scraper',
+                  lambda: _custom_func, raising=False)
+        for cust in (_custom_func, 'sphinx_gallery'):
+            gallery_conf.update(image_scrapers=[cust])
+            fname_template = os.path.join(gallery_conf['gallery_dir'],
+                                          'image{0}.png')
+            image_path_iterator = ImagePathIterator(fname_template)
+            block = ('',) * 3
+            block_vars = dict(image_path_iterator=image_path_iterator)
 
     # degenerate
     gallery_conf.update(image_scrapers=['foo'])
+    complete_args = (gallery_conf, gallery_conf['gallery_dir'], True, False)
     with pytest.raises(ValueError, match='Unknown image scraper'):
-        _complete_gallery_conf(
-            gallery_conf, gallery_conf['gallery_dir'], True, False)
+        _complete_gallery_conf(*complete_args)
     gallery_conf.update(
         image_scrapers=[lambda x, y, z: y['image_path_iterator'].next()])
     with pytest.raises(RuntimeError, match='did not produce expected image'):
@@ -132,6 +147,18 @@ def test_save_mayavi_figures(gallery_conf):
     gallery_conf.update(image_scrapers=[lambda x, y, z: 1.])
     with pytest.raises(TypeError, match='was not a string'):
         save_figures(block, block_vars, gallery_conf)
+    # degenerate string interface
+    gallery_conf.update(image_scrapers=['sphinx_gallery'])
+    with monkeypatch.context() as m:
+        m.setattr(sphinx_gallery, '_get_sg_image_scraper', 'foo',
+                  raising=False)
+        with pytest.raises(ValueError, match='^Unknown image.*\n.*callable'):
+            _complete_gallery_conf(*complete_args)
+    with monkeypatch.context() as m:
+        m.setattr(sphinx_gallery, '_get_sg_image_scraper', lambda: 'foo',
+                  raising=False)
+        with pytest.raises(ValueError, match='^Scraper.*was not callable'):
+            _complete_gallery_conf(*complete_args)
 
 
 @pytest.mark.parametrize('ext', _KNOWN_IMG_EXTS)