Repository: bokeh/bokeh

Issue:
sphinxext.bokeh_plot broken
#### ALL software version info (bokeh, python, notebook, OS, browser, any other relevant pacakges)

This was the result of `conda create -n minimal python=3.5 sphinx bokeh` today on OS X 10.11.6, replicated also on another mac running 10.11. Full result of `conda list` below in case it's helpful. Note also that @handwerkerd verified also not working with latext Sphinx 1.5.3.

```
alabaster                 0.7.10                   py35_0  
babel                     2.3.4                    py35_0  
bokeh                     0.12.4                   py35_0  
docutils                  0.13.1                   py35_0  
imagesize                 0.7.1                    py35_0  
jinja2                    2.9.5                    py35_0  
markupsafe                0.23                     py35_2  
mkl                       2017.0.1                      0  
numpy                     1.12.0                   py35_0  
openssl                   1.0.2k                        0  
pip                       9.0.1                    py35_1  
pygments                  2.2.0                    py35_0  
python                    3.5.3                         0  
python-dateutil           2.6.0                    py35_0  
pytz                      2016.10                  py35_0  
pyyaml                    3.12                     py35_0  
readline                  6.2                           2  
requests                  2.13.0                   py35_0  
setuptools                27.2.0                   py35_0  
six                       1.10.0                   py35_0  
snowballstemmer           1.2.1                    py35_0  
sphinx                    1.5.1                    py35_0  
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
tornado                   4.4.2                    py35_0  
wheel                     0.29.0                   py35_0  
xz                        5.2.2                         1  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3  
```

#### Description of expected behavior and the observed behavior

Clone and build [this](https://github.com/davclark/simple-bokeh-sphinx) with `make html`. We should get a simple sphinx doc with a simple bokeh plot. As I note in the index.rst, I grabbed the bokeh plot code directly from the docs in `bokeh/sphinxext/bokeh_plot.py`.

#### Complete, minimal, self-contained example code that reproduces the issue

https://github.com/davclark/simple-bokeh-sphinx/tree/broken

#### Stack traceback and/or browser JavaScript console output

Trimmed error to console:

```
reading sources... [ 50%] conf                                                                        
Exception occurred:
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/bokeh/sphinxext/bokeh_plot.py", line 125, in _process_script
    js, script = autoload_static(d.roots[0], resources, script_path)
IndexError: list index out of range
The full traceback has been saved in /var/folders/longhashystring.log, if you want to report the issue to the developers.
```

I'll put `/var/folders/longhashystring.log` in a subsequent note.

#### Screenshots or screencasts of the bug in action

Not relevant.


Discussion:
`/var/folders/longhashystring.log`:

```
# Sphinx version: 1.5.1
# Python version: 3.5.3 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.5
# Last messages:
#   Running Sphinx v1.5.1
#   loading pickled environment...
#   not yet created
#   building [mo]: targets for 0 po files that are out of date
#   building [html]: targets for 2 source files that are out of date
#   updating environment:
#   2 added, 0 changed, 0 removed
#   reading sources... [ 50%] conf
# Loaded extensions:
#   sphinx.builders.applehelp (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/applehelp.py
#   sphinx.domains.javascript (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/javascript.py
#   sphinx.builders.devhelp (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/devhelp.py
#   sphinx.builders.epub3 (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/epub3.py
#   sphinx.directives (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/directives/__init__.py
#   sphinx.roles (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/roles.py
#   sphinx.builders.gettext (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/gettext.py
#   sphinx.domains.std (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/std.py
#   sphinx.builders.latex (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/latex.py
#   sphinx.domains.rst (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/rst.py
#   sphinx.builders.linkcheck (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/linkcheck.py
#   sphinx.builders.texinfo (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/texinfo.py
#   sphinx.builders.xml (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/xml.py
#   sphinx.domains.cpp (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/cpp.py
#   sphinx.builders.htmlhelp (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/htmlhelp.py
#   sphinx.builders.manpage (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/manpage.py
#   sphinx.directives.patches (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/directives/patches.py
#   sphinx.directives.other (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/directives/other.py
#   bokeh.sphinxext.bokeh_plot (unknown version) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/bokeh/sphinxext/bokeh_plot.py
#   sphinx.builders.dummy (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/dummy.py
#   sphinx.directives.code (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/directives/code.py
#   sphinx.builders.text (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/text.py
#   sphinx.builders.qthelp (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/qthelp.py
#   sphinx.builders.epub (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/epub.py
#   alabaster (0.7.10) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/alabaster/__init__.py
#   sphinx.builders.websupport (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/websupport.py
#   sphinx.builders.html (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/html.py
#   sphinx.builders.changes (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/changes.py
#   sphinx.domains.c (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/c.py
#   sphinx.domains.python (builtin) from /Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/domains/python.py
Traceback (most recent call last):
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/application.py", line 333, in build
    self.builder.build_update()
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/builders/__init__.py", line 265, in build
    self.doctreedir, self.app))
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/environment/__init__.py", line 556, in update
    self._read_serial(docnames, app)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/environment/__init__.py", line 576, in _read_serial
    self.read_doc(docname, app)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/environment/__init__.py", line 684, in read_doc
    pub.publish()
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/Sphinx-1.5.1-py3.5.egg/sphinx/io.py", line 55, in read
    self.parse()
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/bokeh/sphinxext/bokeh_plot.py", line 164, in parse
    (script, js, js_path, source) = _process_script(source, filename, env.bokeh_plot_auxdir, js_name)
  File "/Users/clarkda/miniconda3/envs/minimal/lib/python3.5/site-packages/bokeh/sphinxext/bokeh_plot.py", line 125, in _process_script
    js, script = autoload_static(d.roots[0], resources, script_path)
IndexError: list index out of range
```

So, in a recent release a new `Parser` was added to parse `.py` files in the source tree. The greatly streamlined the code for generating the few hundred live examples that the Bokeh docs have. The unintended consequence here is that since your source tree is not in a subdir, the directive is parsing and executing your `conf.py` file, and this is what is causing it to fall over. (Executing `conf.py` does not produce any bokeh objects to pass to `autoload_static`, resulting in the index error)

 I will confess that the tool is tailored towards building the Bokeh docs. You're the first other people I am aware of using `bokeh.sphinx_ext` at all, but I'd certainly like for it to be usable more generally. There's a few ideas offhand:
* a (possibly configurable) blacklist, so that things like `conf.py` are ignored
* a configurable whitelist of directory prefixes for executing `.py` examples
* a special comment required to be in`.py` files or else they are ignored 
* a mode that turns off `.py` parsing altogether. 

I'm inclined to think the latter is definitely advised, and in particular make it default to off.  I wrote the new version of the tool with the expectation that all `.py` files under the sphinx docs are bokeh examples and would be executed and example pages generated for them. But if there are arbitrary scripts, and that is not expected then I could imagine it possibly resulting in unpleasant surprises. The second and third options also seem possibly useful in conjunction. 

In the short term. I think your best bet is to configure a sphinx project that has the source under a subdirectory (like the Bokeh docs, e.g.). That would solve the immediate problem and if I may offer advice from experience, that's a preferable configuration anyway. 
Also: Thank you *very much* for the detailed and comprehensive issue. 
Thank you for your detailed reply. We're realizing that it might be easier to create stand alone html or javascript files with bokeh and then integrate that data into a larger document with sphinx or some other program.
That said, focusing on your solution to this bug, I'm a sphinx amateur, but the conf.py file is necessary for sphinx and probably should never be parsed by Bokeh. For making the Bokeh docs, I see you you have fabfile.py in bokeh/sphinx which includes "sys.path.append("source")" to add the location of conf.py to the path. Since fabfile.py is in the sphinx directory, I don't see why it's not being read by the parser and causing an exception. Did you put that file on a blacklist somewhere? Could you or we add conf.py to a similar blacklist?
> That said, focusing on your solution to this bug, I'm a sphinx amateur, but the conf.py file is necessary for sphinx and probably should never be parsed by Bokeh. 

I certainly agree with this, and it was certainly not my intent to execute `conf.py`. In fact, the reason  why  `conf.py` is not considered when the Bokeh docs are run, is because it is explicitly excluded in our `conf.py`:
```
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
#
# NOTE: in these docs all .py script are assumed to be bokeh plot scripts
exclude_patterns = ['conf.py']
```

 I do think the directive needs to be made more configurable and robust to support better general use, but if you add this to your `conf.py` that should get around the immediate problem.


I hadn't noticed the exclude_patterns variable. Adding conf.py there solved the acute problem.
Thank you!
So there are multiple ways this can be fixed. Excluding `conf.py` seems like the easiest fix. I've fixed the master branch in that way, and left a "broken" tag (link updated above) for testing against potential updates to Bokeh. I also made an [example branch](https://github.com/davclark/simple-bokeh-sphinx/tree/components) using autoload_static that is (I think) about as simple as you could do to replace the need for the sphinx extension.

This approach seems more brittle, as there are now 3 places you need to coordinate code:

- The path of the javascript in the plot generation code (bokeh/bokeh_files.py)
- html_extra_path in conf.py to copy the above path (stripping directory structure)
- a `raw:: html` include of the tag that loads the js file in your rst for sphinx

To expand on the use-case, we are at an NIH hackathon working on creating interactive static reports where we can explore brain imaging data using linked plots. So target users are non-software-engineer scientists, and minimizing points for breakage would be great. For this kind of demographic, I think: 

-  the bokeh sphinx extension is useful for other people
- "blacklisting" can already be handled by excluding in conf.py. This should just be documented. 
  - If you'd like, I can do a pull request, though I warn you I'm not super-speedy since I often have pressures pushing me away from OSS contributions! 
  - Default special-case exclusion of conf.py also seems reasonable (though honestly, this would make sense for sphinx in general...).
- the auto-scan may sometimes be useful, though perhaps it could be off by default
- and then if auto-scan is disabled, there should be a whitelist. I guess in conf.py also?
I'm another user that would like to use `bokeh.sphinxext.bokeh_plot` in my Sphinx docs and ran into what I believe is the same issue reported here: https://travis-ci.org/gammapy/gamma-cat/builds/225501459#L751

I didn't read through all the discussion above, but I gather the issue is that by default importing this does a lot of magic stuff? Wouldn't it be better to remove that complete in favour of a declarative scheme listing what to process, or making the "auto-discovery" that you're using opt-in?

Resolving patch:
diff --git a/bokeh/sphinxext/bokeh_plot.py b/bokeh/sphinxext/bokeh_plot.py
--- a/bokeh/sphinxext/bokeh_plot.py
+++ b/bokeh/sphinxext/bokeh_plot.py
@@ -9,6 +9,10 @@
 
     .. bokeh-plot:: path/to/plot.py
 
+.. note::
+    .py scripts are not scanned automatically! In order to include
+    certain directories into .py scanning process use following directive
+    in sphinx conf.py file: bokeh_plot_pyfile_include_dirs = ["dir1","dir2"]
 
 **Inline code** as the content of the directive::
 
@@ -249,6 +253,10 @@ def run(self):
 
 def env_before_read_docs(app, env, docnames):
     docnames.sort(key=lambda x: 2 if "extension" in x else 0 if "examples" in x else 1)
+    for name in [x for x in docnames if env.doc2path(x).endswith(".py")]:
+        if not name.startswith(tuple(env.app.config.bokeh_plot_pyfile_include_dirs)):
+            env.found_docs.remove(name)
+            docnames.remove(name)
 
 def builder_inited(app):
     app.env.bokeh_plot_auxdir = join(app.env.doctreedir, 'bokeh_plot')
@@ -293,6 +301,9 @@ def env_purge_doc(app, env, docname):
         del env.bokeh_plot_files[docname]
 
 def setup(app):
+    """ sphinx config variable to scan .py files in provided directories only """
+    app.add_config_value('bokeh_plot_pyfile_include_dirs', [], 'html')
+
     app.add_source_parser('.py', PlotScriptParser)
 
     app.add_directive('bokeh-plot', BokehPlotDirective)
diff --git a/sphinx/source/conf.py b/sphinx/source/conf.py
--- a/sphinx/source/conf.py
+++ b/sphinx/source/conf.py
@@ -99,8 +99,9 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 #
-# NOTE: in these docs all .py script are assumed to be bokeh plot scripts
-exclude_patterns = ['conf.py']
+# NOTE: in these docs all .py script are assumed to be bokeh plot scripts!
+# with bokeh_plot_pyfile_include_dirs set desired folder to look for .py files
+bokeh_plot_pyfile_include_dirs = ['docs']
 
 # The reST default role (used for this markup: `text`) to use for all documents.
 #default_role = None


Test patch: