Repository: sphinx-gallery/sphinx-gallery

Issue:
Strip in-file sphinx_gallery directives from code
The in-file comment `# sphinx_gallery_line_numbers` and `# sphinx_gallery_thumbnail_number` are meant to control sphinx gallery behavior. They should not appear in the generated example code, as they do not have any meaning for the reader of the generated example.

I have some proof-of-concept code for this but before I poish it up for a PR, I would like to know if this is a desired feature.

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=h1) Report
> Merging [#487](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=desc) into [master](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/commit/40d7e652f4bd5be7b5823c2cb8ac881e79af6b50?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `90.9%`.

[![Impacted file tree graph](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487/graphs/tree.svg?width=650&token=K4eytvhwU5&height=150&src=pr)](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #487      +/-   ##
==========================================
- Coverage   96.31%   96.29%   -0.03%     
==========================================
  Files          29       29              
  Lines        2662     2671       +9     
==========================================
+ Hits         2564     2572       +8     
- Misses         98       99       +1
```


| [Impacted Files](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sphinx\_gallery/gen\_gallery.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvZ2VuX2dhbGxlcnkucHk=) | `91.75% <ø> (ø)` | :arrow_up: |
| [sphinx\_gallery/py\_source\_parser.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvcHlfc291cmNlX3BhcnNlci5weQ==) | `95.34% <100%> (+0.16%)` | :arrow_up: |
| [sphinx\_gallery/tests/test\_py\_source\_parser.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvdGVzdHMvdGVzdF9weV9zb3VyY2VfcGFyc2VyLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [sphinx\_gallery/gen\_rst.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvZ2VuX3JzdC5weQ==) | `97.97% <50%> (-0.28%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=footer). Last update [40d7e65...495895c](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/487?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

larsoner: @drammock can you review and/or try on your MNE branch?
drammock: hmm... it doesn't seem to be working for me. Here's what I did:

- added @timhoffm's fork as remote
- created and checked out new branch that's tracking `timhoffm:strip-infile-config`
- ran `pip install --upgrade -e .` from the sphinx-gallery dir
- added `remove_config_comments = True` to `conf.py`
- ran the doc build; the comment `# sphinx_gallery_thumbnail_number = 13` is still there

On monday I'll have time to test further and review.

larsoner: You can try running `make clean` to ensure it rebuilds the example, but probably did
drammock: Yup, I did run `make clean` first.

On May 11, 2019 8:48:45 PM AKDT, Eric Larson <notifications@github.com> wrote:
>You can try running `make clean` to ensure it rebuilds the example, but
>probably did

timhoffm: Works here locally. Also I've added another test that checks that the comment is or is not part of the generated .rst file depending on the setting.

The only thing I could imaging going wrong is that the pattern is not correctly matched for a specific case. @drammock can you please provide an example file (or code segment) from which the comment was not removed?

Patch:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 582dd66..b5cdb10 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -26,6 +26,7 @@ file:
 - ``default_thumb_file`` (:ref:`custom_default_thumb`)
 - ``thumbnail_size`` (:ref:`setting_thumbnail_size`)
 - ``line_numbers`` (:ref:`adding_line_numbers`)
+- ``remove_config_comments`` (:ref:`removing_config_comments`)
 - ``download_all_examples`` (:ref:`disable_all_scripts_download`)
 - ``plot_gallery`` (:ref:`without_execution`)
 - ``image_scrapers`` (and the deprecated ``find_mayavi_figures``)
@@ -45,6 +46,9 @@ Some options can also be set or overridden on a file-by-file basis:
 - ``# sphinx_gallery_line_numbers`` (:ref:`adding_line_numbers`)
 - ``# sphinx_gallery_thumbnail_number`` (:ref:`choosing_thumbnail`)
 
+See also :ref:`removing_config_comments` to hide these comments from the
+rendered examples.
+
 Some options can be set during the build execution step, e.g. using a Makefile:
 
 - ``make html-noplot`` (:ref:`without_execution`)
@@ -428,6 +432,22 @@ setting::
 Note that for Sphinx < 1.3, the line numbers will not be consistent with the
 original file.
 
+.. _removing_config_comments
+
+Removing config comments
+========================
+
+Some configurations can be done on a file-by-file basis by adding a special
+comment with the pattern :samp:`# sphinx_gallery_{config} = {value}` to the
+example source files. By default, the source files are parsed as is and thus
+the comment will appear in the example.
+
+To remove the comment from the rendered example set the option::
+
+    sphinx_gallery_conf = {
+        ...
+        'remove_config_comments': True,
+    }
 
 .. _first_notebook_cell:
 
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index 62d3ed1..1a4ab02 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -70,6 +70,7 @@ DEFAULT_GALLERY_CONF = {
     'image_scrapers': ('matplotlib',),
     'reset_modules': ('matplotlib', 'seaborn'),
     'first_notebook_cell': '%matplotlib inline',
+    'remove_config_comments': False,
     'show_memory': False,
     'junit': '',
     'log_level': {'backreference_missing': 'warning'},
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index db82ea3..80160dd 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -61,7 +61,7 @@ from . import sphinx_compatibility
 from .backreferences import write_backreferences, _thumbnail_div
 from .downloads import CODE_DOWNLOAD
 from .py_source_parser import (split_code_and_text_blocks,
-                               get_docstring_and_rest)
+                               get_docstring_and_rest, remove_config_comments)
 
 from .notebook import jupyter_notebook, save_notebook
 from .binder import check_binder_conf, gen_binder_rst
@@ -642,6 +642,14 @@ def generate_file_rst(fname, target_dir, src_dir, gallery_conf):
         'target_file': target_file}
 
     file_conf, script_blocks = split_code_and_text_blocks(src_file)
+
+    if gallery_conf['remove_config_comments']:
+        script_blocks = [
+            (label, remove_config_comments(content), line_number)
+            for label, content, line_number in script_blocks
+        ]
+
+
     output_blocks, time_elapsed = execute_script(script_blocks,
                                                  script_vars,
                                                  gallery_conf)
diff --git a/sphinx_gallery/py_source_parser.py b/sphinx_gallery/py_source_parser.py
index 66536f1..94fe085 100644
--- a/sphinx_gallery/py_source_parser.py
+++ b/sphinx_gallery/py_source_parser.py
@@ -26,6 +26,20 @@ SyntaxError
 Example script with invalid Python syntax
 """
 
+# The pattern for in-file config comments is designed to not greedily match
+# newlines at the start and end, except for one newline at the end. This
+# ensures that the matched pattern can be removed from the code without
+# changing the block structure; i.e. empty newlines are preserved, e.g. in
+#
+#     a = 1
+#
+#     # sphinx_gallery_thumbnail_number = 2
+#
+#     b = 2
+INFILE_CONFIG_PATTERN = re.compile(
+    r"^[\ \t]*#\s*sphinx_gallery_([A-Za-z0-9_]+)\s*=\s*(.+)[\ \t]*\n?",
+    re.MULTILINE)
+
 
 def parse_source_file(filename):
     """Parse source file into AST node
@@ -125,12 +139,8 @@ def extract_file_config(content):
     """
     Pull out the file-specific config specified in the docstring.
     """
-
-    prop_pat = re.compile(
-        r"^\s*#\s*sphinx_gallery_([A-Za-z0-9_]+)\s*=\s*(.+)\s*$",
-        re.MULTILINE)
     file_conf = {}
-    for match in re.finditer(prop_pat, content):
+    for match in re.finditer(INFILE_CONFIG_PATTERN, content):
         name = match.group(1)
         value = match.group(2)
         try:
@@ -188,3 +198,19 @@ def split_code_and_text_blocks(source_file):
         blocks.append(('code', remaining_content, lineno))
 
     return file_conf, blocks
+
+
+def remove_config_comments(code_block):
+    """
+    Return the content of *code_block* with in-file config comments removed.
+
+    Comment lines of the pattern '# sphinx_gallery_[option] = [val]' are
+    removed, but surrounding empty lines are preserved.
+
+    Parameters
+    ----------
+    code_block : str
+        A code segment.
+    """
+    parsed_code, _ = re.subn(INFILE_CONFIG_PATTERN, '', code_block)
+    return parsed_code


Test patch:
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index 048b613..899a72e 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -36,6 +36,7 @@ CONTENT = [
     'And this is a second paragraph',
     '"""',
     '',
+    '# sphinx_gallery_thumbnail_number = 1'
     '# and now comes the module code',
     'import logging',
     'import sys',
@@ -247,6 +248,53 @@ def test_fail_example(gallery_conf, log_collector):
             raise ValueError('Did not stop executing script after error')
 
 
+def _generate_rst(gallery_conf, fname, content):
+    """
+    Helper function returning the rST text of a given example content.
+
+    This writes a file gallery_conf['examples_dir']/fname with *content*,
+    creates the corresponding rst file by running generate_file_rst() and
+    returns the generated rST code.
+
+    Parameters
+    ----------
+    gallery_conf
+        A gallery_conf as created by the gallery_conf fixture.
+    fname : str
+        A filename; e.g. 'test.py'. This is relative to
+        gallery_conf['examples_dir']
+    content : str
+        The content of fname.
+
+    Returns
+    -------
+    rst : str
+        The generated rST code.
+    """
+    with codecs.open(os.path.join(gallery_conf['examples_dir'], fname),
+                     mode='w', encoding='utf-8') as f:
+        f.write('\n'.join(content))
+    # generate rst file
+    sg.generate_file_rst(fname, gallery_conf['gallery_dir'],
+                         gallery_conf['examples_dir'], gallery_conf)
+    # read rst file and check if it contains code output
+    rst_fname = os.path.splitext(fname)[0] + '.rst'
+    with codecs.open(os.path.join(gallery_conf['gallery_dir'], rst_fname),
+                     mode='r', encoding='utf-8') as f:
+        rst = f.read()
+    return rst
+
+
+def test_remove_config_comments(gallery_conf):
+    """Test the gallery_conf['remove_config_comments'] setting."""
+    rst = _generate_rst(gallery_conf, 'test.py', CONTENT)
+    assert '# sphinx_gallery_thumbnail_number = 1' in rst
+
+    gallery_conf['remove_config_comments'] = True
+    rst = _generate_rst(gallery_conf, 'test.py', CONTENT)
+    assert '# sphinx_gallery_thumbnail_number = 1' not in rst
+
+
 def test_gen_dir_rst(gallery_conf, fakesphinxapp):
     """Test gen_dir_rst."""
     print(os.listdir(gallery_conf['examples_dir']))
@@ -273,17 +321,8 @@ def test_pattern_matching(gallery_conf, log_collector):
     # create three files in tempdir (only one matches the pattern)
     fnames = ['plot_0.py', 'plot_1.py', 'plot_2.py']
     for fname in fnames:
-        with codecs.open(os.path.join(gallery_conf['examples_dir'], fname),
-                         mode='w', encoding='utf-8') as f:
-            f.write('\n'.join(CONTENT))
-        # generate rst file
-        sg.generate_file_rst(fname, gallery_conf['gallery_dir'],
-                             gallery_conf['examples_dir'], gallery_conf)
-        # read rst file and check if it contains code output
+        rst = _generate_rst(gallery_conf, fname, CONTENT)
         rst_fname = os.path.splitext(fname)[0] + '.rst'
-        with codecs.open(os.path.join(gallery_conf['gallery_dir'], rst_fname),
-                         mode='r', encoding='utf-8') as f:
-            rst = f.read()
         if re.search(gallery_conf['filename_pattern'],
                      os.path.join(gallery_conf['gallery_dir'], rst_fname)):
             assert code_output in rst
diff --git a/sphinx_gallery/tests/test_py_source_parser.py b/sphinx_gallery/tests/test_py_source_parser.py
index b550bb9..93ac792 100644
--- a/sphinx_gallery/tests/test_py_source_parser.py
+++ b/sphinx_gallery/tests/test_py_source_parser.py
@@ -25,3 +25,41 @@ def test_get_docstring_and_rest(unicode_sample, tmpdir):
         fid.write('print("hello")\n')
     with pytest.raises(ValueError, match='Could not find docstring'):
         sg.get_docstring_and_rest(fname)
+
+
+@pytest.mark.parametrize('content, file_conf', [
+    ("No config\nin here.",
+     {}),
+    ("# sphinx_gallery_line_numbers = True",
+     {'line_numbers': True}),
+    ("  #   sphinx_gallery_line_numbers   =   True   ",
+     {'line_numbers': True}),
+    ("#sphinx_gallery_line_numbers=True",
+     {'line_numbers': True}),
+    ("#sphinx_gallery_thumbnail_number\n=\n5",
+     {'thumbnail_number': 5}),
+])
+def test_extract_file_config(content, file_conf):
+    assert sg.extract_file_config(content) == file_conf
+
+
+@pytest.mark.parametrize('contents, result', [
+    ("No config\nin here.",
+     "No config\nin here."),
+    ("# sphinx_gallery_line_numbers = True",
+     ""),
+    ("  #   sphinx_gallery_line_numbers   =   True   ",
+     ""),
+    ("#sphinx_gallery_line_numbers=True",
+     ""),
+    ("#sphinx_gallery_thumbnail_number\n=\n5",
+     ""),
+    ("a = 1\n# sphinx_gallery_line_numbers = True\nb = 1",
+     "a = 1\nb = 1"),
+    ("a = 1\n\n# sphinx_gallery_line_numbers = True\n\nb = 1",
+     "a = 1\n\n\nb = 1"),
+    ("# comment\n# sphinx_gallery_line_numbers = True\n# commment 2",
+     "# comment\n# commment 2"),
+])
+def test_remove_config_comments(contents, result):
+    assert sg.remove_config_comments(contents) == result