Repository: sphinx-gallery/sphinx-gallery

Issue:
Feature request: Add an option for different cell separations
We should add an option so that to give equivalent to what currently is "########################", used as a cell separated.

Candidates:

* "\n\n" to have lighter source files

* "# %%" to be compatible with other softwares such as atom hydrogen, as mentioned below by @lesteve 

Thanks to @ogrisel for having suggested this a while ago.

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=h1) Report
> Merging [#518](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=desc) into [master](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/commit/f2e6cc984aa88f6a2f026d63eb4ab55692ce2e59?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518/graphs/tree.svg?width=650&token=K4eytvhwU5&height=150&src=pr)](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #518   +/-   ##
=======================================
  Coverage   96.23%   96.23%           
=======================================
  Files          29       29           
  Lines        2686     2686           
=======================================
  Hits         2585     2585           
  Misses        101      101
```


| [Impacted Files](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sphinx\_gallery/py\_source\_parser.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvcHlfc291cmNlX3BhcnNlci5weQ==) | `91.76% <ø> (ø)` | :arrow_up: |
| [sphinx\_gallery/tests/test\_gen\_rst.py](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518/diff?src=pr&el=tree#diff-c3BoaW54X2dhbGxlcnkvdGVzdHMvdGVzdF9nZW5fcnN0LnB5) | `98.74% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=footer). Last update [f2e6cc9...3f11b72](https://codecov.io/gh/sphinx-gallery/sphinx-gallery/pull/518?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

choldgraf: shit why didn't we think of this before haha, I think it's a good patch, seems reasonable to me. So if I wanted to add rST underneath it would look something like:

```python
# %%
# my rst
# ======

my = "code"
```

?
lucyleeow: After speaking with @GaelVaroquaux we think both `# %%` and `#%%` (with and without space) should be supported:

- [Spyder](https://docs.spyder-ide.org/editor.html) supports both because Eclipse's automatic code formatting changes `#%%` into `# %%` (see [this](https://github.com/spyder-ide/spyder/issues/852#issuecomment-74599857) issue)
- [Hydrogen](https://nteract.gitbooks.io/hydrogen/docs/Usage/NotebookFiles.html#cell-markers) states that `# %%` is supported in their documentation but on testing both syntaxes work and exports correct cell formatting to jupyter notebook.
- [VSCode](https://code.visualstudio.com/docs/python/jupyter-support) states that `#%%` is supported but on testing both syntaxes work. 
- Could not test in PyCharm as this functionality is only available in their professional version.

If there are no objections, I will change the code and documentation to support both syntaxes. I think I will state that a line of `#`'s or `#%%` can be used and just note that `# %%` is also acceptable for compatibility.
larsoner: Works for me
larsoner: > I think that an entry in the changelog is needed.

I just noticed this bit. FYI we don't need changelog entries because we autogenerate them based on PR labels. But it's fine to leave the one you added in there
lucyleeow: Amended the doc to add a note on using sphinx gallery with code blocks in various IDE's. Hopefully this makes sense.
choldgraf: This seems good to me! @GaelVaroquaux you're still blocking changes just FYI :-)
larsoner: > @GaelVaroquaux you're still blocking changes just FYI :-)

From the above I'm pretty sure it's just for the changelog update and link updating which are done. So I'll merge but if there was something still wrong @GaelVaroquaux feel free to chime in and we'll fix it.

Thanks @lucyleeow !

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 191ecef..932b79c 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -1,7 +1,7 @@
 Change Log
 ==========
 
-v0.4.0
+v0.5.0
 ------
 
 Developer changes
@@ -15,6 +15,10 @@ Incompatible changes
 - Dropped support for Sphinx < 1.8.3.
 - Dropped support for Python < 3.5.
 
+**Implemented enhancements:**
+
+-  ENH: Support ``#%%`` cell separator `#370 <https://github.com/sphinx-gallery/sphinx-gallery/pull/518>`__ (`lucyleeow <https://github.com/lucyleeow>`_)
+
 v0.4.0
 ------
 
diff --git a/doc/syntax.rst b/doc/syntax.rst
index cc37c2e..b0e7ae7 100644
--- a/doc/syntax.rst
+++ b/doc/syntax.rst
@@ -39,8 +39,9 @@ Jupyter Notebooks are structured (in fact, Sphinx-Gallery also **creates** a
 Jupyter Notebook for each example that is built).
 
 You can embed rST in your Python examples by including a line of ``#`` symbols
-that spans >= 20 columns. We recommend using 79 columns, like
-this::
+that spans >= 20 columns or ``#%%``. For compatibility reasons, 
+``# %%`` (with a space) can also be used but ``#%%`` is recommended for 
+consistency. If using ``#``'s, we recommend using 79 columns, like this::
 
   ###############################################################################
 
@@ -62,7 +63,20 @@ gallery examples. For example::
   # commented rST block. Instead, they'll resolve as regular Python comments.
   print('my variable plus 2 is {}'.format(myvariable + 2))
 
-Here are the contents of an example Python file from the snippets above.::
+The ``#%%`` syntax is consistent with the 'code block' (or 'code cell')
+syntax in `Jupyter VSCode plugin
+<https://code.visualstudio.com/docs/python/jupyter-support>`_, `Jupytext
+<https://jupytext.readthedocs.io/en/latest/introduction.html>`_, `Pycharm
+<https://www.jetbrains.com/help/pycharm/running-jupyter-notebook-cells.html>`_, 
+`Hydrogen plugin (for Atom)
+<https://nteract.gitbooks.io/hydrogen/>`_ and `Spyder
+<https://docs.spyder-ide.org/editor.html>`_. In these IDEs/with these IDE 
+plugins, ``#%%`` at the start of a line signifies the start of a code block. 
+The code within a code block can be easily executed all at once. This 
+functionality can be helpful when writing a Sphinx-Gallery ``.py`` script.
+
+Here are the contents of an example Python file using the 'code block' 
+functionality::
 
   """
   This is my example script
@@ -71,21 +85,26 @@ Here are the contents of an example Python file from the snippets above.::
   This example doesn't do much, it just makes a simple plot
   """
 
-  ###############################################################################
+  #%%
   # This is a section header
   # ------------------------
-  #
-  # .. note:: This is the first section!
+  # This is the first section!
+  # The `#%%` signifies to Sphinx-Gallery that this text should be rendered as
+  # rST and if using one of the above IDE/plugin's, also signifies the start of a 
+  # 'code block'.
 
   # This line won't be rendered as rST because there's a space after the last block.
   myvariable = 2
   print("my variable is {}".format(myvariable))
+  # This is the end of the 'code block' (if using an above IDE). All code within
+  # this block can be easily executed all at once.
 
-  ###############################################################################
+  #%%
   # This is another section header
   # ------------------------------
   #
   # In the built documentation, it will be rendered as rST after the code above!
+  # This is also another code block.
 
   print('my variable plus 2 is {}'.format(myvariable + 2))
 
diff --git a/sphinx_gallery/py_source_parser.py b/sphinx_gallery/py_source_parser.py
index dd80f56..166552f 100644
--- a/sphinx_gallery/py_source_parser.py
+++ b/sphinx_gallery/py_source_parser.py
@@ -161,7 +161,7 @@ def split_code_and_text_blocks(source_file):
     file_conf = extract_file_config(rest_of_content)
 
     pattern = re.compile(
-        r'(?P<header_line>^#{20,}.*)\s(?P<text_content>(?:^#.*\s)*)',
+        r'(?P<header_line>^#{20,}.*|^# ?%%.*)\s(?P<text_content>(?:^#.*\s)*)',
         flags=re.M)
     sub_pat = re.compile('^#', flags=re.M)
 
diff --git a/tutorials/plot_parse.py b/tutorials/plot_parse.py
index 8f75828..47bb4f4 100644
--- a/tutorials/plot_parse.py
+++ b/tutorials/plot_parse.py
@@ -27,10 +27,12 @@ import matplotlib.pyplot as plt
 # Now there is free repetition of both
 
 #############################################
-# And a single line of hashes can split your blocks
+# A block an be split by either a single line of ``#``'s (>=20 columns) or 
+# ``#%%``. For compatibility reasons ``# %%`` (with a space) can also be used
+# but we recommend only using ``#%%`` for consistency. All future 
+# 'block splitters' used in the source ``.py`` document will be ``#%%``.
 
-
-###############################################################################
+#%%
 # Latex in the comments does not need to be escaped
 #
 # .. math::
@@ -45,30 +47,32 @@ def dummy():
 
 # this should not be part of a 'text' block
 
-######################################################################
+#%%
 #
 # ####################################################################
 #
 # Making a line cut in sphinx
 
-###############################################################################
+#%%
 # .. warning::
-#     The next kind of comments are not supported and become to hard to escape
+#     The next kind of comments are not supported and become too hard to escape
 #     so just don't code like this::
 #
 #         def dummy2():
 #             """Function docstring"""
 #         ####################################
-#         # This comment inside python indentation
+#         # This comment 
+#         #%%
+#         # and this comment inside python indentation
 #         # breaks the block structure and is not
 #         # supported
 #             dummy2
 #
 
-"""Free strings are not supported they remain part of the code"""
+"""Free strings are not supported. They remain part of the code"""
 
-##############################################################################
-# New lines can be included in you block comments and the parser
+#%%
+# New lines can be included in your block comments and the parser
 # is capable of retaining this significant whitespace to work with sphinx
 #
 # So the reStructuredText headers survive
@@ -77,20 +81,21 @@ def dummy():
 
 print('one')
 
-###############################################################################
+#%%
 # Code block separators
-###############################################################################
-# Surrounding a comment line with lines of # like a block spliter also
-# works and creates a new header for that comment block
-# too. Nevertheless to get rich text formatting we advise to use
-# RestructuredText syntax in the comment blocks.
+####################################################################
+# Surrounding a comment line with a line of ``#``'s (like a block splitter)
+# above and below (or ``#%%`` on top and a line of ``#``'s below, as we have 
+# done here in the source ``.py`` doc) also works and creates a new header for
+# that comment block too. Nevertheless to get rich text formatting we advise to
+# use RestructuredText syntax in the comment blocks.
 
 print('two')
-##################################################
+#%%
 #
 B = 1
 
-##############################################################################
+#%%
 # End comments
 #
 # That's all folks !


Test patch:
diff --git a/sphinx_gallery/tests/reference_parse.txt b/sphinx_gallery/tests/reference_parse.txt
index 376bbee..8eda9f9 100644
--- a/sphinx_gallery/tests/reference_parse.txt
+++ b/sphinx_gallery/tests/reference_parse.txt
@@ -7,27 +7,29 @@
  ('text',
   'There is no need to always alternate between code and comment blocks\nNow there is free repetition of both\n',
   26),
- ('text', 'And a single line of hashes can split your blocks\n', 30),
- ('text', 'Latex in the comments does not need to be escaped\n\n.. math::\n   \\sin\n', 34),
+ ('text',
+  "A block an be split by either a single line of ``#``'s (>=20 columns) or \n``#%%``. For compatibility reasons ``# %%`` (with a space) can also be used\nbut we recommend only using ``#%%`` for consistency. All future \n'block splitters' used in the source ``.py`` document will be ``#%%``.\n",
+  30),
+ ('text', 'Latex in the comments does not need to be escaped\n\n.. math::\n   \\sin\n', 36),
  ('code',
   '\ndef dummy():\n    """This should not be part of a \'text\' block\'"""\n\n    ######################################\n    # Comment inside code to remain here\n    pass\n\n# this should not be part of a \'text\' block\n\n',
-  38),
+  40),
  ('text',
   '####################################################################\n\nMaking a line cut in sphinx\n',
-  49),
+  51),
  ('text',
-  '.. warning::\n    The next kind of comments are not supported and become to hard to escape\n    so just don\'t code like this::\n\n        def dummy2():\n            """Function docstring"""\n        ####################################\n        # This comment inside python indentation\n        # breaks the block structure and is not\n        # supported\n            dummy2\n\n',
-  55),
- ('code', '\n"""Free strings are not supported they remain part of the code"""\n\n', 67),
+  '.. warning::\n    The next kind of comments are not supported and become too hard to escape\n    so just don\'t code like this::\n\n        def dummy2():\n            """Function docstring"""\n        ####################################\n        # This comment \n        #%%\n        # and this comment inside python indentation\n        # breaks the block structure and is not\n        # supported\n            dummy2\n\n',
+  57),
+ ('code', '\n"""Free strings are not supported. They remain part of the code"""\n\n', 71),
  ('text',
-  'New lines can be included in you block comments and the parser\nis capable of retaining this significant whitespace to work with sphinx\n\nSo the reStructuredText headers survive\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
-  71),
- ('code', "\n\nprint('one')\n\n", 76),
+  'New lines can be included in your block comments and the parser\nis capable of retaining this significant whitespace to work with sphinx\n\nSo the reStructuredText headers survive\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
+  75),
+ ('code', "\n\nprint('one')\n\n", 80),
  ('text',
-  'Code block separators\n##############################################################################\n Surrounding a comment line with lines of # like a block spliter also\n works and creates a new header for that comment block\n too. Nevertheless to get rich text formatting we advise to use\n RestructuredText syntax in the comment blocks.\n',
-  81),
- ('code', "\nprint('two')\n", 87),
- ('code', 'B = 1\n\n', 91),
+  "Code block separators\n###################################################################\n Surrounding a comment line with a line of ``#``'s (like a block splitter)\n above and below (or ``#%%`` on top and a line of ``#``'s below, as we have \n done here in the source ``.py`` doc) also works and creates a new header for\n that comment block too. Nevertheless to get rich text formatting we advise to\n use RestructuredText syntax in the comment blocks.\n",
+  85),
+ ('code', "\nprint('two')\n", 92),
+ ('code', 'B = 1\n\n', 96),
  ('text',
   "End comments\n\nThat's all folks !\n\n.. literalinclude:: plot_parse.py\n\n\n",
-  94)]
+  99)]
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index b3d0081..82644ad 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -67,7 +67,8 @@ def test_split_code_and_text_blocks():
 
 def test_bug_cases_of_notebook_syntax():
     """Test over the known requirements of supported syntax in the
-    notebook styled comments"""
+    notebook styled comments. Use both '#'s' and '# %%' as cell 
+    separators"""
 
     with open('sphinx_gallery/tests/reference_parse.txt') as reference:
         ref_blocks = ast.literal_eval(reference.read())
@@ -110,8 +111,11 @@ def test_rst_block_after_docstring(gallery_conf, tmpdir):
                            '####################',
                            '# Paragraph 1',
                            '',
-                           '####################',
+                           '#%%',
                            '# Paragraph 2',
+                           '',
+                           '# %%',
+                           '# Paragraph 3',
                            '']))
     file_conf, blocks = sg.split_code_and_text_blocks(filename)
 
@@ -119,6 +123,7 @@ def test_rst_block_after_docstring(gallery_conf, tmpdir):
     assert blocks[0][0] == 'text'
     assert blocks[1][0] == 'text'
     assert blocks[2][0] == 'text'
+    assert blocks[3][0] == 'text'
 
     script_vars = {'execute_script': ''}
 
@@ -133,6 +138,8 @@ def test_rst_block_after_docstring(gallery_conf, tmpdir):
         '',
         'Paragraph 2',
         '',
+        'Paragraph 3',
+        '',
         ''])