Repository: mwouts/jupytext

Issue:
Escape IPython's magic help ("?" and "??") 
Getting help in a code cell with something like `float?` or `float??` runs in the notebook but fails in the generated .py file. It would be nice to consider this IPython magic and to comment it out in the .py (which otherwise fails to run, or which makes the automatic conversion from Python 2 to Python 3 fail).

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=h1) Report
> Merging [#372](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/20586e5cde0f3502478e5492ccdf04b96687ee00?src=pr&el=desc) will **decrease** coverage by `10.63%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/372/graphs/tree.svg?width=650&token=a1pNjsiILi&height=150&src=pr)](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master     #372       +/-   ##
===========================================
- Coverage   99.02%   88.39%   -10.64%     
===========================================
  Files          75       75               
  Lines        7197     7219       +22     
===========================================
- Hits         7127     6381      -746     
- Misses         70      838      +768
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tests/test\_cli.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jbGkucHk=) | `83.14% <100%> (-16.86%)` | :arrow_down: |
| [tests/test\_read\_simple\_python.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9weXRob24ucHk=) | `95.56% <100%> (-4.44%)` | :arrow_down: |
| [tests/test\_escape\_magics.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9lc2NhcGVfbWFnaWNzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [jupytext/magics.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-anVweXRleHQvbWFnaWNzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [jupytext/cell\_to\_text.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-anVweXRleHQvY2VsbF90b190ZXh0LnB5) | `96.44% <100%> (-0.98%)` | :arrow_down: |
| [tests/test\_hide\_remove\_input\_outputs\_rmarkdown.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9oaWRlX3JlbW92ZV9pbnB1dF9vdXRwdXRzX3JtYXJrZG93bi5weQ==) | `28.57% <0%> (-71.43%)` | :arrow_down: |
| [tests/test\_black.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9ibGFjay5weQ==) | `31.53% <0%> (-68.47%)` | :arrow_down: |
| [tests/test\_mirror.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9taXJyb3IucHk=) | `36.84% <0%> (-63.16%)` | :arrow_down: |
| [tests/test\_cell\_metadata.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jZWxsX21ldGFkYXRhLnB5) | `40.5% <0%> (-59.5%)` | :arrow_down: |
| [tests/test\_read\_simple\_pandoc.py](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9wYW5kb2MucHk=) | `44.82% <0%> (-55.18%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/mwouts/jupytext/pull/372/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=footer). Last update [20586e5...2f4fc99](https://codecov.io/gh/mwouts/jupytext/pull/372?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index aabe4f5..04c4a5e 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -15,6 +15,7 @@ Release History
 - Markdown and raw cells can be quoted with triple quotes in the ``py:percent`` format. And Markdown cells can start with just ``# %% [md]`` (#305)
 - The light format uses ``# + [markdown]`` rather than the previous ``cell_type`` metadata to identify markdown cells with metadata (#356)
 - Explicit Markdown cells in the light format ``# + [markdown]`` can use triple quotes (#356)
+- IPython magic help commands like ``float?`` are classified as magics, and thus commented in Python scripts (#297)
 - Cell metadata can be encoded as either key=value (the new default) or in JSON. An automatic option ``cell_metadata_json`` should help minimize the impact on existing files (#344)
 - R Markdown hidden inputs, outputs, or cells are now mapped to the corresponding Jupyter Book tags by default (#337)
 - The Jupyter Notebook extension for Jupytext is compatible with Jupyter Notebook 6.0 (#346)
diff --git a/jupytext/cell_reader.py b/jupytext/cell_reader.py
index b8154ae..5ea00e2 100644
--- a/jupytext/cell_reader.py
+++ b/jupytext/cell_reader.py
@@ -14,7 +14,7 @@ except (ImportError, SyntaxError):
 from .cell_metadata import is_active, text_to_metadata, is_json_metadata, rmd_options_to_metadata
 from .languages import _JUPYTER_LANGUAGES
 from .stringparser import StringParser
-from .magics import uncomment_magic, is_magic, unescape_code_start
+from .magics import uncomment_magic, is_magic, unescape_code_start, need_explicit_marker
 from .pep8 import pep8_lines_between_cells
 
 _BLANK_LINE = re.compile(r"^\s*$")
@@ -103,6 +103,7 @@ class BaseCellReader(object):
         self.metadata = None
         self.org_content = []
         self.content = []
+        self.explicit_soc = None
         self.explicit_eoc = None
         self.cell_type = None
         self.language = None
@@ -189,6 +190,7 @@ class BaseCellReader(object):
                 self.metadata['lines_to_end_of_cell_marker'] = lines_to_end_of_cell_marker
 
         # Uncomment content
+        self.explicit_soc = cell_start > 0
         self.content = self.extract_content(source)
 
         # Is this an inactive cell?
@@ -416,7 +418,10 @@ class ScriptCellReader(BaseCellReader):  # pylint: disable=W0223
         if self.cell_type == 'code' or self.comment != "#'":
             if self.comment_magics:
                 if is_active(self.ext, self.metadata):
-                    uncomment_magic(lines, self.language or self.default_language)
+                    uncomment_magic(lines, self.language or self.default_language, explicitly_code=self.explicit_soc)
+                    if self.cell_type == 'code' and not self.explicit_soc and \
+                            need_explicit_marker(lines, self.language or self.default_language):
+                        self.metadata['comment_questions'] = False
                 else:
                     lines = uncomment(lines)
 
@@ -636,6 +641,7 @@ class DoublePercentScriptCellReader(LightScriptCellReader):
         self.comment = script['comment']
         self.start_code_re = re.compile(r"^{}\s*%%(%*)\s(.*)$".format(self.comment))
         self.alternative_start_code_re = re.compile(r"^{}\s*(%%|<codecell>|In\[[0-9 ]*\]:?)\s*$".format(self.comment))
+        self.explicit_soc = True
 
     def metadata_and_language_from_option_line(self, line):
         """Parse code options on the given line. When a start of a code cell
diff --git a/jupytext/cell_to_text.py b/jupytext/cell_to_text.py
index 7474b82..7658607 100644
--- a/jupytext/cell_to_text.py
+++ b/jupytext/cell_to_text.py
@@ -7,7 +7,7 @@ from .languages import cell_language, comment_lines
 from .cell_metadata import is_active, _IGNORE_CELL_METADATA
 from .cell_metadata import metadata_to_text, metadata_to_rmd_options, metadata_to_double_percent_options
 from .metadata_filter import filter_metadata
-from .magics import comment_magic, escape_code_start
+from .magics import comment_magic, escape_code_start, need_explicit_marker
 from .cell_reader import LightScriptCellReader, MarkdownCellReader, RMarkdownCellReader
 from .languages import _SCRIPT_EXTENSIONS
 from .pep8 import pep8_lines_between_cells
@@ -115,7 +115,7 @@ class BaseCellExporter(object):
         if self.comment and self.comment != "#'" and is_active(self.ext, self.metadata) and \
                 self.fmt.get('format_name') not in ['percent', 'hydrogen']:
             source = copy(source)
-            comment_magic(source, self.language, self.comment_magics)
+            comment_magic(source, self.language, self.comment_magics, explicitly_code=self.cell_type == 'code')
 
         return comment_lines(source, self.comment)
 
@@ -260,13 +260,15 @@ class LightScriptCellExporter(BaseCellExporter):
         active = is_active(self.ext, self.metadata, self.language == self.default_language)
         source = copy(self.source)
         escape_code_start(source, self.ext, self.language)
+        comment_questions = self.metadata.pop('comment_questions', True)
 
         if active:
-            comment_magic(source, self.language, self.comment_magics)
+            comment_magic(source, self.language, self.comment_magics, comment_questions)
         else:
             source = self.markdown_to_text(source)
 
-        if self.explicit_start_marker(source):
+        if (active and comment_questions and need_explicit_marker(self.source, self.language, self.comment_magics)) \
+                or self.explicit_start_marker(source):
             self.metadata['endofcell'] = self.cell_marker_end or endofcell_marker(source, self.comment)
 
         if not self.metadata or not self.use_cell_markers:
diff --git a/jupytext/magics.py b/jupytext/magics.py
index cf38d4b..3475aa0 100644
--- a/jupytext/magics.py
+++ b/jupytext/magics.py
@@ -29,11 +29,13 @@ _PYTHON_MAGIC_CMD = re.compile(r"^(# |#)*({})($|\s$|\s[^=,])".format('|'.join(
     ['cat', 'cp', 'mv', 'rm', 'rmdir', 'mkdir'] +
     # windows
     ['copy', 'ddir', 'echo', 'ls', 'ldir', 'mkdir', 'ren', 'rmdir'])))
+# Python help commands end with ?
+_IPYTHON_MAGIC_HELP = re.compile(r"^(# )*[^\s]*\?\s*$")
 
 _SCRIPT_LANGUAGES = [_SCRIPT_EXTENSIONS[ext]['language'] for ext in _SCRIPT_EXTENSIONS]
 
 
-def is_magic(line, language, global_escape_flag=True):
+def is_magic(line, language, global_escape_flag=True, explicitly_code=False):
     """Is the current line a (possibly escaped) Jupyter magic, and should it be commented?"""
     if language in ['octave', 'matlab'] or language not in _SCRIPT_LANGUAGES:
         return False
@@ -47,15 +49,31 @@ def is_magic(line, language, global_escape_flag=True):
         return False
     if _PYTHON_HELP_OR_BASH_CMD.match(line):
         return True
+    if explicitly_code and _IPYTHON_MAGIC_HELP.match(line):
+        return True
     return _PYTHON_MAGIC_CMD.match(line)
 
 
-def comment_magic(source, language='python', global_escape_flag=True):
+def need_explicit_marker(source, language='python', global_escape_flag=True, explicitly_code=True):
+    """Does this code needs an explicit cell marker?"""
+    if language != 'python' or not global_escape_flag or not explicitly_code:
+        return False
+
+    parser = StringParser(language)
+    for pos, line in enumerate(source):
+        if not parser.is_quoted() and is_magic(line, language, global_escape_flag, explicitly_code):
+            if not is_magic(line, language, global_escape_flag, False):
+                return True
+        parser.read_line(line)
+    return False
+
+
+def comment_magic(source, language='python', global_escape_flag=True, explicitly_code=True):
     """Escape Jupyter magics with '# '"""
     parser = StringParser(language)
     next_is_magic = False
     for pos, line in enumerate(source):
-        if not parser.is_quoted() and (next_is_magic or is_magic(line, language, global_escape_flag)):
+        if not parser.is_quoted() and (next_is_magic or is_magic(line, language, global_escape_flag, explicitly_code)):
             source[pos] = _COMMENT[language] + ' ' + line
             next_is_magic = language == 'python' and _LINE_CONTINUATION_RE.match(line)
         parser.read_line(line)
@@ -72,12 +90,12 @@ def unesc(line, language):
     return line
 
 
-def uncomment_magic(source, language='python', global_escape_flag=True):
+def uncomment_magic(source, language='python', global_escape_flag=True, explicitly_code=True):
     """Unescape Jupyter magics"""
     parser = StringParser(language)
     next_is_magic = False
     for pos, line in enumerate(source):
-        if not parser.is_quoted() and (next_is_magic or is_magic(line, language, global_escape_flag)):
+        if not parser.is_quoted() and (next_is_magic or is_magic(line, language, global_escape_flag, explicitly_code)):
             source[pos] = unesc(line, language)
             next_is_magic = language == 'python' and _LINE_CONTINUATION_RE.match(line)
         parser.read_line(line)


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c2440aa..5d4712a 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -977,7 +977,7 @@ def test_339_py(tmpdir):
 cat = 42
 """)
 
-    def erroneous_is_magic(line, language, comment_magics):
+    def erroneous_is_magic(line, language, comment_magics, explicitly_code):
         return 'cat' in line
 
     with mock.patch('jupytext.magics.is_magic', erroneous_is_magic):
diff --git a/tests/test_escape_magics.py b/tests/test_escape_magics.py
index 7d5a4e1..1042cc5 100644
--- a/tests/test_escape_magics.py
+++ b/tests/test_escape_magics.py
@@ -133,6 +133,12 @@ def test_markdown_image_is_not_magic():
     assert not is_magic('# ![Image name](image.png', 'python')
 
 
+def test_question_is_not_magic():
+    assert is_magic('float?', 'python', explicitly_code=True)
+    assert is_magic('# float?', 'python', explicitly_code=True)
+    assert not is_magic('# question: float?', 'python', explicitly_code=True)
+
+
 def test_multiline_python_magic(no_jupytext_version_number):
     nb = new_notebook(cells=[new_code_cell("""%load_ext watermark
 %watermark -u -n -t -z \\
diff --git a/tests/test_read_simple_python.py b/tests/test_read_simple_python.py
index 87934af..90b0169 100644
--- a/tests/test_read_simple_python.py
+++ b/tests/test_read_simple_python.py
@@ -389,6 +389,50 @@ def test_isolated_cell_with_magic(pynb="""# ---
     compare(pynb2, pynb)
 
 
+def test_ipython_help_are_commented_297(text="""# This is a markdown cell
+# that ends with a question: float?
+
+# The next cell is also a markdown cell,
+# because it has no code marker:
+
+# float?
+
+# +
+# float?
+
+# +
+# float??
+
+# +
+# Finally a question in a code
+# # cell?
+""", nb=new_notebook(
+    cells=[
+        new_markdown_cell('This is a markdown cell\nthat ends with a question: float?'),
+        new_markdown_cell('The next cell is also a markdown cell,\nbecause it has no code marker:'),
+        new_markdown_cell('float?'),
+        new_code_cell('float?'),
+        new_code_cell('float??'),
+        new_code_cell('# Finally a question in a code\n# cell?')])):
+    nb2 = jupytext.reads(text, 'py')
+    compare_notebooks(nb2, nb)
+
+    text2 = jupytext.writes(nb2, 'py')
+    compare(text2, text)
+
+
+def test_questions_in_unmarked_cells_are_not_uncommented_297(text="""# This cell has no explicit marker
+# question?
+1 + 2
+""", nb=new_notebook(cells=[new_code_cell('# This cell has no explicit marker\n# question?\n1 + 2',
+                                          metadata={'comment_questions': False})])):
+    nb2 = jupytext.reads(text, 'py')
+    compare_notebooks(nb2, nb)
+
+    text2 = jupytext.writes(nb2, 'py')
+    compare(text2, text)
+
+
 def test_read_multiline_comment(pynb="""'''This is a multiline
 comment with "quotes", 'single quotes'
 # and comments