Repository: sphinx-doc/sphinx

Issue:
Form Feed Character followed by class property crashes the Python parser
Subject: Form feed character followed by class property crashes the Python parser

### Problem
When attempting to run Sphinx against my project, the build process crashes while parsing my Python file. The specific error differs between the latest release and master - I am guessing that the error handling in this area has changed?

This problem was not seen when using an old version (1.4.4). Bisecting commits, it looks like the change that is causing this is commit 868382353 ("pycode: Replace pgen2 by own parser").

#### Procedure to reproduce the problem
After some experimentation, a minimum example to reproduce the probem is as follows
```
class MyClass:

    myprop = 'something else'
```

The character in the line above `myprop` is a form feed character, which in the full project is being used to separate blocks of related functions in the class. The following things will fix the issue:
- Removing the character
- Removing the property declaration
- Replacing the property declaration with a function declaration
- Moving the property declaration above the character

Adding functions anywhere in the class (before the character, between the character and the prop, after the prop) does not seem to affect the issue.

#### Error logs / results
Running at the bad commit, you get the following error:

```
C:\Users\james\open_source\lwe-estimator>sphinx-build -M html doc doc\_build
Running Sphinx v1.7+
making output directory...
loading pickled environment... not yet created
[autosummary] generating autosummary for: api_doc.rst, index.rst
[autosummary] generating autosummary for: C:\Users\james\open_source\lwe-estimator\doc\_apidoc\my_file.rst
[autosummary] generating autosummary for: C:\Users\james\open_source\lwe-estimator\doc\_apidoc\my_file\my_file.MyClass.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 2 source files that are out of date
updating environment: 4 added, 0 changed, 0 removed
reading sources... [ 50%] _apidoc/my_file/my_file.MyClass
Exception occurred:
  File "c:\users\james\open_source\sphinx\sphinx\pycode\__init__.py", line 107, in parse
    raise PycodeError('parsing failed: %r' % exc)
sphinx.errors.PycodeError: parsing failed: AssertionError()
The full traceback has been saved in C:\Users\james\AppData\Local\Temp\sphinx-err-7moettvf.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
```

- The trace file is at [sphinx-err-7moettvf.log](https://github.com/sphinx-doc/sphinx/files/2042554/sphinx-err-7moettvf.log)

#### Expected results
The project builds to completion

### Reproducible project / your project
- An example that hits this failure is attached as [example.zip](https://github.com/sphinx-doc/sphinx/files/2042555/example.zip)

### Environment info
- OS: Win (this has also been seen on Linux and Mac with the full project)
- Python version: 3.6.3
- Sphinx version: 1.7+


Discussion:

Patch:
diff --git a/CHANGES b/CHANGES
index 9c48d472c..db36f5d81 100644
--- a/CHANGES
+++ b/CHANGES
@@ -26,6 +26,7 @@ Bugs fixed
 * C++, fix lookup of full template specializations with no template arguments.
 * #4667: C++, fix assertion on missing references in global scope when using
   intersphinx. Thanks to Alan M. Carroll.
+* #5019: autodoc: crashed by Form Feed Character
 
 Testing
 --------
diff --git a/sphinx/pycode/parser.py b/sphinx/pycode/parser.py
index 5c4291d3d..40334f2e3 100644
--- a/sphinx/pycode/parser.py
+++ b/sphinx/pycode/parser.py
@@ -34,6 +34,11 @@
     ASSIGN_NODES = (ast.Assign)
 
 
+def filter_whitespace(code):
+    # type: (unicode) -> unicode
+    return code.replace('\f', ' ')  # replace FF (form feed) with whitespace
+
+
 def get_assign_targets(node):
     # type: (ast.AST) -> List[ast.expr]
     """Get list of targets from Assign and AnnAssign node."""
@@ -466,7 +471,7 @@ class Parser(object):
 
     def __init__(self, code, encoding='utf-8'):
         # type: (unicode, unicode) -> None
-        self.code = code
+        self.code = filter_whitespace(code)
         self.encoding = encoding
         self.comments = {}          # type: Dict[Tuple[unicode, unicode], unicode]
         self.deforders = {}         # type: Dict[unicode, int]


Test patch:
diff --git a/tests/test_pycode_parser.py b/tests/test_pycode_parser.py
index 29363e17e..0875329a4 100644
--- a/tests/test_pycode_parser.py
+++ b/tests/test_pycode_parser.py
@@ -315,3 +315,12 @@ def test_decorators():
                                   'func3': ('def', 7, 9),
                                   'Foo': ('class', 11, 15),
                                   'Foo.method': ('def', 13, 15)}
+
+
+def test_formfeed_char():
+    source = ('class Foo:\n'
+              '\f\n'
+              '    attr = 1234  #: comment\n')
+    parser = Parser(source)
+    parser.parse()
+    assert parser.comments == {('Foo', 'attr'): 'comment'}