Repository: python/mypy

Issue:
error points to wrong place when using `=` f-string debug syntax in wrong version of python
given:

```console
$ python --version
Python 3.7.4
$ mypy --version
mypy 0.770
```

This program, when run with mypy being run by python < 3.8

```python
# unrelated

def huh() -> None:
    pint = "1"
    print(f"{pint=}")
```

Always produces an error pointing at the first line in the file (I believe usually offset by the distance of the first `=` in the f-string`.

```console
$ mypy  --pretty /tmp/blah.py
/tmp/blah.py:1: error: invalid syntax
    # unrelated
          ^
Found 1 error in 1 file (checked 1 source file)
```

When ideally it would point to the last line in the example.

This does not happen when running mypy in a python3.8 virtualenv. It also doesn't happen when running mypy under 3.8 with a `--python-version` flag set to < 3.8, which is probably a separate bug.


Discussion:
Yeah, it's relative to the start of the f-string. You'll encounter this with any SyntaxError inside an f-string. The issue is with the SyntaxError Python raises, e.g. if you were to run your code you'd see:
```
~/dev/mypy master λ python3.6 blah.py
  File "<fstring>", line 1
    (pint=)
         ^
SyntaxError: invalid syntax
```
mypy just takes the lineno 1, offset 6 that the SyntaxError provides and assumes that the error is in the "file" that it's parsing. All in all, might make more sense to report this on bugs.python.org. I assume if this were easily fixable it would have been done, but I can't actually find an existing issue on bugs.python.org for this.

Regarding not getting an error when running on Python 3.8, with `--python-version` set. When run by Python 3.8, mypy uses Python 3.8's AST module to parse source code. Python 3.8's AST supports a `feature_version` to parse using a previous Python's grammar (https://docs.python.org/3/library/ast.html#ast.parse). I filed https://bugs.python.org/issue40614 for this.
Yeah it looks like this is a Python parser bug that we copied into typed_ast. It must have been fixed in Python 3.8.

So I guess this is a typed_ast bug?
Python 3.8 added the debug f-string syntax, so it no longer raises a SyntaxError. However, other SyntaxError's in f-strings in Python 3.8 are still relative to the start of the f-string:
```
~/delete λ cat test85.py 
# unrelated

f"{1+}"

~/delete λ python3.8 test85.py                                                             
  File "<fstring>", line 1
    (1+)
       ^
SyntaxError: unexpected EOF while parsing

~/delete λ python3.8 -m mypy test85.py --pretty
test85.py:1: error: unexpected EOF while parsing
    # unrelated
        ^
Found 1 error in 1 file (checked 1 source file)
```
For what it's worth, it looks like Python 3.9's new parser deals with this better:
```
~/delete λ python3.9 test85.py                 
  File "<fstring>", line 3
    (1+)
       ^
SyntaxError: invalid syntax
~/delete λ python3.9 -m mypy test85.py --pretty
test85.py:3: error: invalid syntax
    f"{1+}"
        ^
Found 1 error in 1 file (checked 1 source file)
```
It still shows the weird <fstring> file, but because it gets the line number right, the mypy error message works out.

The bug I filed earlier (https://bugs.python.org/issue40614) for respecting feature version when parsing debug f-string syntax still holds:
```
Python 3.9.0a6+ (heads/master:75b863a, May 18 2020, 13:10:31) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse('''f"{x=}"''', feature_version=(3, 7))
<ast.Module object at 0x10f626100>
>>> # ^that should raise SyntaxError
```
The feature version bug has now been fixed (and the fix will be backported to 3.8). In summary:
On 3.9, both issues in this thread should no longer be present.
On 3.8, the feature version bug should go away with the next micro version release. For earlier Pythons than that, the bug doesn't apply.
On 3.8 and earlier, working around ast / typed_ast's weird reporting of SyntaxErrors is something I think mypy is unlikely to do.

I think we can close this?
Since this behavior is pretty confusing, let's leave this open in case somebody comes up with a simple workaround for older Python versions. This is fairly low priority, however.

Resolving patch:
diff --git a/mypy/fastparse.py b/mypy/fastparse.py
--- a/mypy/fastparse.py
+++ b/mypy/fastparse.py
@@ -169,6 +169,11 @@ def parse(source: Union[str, bytes],
         tree.path = fnam
         tree.is_stub = is_stub_file
     except SyntaxError as e:
+        if sys.version_info < (3, 9) and e.filename == "<fstring>":
+            # In Python 3.8 and earlier, syntax errors in f-strings have lineno relative to the
+            # start of the f-string. This would be misleading, as mypy will report the error as the
+            # lineno within the file.
+            e.lineno = None
         errors.report(e.lineno if e.lineno is not None else -1, e.offset, e.msg, blocker=True,
                       code=codes.SYNTAX)
         tree = MypyFile([], [], False, {})


Test patch: