Repository: psf/black

Issue:
Provide a more useful error when parsing fails during AST safety checks
### Summary & why

Currently when Black encounters a SyntaxError while parsing during its AST safety check, it tries older versions of Python to see if it will parse then under its grammar (i.e. the syntax - which is an oversimplification but generally words). In other words, if you have a file that is valid Python 2.7 code and are running on Python 3.8, Black will first try will try Python 3.8 which fails, then 3.7 which also fails, then 3.6, 3.5, and finally 2.7. Since 2.7 is last version it tried parsing with, the error message from that version is the one ultimately emitted to the user.

The behaviour described above logically makes sense but is of limited use and sometimes even annoyance of the end-user since chances are that their file is supposed to be valid Python 3, so a message for Python 2 ain't useful (and can be downright confusing). 

### Solutions & technical details 

We should change Black so when a SyntaxError is encountered while parsing during the AST safety checks it reparses with the latest Python it can try. Then we can take its error message and emit that one to the user, and voilà, a much more useful error message.  

Another option (from @ichard26 personally) is that we could record the error message of the latest Python parse attempt when it fails and then if all of the parse attempts fail, emit the recorded error message instead of freshest error message (probably Python 2). 

One slight issue is that we emit an error telling users if they need Python 2 support they should reinstall Black with the `python2` extra when we try Python 2 parsing (which fails due to missing dependencies). This could be downgraded to a warning so the rest of this enhancement can still do its thing. Or it could just be eliminated which is probably fine since Python 2 is basically dead. 

To potential contributors: 

- [Checkout our contributing docs to get started](https://github.com/psf/black/blob/master/CONTRIBUTING.md). If you need help, feel free to reach out!
- You'll need to definitely touch this part of the codebase: https://github.com/psf/black/blob/14c76e89716b5b53c97ece80bb935ea956b7dd89/src/black/__init__.py#L6422-L6443
- You'll probably also need to add / modify some tests in `tests/` 
- And finally add a changelog entry

\- Richard (@ichard26) (added for anyone looking to contribute)

---

## Original Report

This one had me scratching my head for a while.

**How to reproduce**

Feed the following into black with -t py37:

```
reported_error = f""

e=""
# Actual error
malformed_fstring = f"{e}  ({type(e).__name__], {e.args})"

```
Please note the typo in `malformed_fstring`: I put a square closing bracket "]" in place of a curly closing bracket "}".
Please also note that if the e variable is not set, the error does not show up.

**Expected result**

I expected that black would form a usable error message about "malformed_fstring having unmatched parenteses" or some such thing.

**Actual result**

```
black -l 999 -t py37 "test.py"
error: cannot format /tmp/test.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 1)
Oh no! 💥 💔 💥
1 files left unchanged, 1 file failed to reformat.
```

Please note that it reports an error on line 1, even if the error is on line 5.

**Environment**

```
$ black --version
black, version 20.8b1
$ python --version
Python 3.7.4
$ uname -a
Linux y8 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

```
**Possible causes**

After digging for a bit it it seems this happens when black fails to use the AST for python 3.x for some reason and falls back to trying the AST for 2.x which does not support f-strings at all and then barfs on the very first instance of an f-string. In the future, I will take note that whenever black barfs on an f-string, it is because it is parsing thje file as 2.x because it completely failed to parse as 3.x, and instead of looking at the f-string mentioned, start going all out Newton on the rest of the file.

As for why this exact string produces this outcome I can only speculate it has something to do with the "type" keyword being misinterpreted somehow.





Discussion:
Agreed we should probably reparse with latest Python 3 in case of a syntax error. It's much more useful behavior now that Python 2 is dead.
Also, the original error is because you have a `]` instead of a `}` inside your f-string.
FYI, I've edited this issue to provide something actionable. I've also added a `good first issue` label since this is probably one of the most well defined enhancement issues we got and is also rather simple yet still is a good dive into Python development  :)

To fellow maintainers: feel free to edit my part above the original report! (I'd especially appreciate edits to the "technical details" section)

Resolving patch:
diff --git a/src/black/parsing.py b/src/black/parsing.py
--- a/src/black/parsing.py
+++ b/src/black/parsing.py
@@ -3,7 +3,7 @@
 """
 import ast
 import sys
-from typing import Iterable, Iterator, List, Set, Union
+from typing import Iterable, Iterator, List, Set, Union, Tuple
 
 # lib2to3 fork
 from blib2to3.pytree import Node, Leaf
@@ -106,28 +106,36 @@ def lib2to3_unparse(node: Node) -> str:
     return code
 
 
-def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
+def parse_single_version(
+    src: str, version: Tuple[int, int]
+) -> Union[ast.AST, ast3.AST, ast27.AST]:
     filename = "<unknown>"
-    if sys.version_info >= (3, 8):
-        # TODO: support Python 4+ ;)
-        for minor_version in range(sys.version_info[1], 4, -1):
-            try:
-                return ast.parse(src, filename, feature_version=(3, minor_version))
-            except SyntaxError:
-                continue
-    else:
-        for feature_version in (7, 6):
-            try:
-                return ast3.parse(src, filename, feature_version=feature_version)
-            except SyntaxError:
-                continue
-    if ast27.__name__ == "ast":
-        raise SyntaxError(
-            "The requested source code has invalid Python 3 syntax.\n"
-            "If you are trying to format Python 2 files please reinstall Black"
-            " with the 'python2' extra: `python3 -m pip install black[python2]`."
-        )
-    return ast27.parse(src)
+    # typed_ast is needed because of feature version limitations in the builtin ast
+    if sys.version_info >= (3, 8) and version >= (3,):
+        return ast.parse(src, filename, feature_version=version)
+    elif version >= (3,):
+        return ast3.parse(src, filename, feature_version=version[1])
+    elif version == (2, 7):
+        return ast27.parse(src)
+    raise AssertionError("INTERNAL ERROR: Tried parsing unsupported Python version!")
+
+
+def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
+    # TODO: support Python 4+ ;)
+    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]
+
+    if ast27.__name__ != "ast":
+        versions.append((2, 7))
+
+    first_error = ""
+    for version in sorted(versions, reverse=True):
+        try:
+            return parse_single_version(src, version)
+        except SyntaxError as e:
+            if not first_error:
+                first_error = str(e)
+
+    raise SyntaxError(first_error)
 
 
 def stringify_ast(


Test patch:
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -451,38 +451,6 @@ def test_skip_magic_trailing_comma(self) -> None:
             )
             self.assertEqual(expected, actual, msg)
 
-    @pytest.mark.no_python2
-    def test_python2_should_fail_without_optional_install(self) -> None:
-        if sys.version_info < (3, 8):
-            self.skipTest(
-                "Python 3.6 and 3.7 will install typed-ast to work and as such will be"
-                " able to parse Python 2 syntax without explicitly specifying the"
-                " python2 extra"
-            )
-
-        source = "x = 1234l"
-        tmp_file = Path(black.dump_to_file(source))
-        try:
-            runner = BlackRunner()
-            result = runner.invoke(black.main, [str(tmp_file)])
-            self.assertEqual(result.exit_code, 123)
-        finally:
-            os.unlink(tmp_file)
-        assert result.stderr_bytes is not None
-        actual = (
-            result.stderr_bytes.decode()
-            .replace("\n", "")
-            .replace("\\n", "")
-            .replace("\\r", "")
-            .replace("\r", "")
-        )
-        msg = (
-            "The requested source code has invalid Python 3 syntax."
-            "If you are trying to format Python 2 files please reinstall Black"
-            " with the 'python2' extra: `python3 -m pip install black[python2]`."
-        )
-        self.assertIn(msg, actual)
-
     @pytest.mark.python2
     @patch("black.dump_to_file", dump_to_stderr)
     def test_python2_print_function(self) -> None: