Repository: psf/black

Issue:
[python 3.9] black doesn't support PEP 614
**Describe the bug**
[PEP 614](https://www.python.org/dev/peps/pep-0614/) relaxed the syntax for decorators in python 3.9. However, black doesn't (yet ?) have a `py39` target, and doesn't support this syntax.

**To Reproduce**

1. Take this very simple file `test.py`
```python
from dataclasses import dataclass
d = [dataclass]

@d[0]
class A:
    attr: int

print(A)
```
2. Run _Black_ on it : `black test.py`
3. See error `error: cannot format test.py: Cannot parse: 5:2: @d[0]`

**Expected behavior** _Black_ should accept and format code that is valid for python 3.9 (`python test.py` prints `<class '__main__.A'>` as expected).

**Environment (please complete the following information):**

- Version: master (commit 811decd7f10fb2fb3ae343b9d9d0a3ae53b86a53)
- OS and Python version: Linux, Python 3.9.0rc1

**Does this bug also happen on master?**

1. Use the online formatter at https://black.now.sh/?version=master
This produce the above error
2. Or run _Black_ on your machine:
   - create a new virtualenv (make sure it's the same Python version);
   - clone this repository;
   - run `pip install -e .`;
   - make sure it's sane by running `python -m unittest` (this fails, see #1441 )
   - run `black` like you did last time.

This also produce the above error

**Additional context** Python 3.9 is in pre-release stage, so I'm not sure if this is a bug report or a feature request - I can change to the latter. [PEP 614](https://www.python.org/dev/peps/pep-0614/) highlights the use-case of the syntax (with examples from PyQt5), the above file is a minimalist file to illustrate the issue


Discussion:
So I'm considering working on a PR for that if I can manage. Do I need to update `blib2to3` to a newer version of `lib2to3` (python original), or should I create a new grammar file to load in `blid2to3.pygram.initialize()` where `decorator` is changed from `'@' dotted_name [ '(' [arglist] ')' ] NEWLINE` to `'@' namedexpr_test NEWLINE` as the PEP explains ? (And perhaps replicate all changes to the grammer that leads to the python3.6+ grammar).

It seems to me I need to do the second but I'm new to Black's codebase
If lib2to3 has been updated, you should feel free to reuse aspects of the update, but I'm not sure CPython is still updating it.

I don't know blib2to3 too well but it sounds like we should just add to the grammar, similar to what we've done for other new grammar features like the walrus operator in the past. We'll also likely need to add to the Feature enum in https://github.com/psf/black/blob/master/src/black/__init__.py#L189. That can be used so that Black can auto-detect Python 3.9-only code.

Thanks for working on this!
That's right, I can draw inspiration from the walrus operator support. I think I should also add a detection of relaxed decorators in [`get_features_used`](https://github.com/psf/black/blob/master/src/black/__init__.py#L5679) ?
Quick questions:
- is `RELAXED_DECORATORS` a suitable feature name ?
- should I add the new `PY39` target to the [`PY36_VERSIONS`](https://github.com/psf/black/blob/master/src/black/__init__.py#L187) constant ?
Yes to all your questions. 
Perhaps also rename `PY36_VERSIONS`, whilst Black is still in pre-stable? The name is probably a remnant of the replaced and removed `--py36` option.

In fact, it's only used by tests:

```console
$ git grep PY36_VERSIONS
src/black/__init__.py:PY36_VERSIONS = {TargetVersion.PY36, TargetVersion.PY37, TargetVersion.PY38}
tests/test_black.py:    f"--target-version={version.name.lower()}" for version in black.PY36_VERSIONS
tests/test_black.py:        mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
tests/test_black.py:        mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
tests/test_black.py:        py36_mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
tests/test_black.py:        py36_mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
```

Does it need to be part of the public API, or be underscore prefixed?
Good catch, we should just remove it from the code, and define it in test_black.py if we still need it there.
I moved `PY36_VERSIONS` to `tests/test_black.py` and started support for PEP 614. Turns out detecting old decorator syntax while the grammar accepts the new syntax isn't that easy. I'd like to have some input from people more used to that grammar, I have surely forgotten some use-case.

Would pushing to my fork multiple commits (and perhaps rebasing at the end so that there is only a single pretty commit to PR) be appropriate ? I see that there are quite some pre-commit hooks, and since I'm not used to that I don't want to mess up the project history
It's fine to prepare your PR in whatever way you want. The precommit hook are just there as a way to help people writing code, and you're free to not use them if you prefer (I personally like to commit early and fast, so I don't like the friction pre-commit hooks add). As long as CI is happy when you put up your PR, we're happy.

Are you referring to how `get_features_used()` should work? I haven't thought too much about it but I think you can just detect whether there's a node that contains `@` followed by something previously illegal, like an index expression. It's also not as important to get that function right as it is to make sure Black can format files that use the new syntax, so we can also just punt `get_features_used()` to later.

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -178,14 +178,12 @@ class TargetVersion(Enum):
     PY36 = 6
     PY37 = 7
     PY38 = 8
+    PY39 = 9
 
     def is_python2(self) -> bool:
         return self is TargetVersion.PY27
 
 
-PY36_VERSIONS = {TargetVersion.PY36, TargetVersion.PY37, TargetVersion.PY38}
-
-
 class Feature(Enum):
     # All string literals are unicode
     UNICODE_LITERALS = 1
@@ -199,6 +197,7 @@ class Feature(Enum):
     ASYNC_KEYWORDS = 7
     ASSIGNMENT_EXPRESSIONS = 8
     POS_ONLY_ARGUMENTS = 9
+    RELAXED_DECORATORS = 10
     FORCE_OPTIONAL_PARENTHESES = 50
 
 
@@ -237,6 +236,17 @@ class Feature(Enum):
         Feature.ASSIGNMENT_EXPRESSIONS,
         Feature.POS_ONLY_ARGUMENTS,
     },
+    TargetVersion.PY39: {
+        Feature.UNICODE_LITERALS,
+        Feature.F_STRINGS,
+        Feature.NUMERIC_UNDERSCORES,
+        Feature.TRAILING_COMMA_IN_CALL,
+        Feature.TRAILING_COMMA_IN_DEF,
+        Feature.ASYNC_KEYWORDS,
+        Feature.ASSIGNMENT_EXPRESSIONS,
+        Feature.RELAXED_DECORATORS,
+        Feature.POS_ONLY_ARGUMENTS,
+    },
 }
 
 
@@ -2184,6 +2194,9 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool) -> str:  # noqa: C901
         ):
             # Python 2 print chevron
             return NO
+        elif prevp.type == token.AT and p.parent and p.parent.type == syms.decorator:
+            # no space in decorators
+            return NO
 
     elif prev.type in OPENING_BRACKETS:
         return NO
@@ -5499,6 +5512,49 @@ def is_walrus_assignment(node: LN) -> bool:
     return inner is not None and inner.type == syms.namedexpr_test
 
 
+def is_simple_decorator_trailer(node: LN, last: bool = False) -> bool:
+    """Return True iff `node` is a trailer valid in a simple decorator"""
+    return node.type == syms.trailer and (
+        (
+            len(node.children) == 2
+            and node.children[0].type == token.DOT
+            and node.children[1].type == token.NAME
+        )
+        # last trailer can be arguments
+        or (
+            last
+            and len(node.children) == 3
+            and node.children[0].type == token.LPAR
+            # and node.children[1].type == syms.argument
+            and node.children[2].type == token.RPAR
+        )
+    )
+
+
+def is_simple_decorator_expression(node: LN) -> bool:
+    """Return True iff `node` could be a 'dotted name' decorator
+
+    This function takes the node of the 'namedexpr_test' of the new decorator
+    grammar and test if it would be valid under the old decorator grammar.
+
+    The old grammar was: decorator: @ dotted_name [arguments] NEWLINE
+    The new grammar is : decorator: @ namedexpr_test NEWLINE
+    """
+    if node.type == token.NAME:
+        return True
+    if node.type == syms.power:
+        if node.children:
+            return (
+                node.children[0].type == token.NAME
+                and all(map(is_simple_decorator_trailer, node.children[1:-1]))
+                and (
+                    len(node.children) < 2
+                    or is_simple_decorator_trailer(node.children[-1], last=True)
+                )
+            )
+    return False
+
+
 def is_yield(node: LN) -> bool:
     """Return True if `node` holds a `yield` or `yield from` expression."""
     if node.type == syms.yield_expr:
@@ -5684,6 +5740,8 @@ def get_features_used(node: Node) -> Set[Feature]:
     - underscores in numeric literals;
     - trailing commas after * or ** in function signatures and calls;
     - positional only arguments in function signatures and lambdas;
+    - assignment expression;
+    - relaxed decorator syntax;
     """
     features: Set[Feature] = set()
     for n in node.pre_order():
@@ -5703,6 +5761,12 @@ def get_features_used(node: Node) -> Set[Feature]:
         elif n.type == token.COLONEQUAL:
             features.add(Feature.ASSIGNMENT_EXPRESSIONS)
 
+        elif n.type == syms.decorator:
+            if len(n.children) > 1 and not is_simple_decorator_expression(
+                n.children[1]
+            ):
+                features.add(Feature.RELAXED_DECORATORS)
+
         elif (
             n.type in {syms.typedargslist, syms.arglist}
             and n.children


Test patch:
diff --git a/tests/data/decorators.py b/tests/data/decorators.py
new file mode 100644
--- /dev/null
+++ b/tests/data/decorators.py
@@ -0,0 +1,176 @@
+# This file doesn't use the standard decomposition.
+# Decorator syntax test cases are separated by double # comments.
+# Those before the 'output' comment are valid under the old syntax.
+# Those after the 'ouput' comment require PEP614 relaxed syntax.
+# Do not remove the double # separator before the first test case, it allows
+# the comment before the test case to be ignored.
+
+##
+
+@decorator
+def f():
+    ...
+
+##
+
+@decorator(arg)
+def f():
+    ...
+    
+##
+
+@decorator(kwarg=0)
+def f():
+    ...
+
+##
+
+@decorator(*args)
+def f():
+    ...
+
+##
+
+@decorator(**kwargs)
+def f():
+    ...
+
+##
+
+@decorator(*args, **kwargs)
+def f():
+    ...
+
+##
+
+@decorator(*args, **kwargs,)
+def f():
+    ...
+
+##
+
+@dotted.decorator
+def f():
+    ...
+
+##
+
+@dotted.decorator(arg)
+def f():
+    ...
+    
+##
+
+@dotted.decorator(kwarg=0)
+def f():
+    ...
+
+##
+
+@dotted.decorator(*args)
+def f():
+    ...
+
+##
+
+@dotted.decorator(**kwargs)
+def f():
+    ...
+
+##
+
+@dotted.decorator(*args, **kwargs)
+def f():
+    ...
+
+##
+
+@dotted.decorator(*args, **kwargs,)
+def f():
+    ...
+
+##
+
+@double.dotted.decorator
+def f():
+    ...
+
+##
+
+@double.dotted.decorator(arg)
+def f():
+    ...
+    
+##
+
+@double.dotted.decorator(kwarg=0)
+def f():
+    ...
+
+##
+
+@double.dotted.decorator(*args)
+def f():
+    ...
+
+##
+
+@double.dotted.decorator(**kwargs)
+def f():
+    ...
+
+##
+
+@double.dotted.decorator(*args, **kwargs)
+def f():
+    ...
+
+##
+
+@double.dotted.decorator(*args, **kwargs,)
+def f():
+    ...
+
+##
+
+@_(sequence["decorator"])
+def f():
+    ...
+
+##
+
+@eval("sequence['decorator']")
+def f():
+    ...
+
+# output
+
+##
+
+@decorator()()
+def f():
+    ...
+
+##
+
+@(decorator)
+def f():
+    ...
+
+##
+
+@sequence["decorator"]
+def f():
+    ...
+
+##
+
+@decorator[List[str]]
+def f():
+    ...
+
+##
+
+@var := decorator
+def f():
+    ...
\ No newline at end of file
diff --git a/tests/data/python39.py b/tests/data/python39.py
new file mode 100644
--- /dev/null
+++ b/tests/data/python39.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3.9
+
+@relaxed_decorator[0]
+def f():
+    ...
+
+@relaxed_decorator[extremely_long_name_that_definitely_will_not_fit_on_one_line_of_standard_length]
+def f():
+    ...
+
+@extremely_long_variable_name_that_doesnt_fit := complex.expression(with_long="arguments_value_that_wont_fit_at_the_end_of_the_line")
+def f():
+    ...
+
+# output
+
+
+#!/usr/bin/env python3.9
+
+
+@relaxed_decorator[0]
+def f():
+    ...
+
+
+@relaxed_decorator[
+    extremely_long_name_that_definitely_will_not_fit_on_one_line_of_standard_length
+]
+def f():
+    ...
+
+
+@extremely_long_variable_name_that_doesnt_fit := complex.expression(
+    with_long="arguments_value_that_wont_fit_at_the_end_of_the_line"
+)
+def f():
+    ...
\ No newline at end of file
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -59,9 +59,13 @@
 PROJECT_ROOT = THIS_DIR.parent
 DETERMINISTIC_HEADER = "[Deterministic header]"
 EMPTY_LINE = "# EMPTY LINE WITH WHITESPACE" + " (this comment will be removed)"
-PY36_ARGS = [
-    f"--target-version={version.name.lower()}" for version in black.PY36_VERSIONS
-]
+PY36_VERSIONS = {
+    TargetVersion.PY36,
+    TargetVersion.PY37,
+    TargetVersion.PY38,
+    TargetVersion.PY39,
+}
+PY36_ARGS = [f"--target-version={version.name.lower()}" for version in PY36_VERSIONS]
 T = TypeVar("T")
 R = TypeVar("R")
 
@@ -705,7 +709,7 @@ def test_string_prefixes(self) -> None:
     @patch("black.dump_to_file", dump_to_stderr)
     def test_numeric_literals(self) -> None:
         source, expected = read_data("numeric_literals")
-        mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
+        mode = replace(DEFAULT_MODE, target_versions=PY36_VERSIONS)
         actual = fs(source, mode=mode)
         self.assertFormatEqual(expected, actual)
         black.assert_equivalent(source, actual)
@@ -714,7 +718,7 @@ def test_numeric_literals(self) -> None:
     @patch("black.dump_to_file", dump_to_stderr)
     def test_numeric_literals_ignoring_underscores(self) -> None:
         source, expected = read_data("numeric_literals_skip_underscores")
-        mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
+        mode = replace(DEFAULT_MODE, target_versions=PY36_VERSIONS)
         actual = fs(source, mode=mode)
         self.assertFormatEqual(expected, actual)
         black.assert_equivalent(source, actual)
@@ -800,6 +804,16 @@ def test_python38(self) -> None:
             black.assert_equivalent(source, actual)
         black.assert_stable(source, actual, DEFAULT_MODE)
 
+    @patch("black.dump_to_file", dump_to_stderr)
+    def test_python39(self) -> None:
+        source, expected = read_data("python39")
+        actual = fs(source)
+        self.assertFormatEqual(expected, actual)
+        major, minor = sys.version_info[:2]
+        if major > 3 or (major == 3 and minor >= 9):
+            black.assert_equivalent(source, actual)
+        black.assert_stable(source, actual, DEFAULT_MODE)
+
     @patch("black.dump_to_file", dump_to_stderr)
     def test_fmtonoff(self) -> None:
         source, expected = read_data("fmtonoff")
@@ -1212,6 +1226,39 @@ def test_lib2to3_parse(self) -> None:
         black.lib2to3_parse(py3_only, {TargetVersion.PY36})
         black.lib2to3_parse(py3_only, {TargetVersion.PY27, TargetVersion.PY36})
 
+    def test_get_features_used_decorator(self) -> None:
+        # Test the feature detection of new decorator syntax
+        # since this makes some test cases of test_get_features_used()
+        # fails if it fails, this is tested first so that a useful case
+        # is identified
+        simples, relaxed = read_data("decorators")
+        # skip explanation comments at the top of the file
+        for simple_test in simples.split("##")[1:]:
+            node = black.lib2to3_parse(simple_test)
+            decorator = str(node.children[0].children[0]).strip()
+            self.assertNotIn(
+                Feature.RELAXED_DECORATORS,
+                black.get_features_used(node),
+                msg=(
+                    f"decorator '{decorator}' follows python<=3.8 syntax"
+                    "but is detected as 3.9+"
+                    # f"The full node is\n{node!r}"
+                ),
+            )
+        # skip the '# output' comment at the top of the output part
+        for relaxed_test in relaxed.split("##")[1:]:
+            node = black.lib2to3_parse(relaxed_test)
+            decorator = str(node.children[0].children[0]).strip()
+            self.assertIn(
+                Feature.RELAXED_DECORATORS,
+                black.get_features_used(node),
+                msg=(
+                    f"decorator '{decorator}' uses python3.9+ syntax"
+                    "but is detected as python<=3.8"
+                    # f"The full node is\n{node!r}"
+                ),
+            )
+
     def test_get_features_used(self) -> None:
         node = black.lib2to3_parse("def f(*, arg): ...\n")
         self.assertEqual(black.get_features_used(node), set())
@@ -1628,7 +1675,7 @@ def test_pipe_force_pyi(self) -> None:
 
     def test_single_file_force_py36(self) -> None:
         reg_mode = DEFAULT_MODE
-        py36_mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
+        py36_mode = replace(DEFAULT_MODE, target_versions=PY36_VERSIONS)
         source, expected = read_data("force_py36")
         with cache_dir() as workspace:
             path = (workspace / "file.py").resolve()
@@ -1647,7 +1694,7 @@ def test_single_file_force_py36(self) -> None:
     @event_loop()
     def test_multi_file_force_py36(self) -> None:
         reg_mode = DEFAULT_MODE
-        py36_mode = replace(DEFAULT_MODE, target_versions=black.PY36_VERSIONS)
+        py36_mode = replace(DEFAULT_MODE, target_versions=PY36_VERSIONS)
         source, expected = read_data("force_py36")
         with cache_dir() as workspace:
             paths = [