Repository: psf/black

Issue:
fmt: skip for long line of context managers leads to INTERNAL ERROR: Black produced different code on the second pass of the formatter.
Adding `# fmt: skip` after a long `with ... as ...:` line leads to black raising an internal error.

**To Reproduce** Steps to reproduce the behavior:

1. Take this file `foo.py`:
    ```python
    name = "foo.py"


    def foo():
        with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(name) as f5:  # fmt: skip
            print(f1, f2, f3, f4, f5)
    ```
2. Run _Black_ on it with these arguments `black foo.py`
3. See error: `error: cannot format foo.py: INTERNAL ERROR: Black produced different code on the second pass of the formatter.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_lejlacgg.log`
4. The helpful diff (attached)

**Expected behavior** Black formats the source file, leaving the line marked to be skipped alone.

**Environment (please complete the following information):**

- Version: black, version 21.4b0
- OS and Python version: Linux, Python 3.9.4

**Does this bug also happen on master?** [Yes it does](https://black.vercel.app/?version=master&state=_Td6WFoAAATm1rRGAgAhARYAAAB0L-Wj4AEEAJFdAD2IimZxl1N_WmFm63wsQYxa3iKd5NgD9wTuCPA8tXPZto6Z99Hjctu-G04oaNFwP0prhun9D_WwMRE2qvt9XrGWUmhvMlrPsdHF8rb6YbTjYiezBhLlkOtY8bqbawGuKbY8MsSA1n6NE-6hT0_Pz-gdHZUSmcScrbgrszUMecOfLe-OfEh6y2WRdiiwisWNhecAAAAAjHfk5WJS8PcAAa0BhQIAADpttIKxxGf7AgAAAAAEWVo=)

**Additional context** Add any other context about the problem here.

"fmt: skip" on colon-terminated lines either fails or moves the directive
<!--
Please make sure that the bug is not already fixed either in newer versions or the
current development version. To confirm this, you have three options:

1. Update Black's version if a newer release exists: `pip install -U black`
2. Use the online formatter at <https://black.vercel.app/?version=main>, which will use
   the latest main branch.
3. Or run _Black_ on your machine:
   - create a new virtualenv (make sure it's the same Python version);
   - clone this repository;
   - run `pip install -e .[d,python2]`;
   - run `pip install -r test_requirements.txt`
   - make sure it's sane by running `python -m pytest`; and
   - run `black` like you did last time.
-->

**Describe the bug**

Adding `# fmt: skip` after a colon (e.g. on the same line as `def`, `if`, `with` or `class`) either produces the error "Black produced different code on the second pass of the formatter." or moves the `# fmt: skip` directive below the suite (with a 2-space indent, as well as inserting a blank line between the statement introduction and the suite).

<!-- A clear and concise description of what the bug is. -->

**To Reproduce**

<!--
Minimal steps to reproduce the behavior with source code and Black's configuration.
-->

For example, take this code:

```python
def f():  # fmt: skip
    pass
```

And run it with these arguments:

```sh
$ black file.py
```

The resulting error is:

> error: cannot format test.py: INTERNAL ERROR: Black produced different code on the second pass of the formatter.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_cwq0dubx.log

```diff
Mode(target_versions=set(), line_length=88, string_normalization=True, is_pyi=False, is_ipynb=False, magic_trailing_comma=True, experimental_string_processing=False)
--- source
+++ first pass
@@ -1,2 +1,6 @@
-def f(): # fmt: skip
+def f():
+
     pass
+
+
+# fmt: skip
--- first pass
+++ second pass
@@ -1,6 +1,4 @@
 def f():
 
     pass
-
-
-# fmt: skip
+  # fmt: skip
```

On the other hand, with `if` there is no internal error:

```python
if True: # fmt: skip
    pass
```

But it does become:

```python
if True:

    pass
  # fmt: skip
```

Which is not what I expected. `with` and `for` behave like `if`, `class` behaves like `def`.

**Expected behavior**

In all cases I expected black to exit without error and for the `fmt: skip` directive to remain on the same line.

**Environment**

<!-- Please complete the following information: -->

- Black's version: <!-- e.g. [main] --> black, 21.11b2.dev6+g17e42cb (compiled: no)
- OS and Python version: <!-- e.g. [Linux/Python 3.7.4rc1] --> Debian Bullseye/ Python 3.9.2



Discussion:
I'm failing to attach the helpful diff so here it is inline:
```diff
Mode(target_versions=set(), line_length=88, string_normalization=True, magic_trailing_comma=True, experimental_string_processing=False, is_pyi=False)
--- source
+++ first pass
@@ -1,6 +1,12 @@
 name = "foo.py"
 
 
 def foo():
-    with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(name) as f5:  # fmt: skip
+    with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(
+        name
+    ) as f5:
+
         print(f1, f2, f3, f4, f5)
+
+
+# fmt: skip
--- first pass
+++ second pass
@@ -5,8 +5,6 @@
     with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(
         name
     ) as f5:
 
         print(f1, f2, f3, f4, f5)
-
-
-# fmt: skip
+  # fmt: skip
```
We would also appreciate this fix 
This no longer crashes but now it puts `# fmt: skip` on its own line, which isn't great.

```
 def foo():
-    with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(name) as f5:  # fmt: skip
+    with open(name) as f1, open(name) as f2, open(name) as f3, open(name) as f4, open(
+        name
+    ) as f5:
+
         print(f1, f2, f3, f4, f5)
+
+
+# fmt: skip
```
My naive attempt to work around this using `fmt: off` and `fmt: on` produced a different error, which may be related:

```python
class C:
    # fmt: off
    def f(self):
    # fmt: on
        pass


def g():
    pass
```

> INTERNAL ERROR: Black produced code that is not equivalent to the source on pass 1.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_c_aho8k6.log

```diff
--- src
+++ dst
@@ -25,36 +25,36 @@
           returns=
             None,  # NoneType
           type_comment=
             None,  # NoneType
         )  # /FunctionDef
+        FunctionDef(
+          args=
+            arguments(
+              args=
+              defaults=
+              kw_defaults=
+              kwarg=
+                None,  # NoneType
+              kwonlyargs=
+              posonlyargs=
+              vararg=
+                None,  # NoneType
+            )  # /arguments
+          body=
+            Pass(
+            )  # /Pass
+          decorator_list=
+          name=
+            'g',  # str
+          returns=
+            None,  # NoneType
+          type_comment=
+            None,  # NoneType
+        )  # /FunctionDef
       decorator_list=
       keywords=
       name=
         'C',  # str
     )  # /ClassDef
-    FunctionDef(
-      args=
-        arguments(
-          args=
-          defaults=
-          kw_defaults=
-          kwarg=
-            None,  # NoneType
-          kwonlyargs=
-          posonlyargs=
-          vararg=
-            None,  # NoneType
-        )  # /arguments
-      body=
-        Pass(
-        )  # /Pass
-      decorator_list=
-      name=
-        'g',  # str
-      returns=
-        None,  # NoneType
-      type_comment=
-        None,  # NoneType
-    )  # /FunctionDef
   type_ignores=
 )  # /Module
\ No newline at end of file
```

I was again hoping that nothing would be changed successfully, although I acknowledge that `fmt: off` and `fmt: on` may not be intended to work like that.

EDIT: the following does remain successfully unchanged, but of course it disables formatting for the function body, which is not what I want.

```python
class C:
    # fmt: off
    def f(self):
        pass
    # fmt: on


def g():
    pass
```
Looks like this Issue is a duplicate of #2421.

I just ran into this bug too and it is pretty annoying. Is there any plan to fix this?
If no one is currently looking at this, i'll work on a fix and send a PR.

Resolving patch:
diff --git a/src/black/comments.py b/src/black/comments.py
--- a/src/black/comments.py
+++ b/src/black/comments.py
@@ -9,7 +9,7 @@
 else:
     from typing_extensions import Final
 
-from blib2to3.pytree import Node, Leaf
+from blib2to3.pytree import Node, Leaf, type_repr
 from blib2to3.pgen2 import token
 
 from black.nodes import first_leaf_column, preceding_leaf, container_of
@@ -174,6 +174,11 @@ def convert_one_fmt_off_pair(node: Node, *, preview: bool) -> bool:
                 first.prefix = prefix[comment.consumed :]
             if comment.value in FMT_SKIP:
                 first.prefix = ""
+                standalone_comment_prefix = prefix
+            else:
+                standalone_comment_prefix = (
+                    prefix[:previous_consumed] + "\n" * comment.newlines
+                )
             hidden_value = "".join(str(n) for n in ignored_nodes)
             if comment.value in FMT_OFF:
                 hidden_value = comment.value + "\n" + hidden_value
@@ -195,7 +200,7 @@ def convert_one_fmt_off_pair(node: Node, *, preview: bool) -> bool:
                 Leaf(
                     STANDALONE_COMMENT,
                     hidden_value,
-                    prefix=prefix[:previous_consumed] + "\n" * comment.newlines,
+                    prefix=standalone_comment_prefix,
                 ),
             )
             return True
@@ -211,26 +216,10 @@ def generate_ignored_nodes(
     If comment is skip, returns leaf only.
     Stops at the end of the block.
     """
-    container: Optional[LN] = container_of(leaf)
     if comment.value in FMT_SKIP:
-        prev_sibling = leaf.prev_sibling
-        # Need to properly format the leaf prefix to compare it to comment.value,
-        # which is also formatted
-        comments = list_comments(leaf.prefix, is_endmarker=False, preview=preview)
-        if comments and comment.value == comments[0].value and prev_sibling is not None:
-            leaf.prefix = ""
-            siblings = [prev_sibling]
-            while (
-                "\n" not in prev_sibling.prefix
-                and prev_sibling.prev_sibling is not None
-            ):
-                prev_sibling = prev_sibling.prev_sibling
-                siblings.insert(0, prev_sibling)
-            for sibling in siblings:
-                yield sibling
-        elif leaf.parent is not None:
-            yield leaf.parent
+        yield from _generate_ignored_nodes_from_fmt_skip(leaf, comment, preview=preview)
         return
+    container: Optional[LN] = container_of(leaf)
     while container is not None and container.type != token.ENDMARKER:
         if is_fmt_on(container, preview=preview):
             return
@@ -246,6 +235,51 @@ def generate_ignored_nodes(
             container = container.next_sibling
 
 
+def _generate_ignored_nodes_from_fmt_skip(
+    leaf: Leaf, comment: ProtoComment, *, preview: bool
+) -> Iterator[LN]:
+    """Generate all leaves that should be ignored by the `# fmt: skip` from `leaf`."""
+    prev_sibling = leaf.prev_sibling
+    parent = leaf.parent
+    # Need to properly format the leaf prefix to compare it to comment.value,
+    # which is also formatted
+    comments = list_comments(leaf.prefix, is_endmarker=False, preview=preview)
+    if not comments or comment.value != comments[0].value:
+        return
+    if prev_sibling is not None:
+        leaf.prefix = ""
+        siblings = [prev_sibling]
+        while "\n" not in prev_sibling.prefix and prev_sibling.prev_sibling is not None:
+            prev_sibling = prev_sibling.prev_sibling
+            siblings.insert(0, prev_sibling)
+        for sibling in siblings:
+            yield sibling
+    elif (
+        parent is not None
+        and type_repr(parent.type) == "suite"
+        and leaf.type == token.NEWLINE
+    ):
+        # The `# fmt: skip` is on the colon line of the if/while/def/class/...
+        # statements. The ignored nodes should be previous siblings of the
+        # parent suite node.
+        leaf.prefix = ""
+        ignored_nodes: List[LN] = []
+        parent_sibling = parent.prev_sibling
+        while parent_sibling is not None and type_repr(parent_sibling.type) != "suite":
+            ignored_nodes.insert(0, parent_sibling)
+            parent_sibling = parent_sibling.prev_sibling
+        # Special case for `async_stmt` where the ASYNC token is on the
+        # grandparent node.
+        grandparent = parent.parent
+        if (
+            grandparent is not None
+            and grandparent.prev_sibling is not None
+            and grandparent.prev_sibling.type == token.ASYNC
+        ):
+            ignored_nodes.insert(0, grandparent.prev_sibling)
+        yield from iter(ignored_nodes)
+
+
 def is_fmt_on(container: LN, preview: bool) -> bool:
     """Determine whether formatting is switched on within a container.
     Determined by whether the last `# fmt:` comment is `on` or `off`.
diff --git a/src/black/linegen.py b/src/black/linegen.py
--- a/src/black/linegen.py
+++ b/src/black/linegen.py
@@ -220,7 +220,9 @@ def visit_async_stmt(self, node: Node) -> Iterator[Line]:
         for child in children:
             yield from self.visit(child)
 
-            if child.type == token.ASYNC:
+            if child.type == token.ASYNC or child.type == STANDALONE_COMMENT:
+                # STANDALONE_COMMENT happens when `# fmt: skip` is applied on the async
+                # line.
                 break
 
         internal_stmt = next(children)


Test patch:
diff --git a/tests/data/simple_cases/fmtskip8.py b/tests/data/simple_cases/fmtskip8.py
new file mode 100644
--- /dev/null
+++ b/tests/data/simple_cases/fmtskip8.py
@@ -0,0 +1,62 @@
+# Make sure a leading comment is not removed.
+def some_func(  unformatted,  args  ):  # fmt: skip
+    print("I am some_func")
+    return 0
+    # Make sure this comment is not removed.
+
+
+# Make sure a leading comment is not removed.
+async def some_async_func(  unformatted,   args):  # fmt: skip
+    print("I am some_async_func")
+    await asyncio.sleep(1)
+
+
+# Make sure a leading comment is not removed.
+class SomeClass(  Unformatted,  SuperClasses  ):  # fmt: skip
+    def some_method(  self,  unformatted,  args  ):  # fmt: skip
+        print("I am some_method")
+        return 0
+
+    async def some_async_method(  self,  unformatted,  args  ):  # fmt: skip
+        print("I am some_async_method")
+        await asyncio.sleep(1)
+
+
+# Make sure a leading comment is not removed.
+if  unformatted_call(  args  ):  # fmt: skip
+    print("First branch")
+    # Make sure this is not removed.
+elif  another_unformatted_call(  args  ):  # fmt: skip
+    print("Second branch")
+else  :  # fmt: skip
+    print("Last branch")
+
+
+while  some_condition(  unformatted,  args  ):  # fmt: skip
+    print("Do something")
+
+
+for  i  in  some_iter(  unformatted,  args  ):  # fmt: skip
+    print("Do something")
+
+
+async def test_async_for():
+    async  for  i  in  some_async_iter(  unformatted,  args  ):  # fmt: skip
+        print("Do something")
+
+
+try  :  # fmt: skip
+    some_call()
+except  UnformattedError  as  ex:  # fmt: skip
+    handle_exception()
+finally  :  # fmt: skip
+    finally_call()
+
+
+with  give_me_context(  unformatted,  args  ):  # fmt: skip
+    print("Do something")
+
+
+async def test_async_with():
+    async  with  give_me_async_context(  unformatted,  args  ):  # fmt: skip
+        print("Do something")