Repository: psf/black
Pull request: dont skip formatting #%%
<!-- Hello! Thanks for submitting a PR. To help make things go a bit more
     smoothly we would appreciate that you go through this template. -->

### Description

<!-- Good things to put here include: reasoning for the change (please link
     any relevant issues!), any noteworthy (or hacky) choices to be aware of,
     or what the problem resolved here looked like ... we won't mind a ranty
     story :) -->

### Checklist - did you ...

<!-- If any of the following items aren't relevant for your contribution
     please still tick them so we know you've gone through the checklist.

    All user-facing changes should get an entry. Otherwise, signal to us
    this should get the magical label to silence the CHANGELOG entry check.
    Tests are required for bugfixes and new features. Documentation changes
    are necessary for formatting and most enhancement changes. -->

- [x] Add a CHANGELOG entry if necessary?
- [x] Add / update tests if necessary?
- [x] Add new / update outdated documentation?

<!-- Just as a reminder, everyone in all psf/black spaces including PRs
     must follow the PSF Code of Conduct (link below).

     Finally, once again thanks for your time and effort. If you have any
     feedback in regards to your experience contributing here, please
     let us know!

     Helpful links:

      PSF COC: https://www.python.org/psf/conduct/
      Contributing docs: https://black.readthedocs.io/en/latest/contributing/index.html
      Chat on Python Discord: https://discord.gg/RtVdv86PrH -->

closes #2588 

Problem statement:
black didn't format the cell note
**Describe the style change**

<!-- A clear and concise description of how the style can be
improved. -->

**Examples in the current _Black_ style**

<!-- Think of some short code snippets that show
how the current _Black_ style is not great: -->

```python
#%% this is a cell
a = 12
```

**Desired style**

<!-- How do you think _Black_ should format the above snippets: -->

```python
# %% this is a cell
a = 12

```

**Additional context**

<!-- Add any other context about the problem here. -->
I found black can't format the cell note.

Hints from issue discussion:
Looks like this is intentional from https://github.com/psf/black/commit/c6c8ef76a4d991cd8a7a3b00a816b6db72ecdf04 , as `#%%` has a special meaning in the Spyder IDE
I use vscode to write my code, and in vscode, the default note of cell is "# %%", rather than "#%%". And in spyder, "# %%" also can be distinguished as the begin of the cell. 
Whether it is better  to change the role?
I am a freshman, and could you tell me more about the difference among "##" , "#!" , "#:" and "#'"?
Asking around it seems like Spyder _may have_ added support for `# %%`, `# %%%` and so on since we added that special case linked above. If that's the case we might want to remove it and the comment be reformatted normally. I'd appreciate comments from people who use Spyder though!
Did someone already fix this?
Not yet - I'll make a PR now, from https://docs.spyder-ide.org/4/panes/editor.html it seems like this is safe to do anyway as they accept both `# %%` and `#%%`

Commit message: use self.mode.preview

Patch to review:
diff --git a/CHANGES.md b/CHANGES.md
index edca0dcdad4..dad60053a88 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -14,6 +14,8 @@
 
 <!-- Changes that affect Black's preview style -->
 
+- Code cell separators `#%%` are now standardised to `# %%` (#2919)
+
 ### _Blackd_
 
 <!-- Changes to blackd -->
diff --git a/src/black/__init__.py b/src/black/__init__.py
index c4ec99b441f..51e31e9598b 100644
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -1166,7 +1166,7 @@ def _format_str_once(src_contents: str, *, mode: Mode) -> str:
     else:
         versions = detect_target_versions(src_node, future_imports=future_imports)
 
-    normalize_fmt_off(src_node)
+    normalize_fmt_off(src_node, preview=mode.preview)
     lines = LineGenerator(mode=mode)
     elt = EmptyLineTracker(is_pyi=mode.is_pyi)
     empty_line = Line(mode=mode)
diff --git a/src/black/comments.py b/src/black/comments.py
index 28b9117101d..455326469f0 100644
--- a/src/black/comments.py
+++ b/src/black/comments.py
@@ -23,6 +23,8 @@
 FMT_PASS: Final = {*FMT_OFF, *FMT_SKIP}
 FMT_ON: Final = {"# fmt: on", "# fmt:on", "# yapf: enable"}
 
+COMMENT_EXCEPTIONS = {True: " !:#'", False: " !:#'%"}
+
 
 @dataclass
 class ProtoComment:
@@ -42,7 +44,7 @@ class ProtoComment:
     consumed: int  # how many characters of the original leaf's prefix did we consume
 
 
-def generate_comments(leaf: LN) -> Iterator[Leaf]:
+def generate_comments(leaf: LN, *, preview: bool) -> Iterator[Leaf]:
     """Clean the prefix of the `leaf` and generate comments from it, if any.
 
     Comments in lib2to3 are shoved into the whitespace prefix.  This happens
@@ -61,12 +63,16 @@ def generate_comments(leaf: LN) -> Iterator[Leaf]:
     Inline comments are emitted as regular token.COMMENT leaves.  Standalone
     are emitted with a fake STANDALONE_COMMENT token identifier.
     """
-    for pc in list_comments(leaf.prefix, is_endmarker=leaf.type == token.ENDMARKER):
+    for pc in list_comments(
+        leaf.prefix, is_endmarker=leaf.type == token.ENDMARKER, preview=preview
+    ):
         yield Leaf(pc.type, pc.value, prefix="\n" * pc.newlines)
 
 
 @lru_cache(maxsize=4096)
-def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
+def list_comments(
+    prefix: str, *, is_endmarker: bool, preview: bool
+) -> List[ProtoComment]:
     """Return a list of :class:`ProtoComment` objects parsed from the given `prefix`."""
     result: List[ProtoComment] = []
     if not prefix or "#" not in prefix:
@@ -92,7 +98,7 @@ def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
             comment_type = token.COMMENT  # simple trailing comment
         else:
             comment_type = STANDALONE_COMMENT
-        comment = make_comment(line)
+        comment = make_comment(line, preview=preview)
         result.append(
             ProtoComment(
                 type=comment_type, value=comment, newlines=nlines, consumed=consumed
@@ -102,10 +108,10 @@ def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
     return result
 
 
-def make_comment(content: str) -> str:
+def make_comment(content: str, *, preview: bool) -> str:
     """Return a consistently formatted comment from the given `content` string.
 
-    All comments (except for "##", "#!", "#:", '#'", "#%%") should have a single
+    All comments (except for "##", "#!", "#:", '#'") should have a single
     space between the hash sign and the content.
 
     If `content` didn't start with a hash sign, one is provided.
@@ -123,26 +129,26 @@ def make_comment(content: str) -> str:
         and not content.lstrip().startswith("type:")
     ):
         content = " " + content[1:]  # Replace NBSP by a simple space
-    if content and content[0] not in " !:#'%":
+    if content and content[0] not in COMMENT_EXCEPTIONS[preview]:
         content = " " + content
     return "#" + content
 
 
-def normalize_fmt_off(node: Node) -> None:
+def normalize_fmt_off(node: Node, *, preview: bool) -> None:
     """Convert content between `# fmt: off`/`# fmt: on` into standalone comments."""
     try_again = True
     while try_again:
-        try_again = convert_one_fmt_off_pair(node)
+        try_again = convert_one_fmt_off_pair(node, preview=preview)
 
 
-def convert_one_fmt_off_pair(node: Node) -> bool:
+def convert_one_fmt_off_pair(node: Node, *, preview: bool) -> bool:
     """Convert content of a single `# fmt: off`/`# fmt: on` into a standalone comment.
 
     Returns True if a pair was converted.
     """
     for leaf in node.leaves():
         previous_consumed = 0
-        for comment in list_comments(leaf.prefix, is_endmarker=False):
+        for comment in list_comments(leaf.prefix, is_endmarker=False, preview=preview):
             if comment.value not in FMT_PASS:
                 previous_consumed = comment.consumed
                 continue
@@ -157,7 +163,7 @@ def convert_one_fmt_off_pair(node: Node) -> bool:
                     if comment.value in FMT_SKIP and prev.type in WHITESPACE:
                         continue
 
-            ignored_nodes = list(generate_ignored_nodes(leaf, comment))
+            ignored_nodes = list(generate_ignored_nodes(leaf, comment, preview=preview))
             if not ignored_nodes:
                 continue
 
@@ -197,7 +203,9 @@ def convert_one_fmt_off_pair(node: Node) -> bool:
     return False
 
 
-def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
+def generate_ignored_nodes(
+    leaf: Leaf, comment: ProtoComment, *, preview: bool
+) -> Iterator[LN]:
     """Starting from the container of `leaf`, generate all leaves until `# fmt: on`.
 
     If comment is skip, returns leaf only.
@@ -221,13 +229,13 @@ def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
             yield leaf.parent
         return
     while container is not None and container.type != token.ENDMARKER:
-        if is_fmt_on(container):
+        if is_fmt_on(container, preview=preview):
             return
 
         # fix for fmt: on in children
-        if contains_fmt_on_at_column(container, leaf.column):
+        if contains_fmt_on_at_column(container, leaf.column, preview=preview):
             for child in container.children:
-                if contains_fmt_on_at_column(child, leaf.column):
+                if contains_fmt_on_at_column(child, leaf.column, preview=preview):
                     return
                 yield child
         else:
@@ -235,12 +243,12 @@ def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
             container = container.next_sibling
 
 
-def is_fmt_on(container: LN) -> bool:
+def is_fmt_on(container: LN, preview: bool) -> bool:
     """Determine whether formatting is switched on within a container.
     Determined by whether the last `# fmt:` comment is `on` or `off`.
     """
     fmt_on = False
-    for comment in list_comments(container.prefix, is_endmarker=False):
+    for comment in list_comments(container.prefix, is_endmarker=False, preview=preview):
         if comment.value in FMT_ON:
             fmt_on = True
         elif comment.value in FMT_OFF:
@@ -248,7 +256,7 @@ def is_fmt_on(container: LN) -> bool:
     return fmt_on
 
 
-def contains_fmt_on_at_column(container: LN, column: int) -> bool:
+def contains_fmt_on_at_column(container: LN, column: int, *, preview: bool) -> bool:
     """Determine if children at a given column have formatting switched on."""
     for child in container.children:
         if (
@@ -257,7 +265,7 @@ def contains_fmt_on_at_column(container: LN, column: int) -> bool:
             or isinstance(child, Leaf)
             and child.column == column
         ):
-            if is_fmt_on(child):
+            if is_fmt_on(child, preview=preview):
                 return True
 
     return False
diff --git a/src/black/linegen.py b/src/black/linegen.py
index 4dc242a1dfe..79475a83f0e 100644
--- a/src/black/linegen.py
+++ b/src/black/linegen.py
@@ -72,7 +72,7 @@ def visit_default(self, node: LN) -> Iterator[Line]:
         """Default `visit_*()` implementation. Recurses to children of `node`."""
         if isinstance(node, Leaf):
             any_open_brackets = self.current_line.bracket_tracker.any_open_brackets()
-            for comment in generate_comments(node):
+            for comment in generate_comments(node, preview=self.mode.preview):
                 if any_open_brackets:
                     # any comment within brackets is subject to splitting
                     self.current_line.append(comment)
@@ -132,7 +132,7 @@ def visit_stmt(
         `parens` holds a set of string leaf values immediately after which
         invisible parens should be put.
         """
-        normalize_invisible_parens(node, parens_after=parens)
+        normalize_invisible_parens(node, parens_after=parens, preview=self.mode.preview)
         for child in node.children:
             if is_name_token(child) and child.value in keywords:
                 yield from self.line()
@@ -141,7 +141,7 @@ def visit_stmt(
 
     def visit_match_case(self, node: Node) -> Iterator[Line]:
         """Visit either a match or case statement."""
-        normalize_invisible_parens(node, parens_after=set())
+        normalize_invisible_parens(node, parens_after=set(), preview=self.mode.preview)
 
         yield from self.line()
         for child in node.children:
@@ -802,7 +802,9 @@ def normalize_prefix(leaf: Leaf, *, inside_brackets: bool) -> None:
     leaf.prefix = ""
 
 
-def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
+def normalize_invisible_parens(
+    node: Node, parens_after: Set[str], *, preview: bool
+) -> None:
     """Make existing optional parentheses invisible or create new ones.
 
     `parens_after` is a set of string leaf values immediately after which parens
@@ -811,7 +813,7 @@ def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
     Standardizes on visible parentheses for single-element tuples, and keeps
     existing visible parentheses for other tuples and generator expressions.
     """
-    for pc in list_comments(node.prefix, is_endmarker=False):
+    for pc in list_comments(node.prefix, is_endmarker=False, preview=preview):
         if pc.value in FMT_OFF:
             # This `node` has a prefix with `# fmt: off`, don't mess with parens.
             return
@@ -820,7 +822,9 @@ def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
         # Fixes a bug where invisible parens are not properly stripped from
         # assignment statements that contain type annotations.
         if isinstance(child, Node) and child.type == syms.annassign:
-            normalize_invisible_parens(child, parens_after=parens_after)
+            normalize_invisible_parens(
+                child, parens_after=parens_after, preview=preview
+            )
 
         # Add parentheses around long tuple unpacking in assignments.
         if (
diff --git a/tests/data/comments8.py b/tests/data/comments8.py
new file mode 100644
index 00000000000..b7f150d3031
--- /dev/null
+++ b/tests/data/comments8.py
@@ -0,0 +1,101 @@
+# The percent-percent comments are Spyder IDE cells.
+# Both `#%%`` and `# %%` are accepted, so `black` standardises
+# to the latter.
+
+#%%
+def func():
+    x = """
+    a really long string
+    """
+    lcomp3 = [
+        # This one is actually too long to fit in a single line.
+        element.split("\n", 1)[0]
+        # yup
+        for element in collection.select_elements()
+        # right
+        if element is not None
+    ]
+    # Capture each of the exceptions in the MultiError along with each of their causes and contexts
+    if isinstance(exc_value, MultiError):
+        embedded = []
+        for exc in exc_value.exceptions:
+            if exc not in _seen:
+                embedded.append(
+                    # This should be left alone (before)
+                    traceback.TracebackException.from_exception(
+                        exc,
+                        limit=limit,
+                        lookup_lines=lookup_lines,
+                        capture_locals=capture_locals,
+                        # copy the set of _seen exceptions so that duplicates
+                        # shared between sub-exceptions are not omitted
+                        _seen=set(_seen),
+                    )
+                    # This should be left alone (after)
+                )
+
+    # everything is fine if the expression isn't nested
+    traceback.TracebackException.from_exception(
+        exc,
+        limit=limit,
+        lookup_lines=lookup_lines,
+        capture_locals=capture_locals,
+        # copy the set of _seen exceptions so that duplicates
+        # shared between sub-exceptions are not omitted
+        _seen=set(_seen),
+    )
+
+
+#%%
+
+# output
+
+# The percent-percent comments are Spyder IDE cells.
+# Both `#%%`` and `# %%` are accepted, so `black` standardises
+# to the latter.
+
+# %%
+def func():
+    x = """
+    a really long string
+    """
+    lcomp3 = [
+        # This one is actually too long to fit in a single line.
+        element.split("\n", 1)[0]
+        # yup
+        for element in collection.select_elements()
+        # right
+        if element is not None
+    ]
+    # Capture each of the exceptions in the MultiError along with each of their causes and contexts
+    if isinstance(exc_value, MultiError):
+        embedded = []
+        for exc in exc_value.exceptions:
+            if exc not in _seen:
+                embedded.append(
+                    # This should be left alone (before)
+                    traceback.TracebackException.from_exception(
+                        exc,
+                        limit=limit,
+                        lookup_lines=lookup_lines,
+                        capture_locals=capture_locals,
+                        # copy the set of _seen exceptions so that duplicates
+                        # shared between sub-exceptions are not omitted
+                        _seen=set(_seen),
+                    )
+                    # This should be left alone (after)
+                )
+
+    # everything is fine if the expression isn't nested
+    traceback.TracebackException.from_exception(
+        exc,
+        limit=limit,
+        lookup_lines=lookup_lines,
+        capture_locals=capture_locals,
+        # copy the set of _seen exceptions so that duplicates
+        # shared between sub-exceptions are not omitted
+        _seen=set(_seen),
+    )
+
+
+# %%
\ No newline at end of file
diff --git a/tests/test_format.py b/tests/test_format.py
index 04eda43d5cf..2febb41b291 100644
--- a/tests/test_format.py
+++ b/tests/test_format.py
@@ -75,6 +75,7 @@
     # string processing
     "cantfit",
     "comments7",
+    "comments8",
     "long_strings",
     "long_strings__edge_case",
     "long_strings__regression",


Code review comments:
diff_hunk: @@ -0,0 +1,101 @@
+# The percent-percent comments are Spyder IDE cells.
+# Both `#%%`` and `# %%` are accepted, so `black` standardises
+# to the latter.
+
+#%%
+def func():
+    x = """
+    a really long string
+    """
+    lcomp3 = [
path: tests/data/comments8.py
text: @user1:
Can we shorten this code? The test can focus on just `#%%`, so no need for everything else.

@author:
sure, I was meaning it to be a `--preview` version of `comments3.py` - but shortening it and just testing `# %%` seems fine