Repository: psf/black

Issue:
Future style for stub file docstrings that use C/C++ extensions
Hello Black community. 
I write to you today to discuss the style outlined in the documentation for stub files (Python files meant for typing purposes with a `.pyi` extension). In the _Black Code Style_ section of the documentation there is a section describing future style guidelines for the `.pyi` files: [Typing Stub Files](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#typing-stub-files).

**Description**
<!-- A clear and concise description of how the style can be
improved. -->

The point I wish to argue against today is the line: "do not use docstrings". I imagine this is referring to a Python stub file that is providing type hints for another Python file, like so: 
_add.py_
```Python
def add(a, b):
    " here's an existing docstring"
    return a + b
```
_add.pyi_
```python
def add(a: int, b: int) -> int:
    " here's another stub file docstring"
    ...
```
In which case, it would make sense to either include the docstring in one file or the other. 

However, the edge case I wish to bring to your attention is when using C/C++ extensions for Python code. In particular I will discuss the use of [PyBind11](https://pybind11.readthedocs.io/en/stable/basics.html). PyBind11 produces a shared library with a name such as `add.cpython-38-x86_64-linux-gnu.so`, which contains inside it Python objects that you can import as any other Python object. However, modern Intellisense servers like [Pylance](https://github.com/microsoft/pylance-release) do not extract information from binary files by [design](https://github.com/microsoft/pylance-release/issues/70), and therefore you _cannot_ get any documentation from a binary file, be this docstrings or type information. 

Therefore, a stub file can be used to make up for the design limitation, but you need to include docstrings in the stub file in order to get useful information about a function written with PyBind11. 

_add_pybind11.py_
```python
# import the add function from a PyBind11 produced binary
import add from _add_pybind11

# the above will generate no intellisense due to the design limitation of tools like pylance
```

_add_pybind11.pyi_
```python
# define the docstring and type information for the C/C++ bound object
def add(a: int, b: int) -> int:
    " This function was generated by pybind11 and therefore needs a stub file to use with intellisense"
    ...
```

**Conclusion**
I'd like to offer the example of using stub files with binary libraries, such as those produced by PyBind11, as a reason to allow docstrings in stub files in future versions of the formatter. This can be with a flag or configuration, but rest assured having Black delete my docstrings for my PyBind11 objects would cause headaches. 

<!-- Add any other context about the problem here. -->
![image](https://user-images.githubusercontent.com/29968424/159010748-6203be64-527a-4d0b-a2f7-401da9023014.png)


Discussion:

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index bb3ccb9..d0faf7c 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -14,6 +14,8 @@
 
 <!-- Changes that affect Black's preview style -->
 
+- Code cell separators `#%%` are now standardised to `# %%` (#2919)
+
 ### _Blackd_
 
 <!-- Changes to blackd -->
diff --git a/docs/the_black_code_style/current_style.md b/docs/the_black_code_style/current_style.md
index 0bf5894..d54c7ab 100644
--- a/docs/the_black_code_style/current_style.md
+++ b/docs/the_black_code_style/current_style.md
@@ -399,16 +399,11 @@ recommended code style for those files is more terse than PEP 8:
 _Black_ enforces the above rules. There are additional guidelines for formatting `.pyi`
 file that are not enforced yet but might be in a future version of the formatter:
 
-- all function bodies should be empty (contain `...` instead of the body);
-- do not use docstrings;
 - prefer `...` over `pass`;
-- for arguments with a default, use `...` instead of the actual default;
 - avoid using string literals in type annotations, stub files support forward references
   natively (like Python 3.7 code with `from __future__ import annotations`);
 - use variable annotations instead of type comments, even for stubs that target older
-  versions of Python;
-- for arguments that default to `None`, use `Optional[]` explicitly;
-- use `float` instead of `Union[int, float]`.
+  versions of Python.
 
 ## Pragmatism
 
diff --git a/src/black/__init__.py b/src/black/__init__.py
index c4ec99b..51e31e9 100644
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -1166,7 +1166,7 @@ def _format_str_once(src_contents: str, *, mode: Mode) -> str:
     else:
         versions = detect_target_versions(src_node, future_imports=future_imports)
 
-    normalize_fmt_off(src_node)
+    normalize_fmt_off(src_node, preview=mode.preview)
     lines = LineGenerator(mode=mode)
     elt = EmptyLineTracker(is_pyi=mode.is_pyi)
     empty_line = Line(mode=mode)
diff --git a/src/black/comments.py b/src/black/comments.py
index 28b9117..4553264 100644
--- a/src/black/comments.py
+++ b/src/black/comments.py
@@ -23,6 +23,8 @@
 FMT_PASS: Final = {*FMT_OFF, *FMT_SKIP}
 FMT_ON: Final = {"# fmt: on", "# fmt:on", "# yapf: enable"}
 
+COMMENT_EXCEPTIONS = {True: " !:#'", False: " !:#'%"}
+
 
 @dataclass
 class ProtoComment:
@@ -42,7 +44,7 @@ class ProtoComment:
     consumed: int  # how many characters of the original leaf's prefix did we consume
 
 
-def generate_comments(leaf: LN) -> Iterator[Leaf]:
+def generate_comments(leaf: LN, *, preview: bool) -> Iterator[Leaf]:
     """Clean the prefix of the `leaf` and generate comments from it, if any.
 
     Comments in lib2to3 are shoved into the whitespace prefix.  This happens
@@ -61,12 +63,16 @@ def generate_comments(leaf: LN) -> Iterator[Leaf]:
     Inline comments are emitted as regular token.COMMENT leaves.  Standalone
     are emitted with a fake STANDALONE_COMMENT token identifier.
     """
-    for pc in list_comments(leaf.prefix, is_endmarker=leaf.type == token.ENDMARKER):
+    for pc in list_comments(
+        leaf.prefix, is_endmarker=leaf.type == token.ENDMARKER, preview=preview
+    ):
         yield Leaf(pc.type, pc.value, prefix="\n" * pc.newlines)
 
 
 @lru_cache(maxsize=4096)
-def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
+def list_comments(
+    prefix: str, *, is_endmarker: bool, preview: bool
+) -> List[ProtoComment]:
     """Return a list of :class:`ProtoComment` objects parsed from the given `prefix`."""
     result: List[ProtoComment] = []
     if not prefix or "#" not in prefix:
@@ -92,7 +98,7 @@ def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
             comment_type = token.COMMENT  # simple trailing comment
         else:
             comment_type = STANDALONE_COMMENT
-        comment = make_comment(line)
+        comment = make_comment(line, preview=preview)
         result.append(
             ProtoComment(
                 type=comment_type, value=comment, newlines=nlines, consumed=consumed
@@ -102,10 +108,10 @@ def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
     return result
 
 
-def make_comment(content: str) -> str:
+def make_comment(content: str, *, preview: bool) -> str:
     """Return a consistently formatted comment from the given `content` string.
 
-    All comments (except for "##", "#!", "#:", '#'", "#%%") should have a single
+    All comments (except for "##", "#!", "#:", '#'") should have a single
     space between the hash sign and the content.
 
     If `content` didn't start with a hash sign, one is provided.
@@ -123,26 +129,26 @@ def make_comment(content: str) -> str:
         and not content.lstrip().startswith("type:")
     ):
         content = " " + content[1:]  # Replace NBSP by a simple space
-    if content and content[0] not in " !:#'%":
+    if content and content[0] not in COMMENT_EXCEPTIONS[preview]:
         content = " " + content
     return "#" + content
 
 
-def normalize_fmt_off(node: Node) -> None:
+def normalize_fmt_off(node: Node, *, preview: bool) -> None:
     """Convert content between `# fmt: off`/`# fmt: on` into standalone comments."""
     try_again = True
     while try_again:
-        try_again = convert_one_fmt_off_pair(node)
+        try_again = convert_one_fmt_off_pair(node, preview=preview)
 
 
-def convert_one_fmt_off_pair(node: Node) -> bool:
+def convert_one_fmt_off_pair(node: Node, *, preview: bool) -> bool:
     """Convert content of a single `# fmt: off`/`# fmt: on` into a standalone comment.
 
     Returns True if a pair was converted.
     """
     for leaf in node.leaves():
         previous_consumed = 0
-        for comment in list_comments(leaf.prefix, is_endmarker=False):
+        for comment in list_comments(leaf.prefix, is_endmarker=False, preview=preview):
             if comment.value not in FMT_PASS:
                 previous_consumed = comment.consumed
                 continue
@@ -157,7 +163,7 @@ def convert_one_fmt_off_pair(node: Node) -> bool:
                     if comment.value in FMT_SKIP and prev.type in WHITESPACE:
                         continue
 
-            ignored_nodes = list(generate_ignored_nodes(leaf, comment))
+            ignored_nodes = list(generate_ignored_nodes(leaf, comment, preview=preview))
             if not ignored_nodes:
                 continue
 
@@ -197,7 +203,9 @@ def convert_one_fmt_off_pair(node: Node) -> bool:
     return False
 
 
-def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
+def generate_ignored_nodes(
+    leaf: Leaf, comment: ProtoComment, *, preview: bool
+) -> Iterator[LN]:
     """Starting from the container of `leaf`, generate all leaves until `# fmt: on`.
 
     If comment is skip, returns leaf only.
@@ -221,13 +229,13 @@ def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
             yield leaf.parent
         return
     while container is not None and container.type != token.ENDMARKER:
-        if is_fmt_on(container):
+        if is_fmt_on(container, preview=preview):
             return
 
         # fix for fmt: on in children
-        if contains_fmt_on_at_column(container, leaf.column):
+        if contains_fmt_on_at_column(container, leaf.column, preview=preview):
             for child in container.children:
-                if contains_fmt_on_at_column(child, leaf.column):
+                if contains_fmt_on_at_column(child, leaf.column, preview=preview):
                     return
                 yield child
         else:
@@ -235,12 +243,12 @@ def generate_ignored_nodes(leaf: Leaf, comment: ProtoComment) -> Iterator[LN]:
             container = container.next_sibling
 
 
-def is_fmt_on(container: LN) -> bool:
+def is_fmt_on(container: LN, preview: bool) -> bool:
     """Determine whether formatting is switched on within a container.
     Determined by whether the last `# fmt:` comment is `on` or `off`.
     """
     fmt_on = False
-    for comment in list_comments(container.prefix, is_endmarker=False):
+    for comment in list_comments(container.prefix, is_endmarker=False, preview=preview):
         if comment.value in FMT_ON:
             fmt_on = True
         elif comment.value in FMT_OFF:
@@ -248,7 +256,7 @@ def is_fmt_on(container: LN) -> bool:
     return fmt_on
 
 
-def contains_fmt_on_at_column(container: LN, column: int) -> bool:
+def contains_fmt_on_at_column(container: LN, column: int, *, preview: bool) -> bool:
     """Determine if children at a given column have formatting switched on."""
     for child in container.children:
         if (
@@ -257,7 +265,7 @@ def contains_fmt_on_at_column(container: LN, column: int) -> bool:
             or isinstance(child, Leaf)
             and child.column == column
         ):
-            if is_fmt_on(child):
+            if is_fmt_on(child, preview=preview):
                 return True
 
     return False
diff --git a/src/black/linegen.py b/src/black/linegen.py
index 4dc242a..79475a8 100644
--- a/src/black/linegen.py
+++ b/src/black/linegen.py
@@ -72,7 +72,7 @@ def visit_default(self, node: LN) -> Iterator[Line]:
         """Default `visit_*()` implementation. Recurses to children of `node`."""
         if isinstance(node, Leaf):
             any_open_brackets = self.current_line.bracket_tracker.any_open_brackets()
-            for comment in generate_comments(node):
+            for comment in generate_comments(node, preview=self.mode.preview):
                 if any_open_brackets:
                     # any comment within brackets is subject to splitting
                     self.current_line.append(comment)
@@ -132,7 +132,7 @@ def visit_stmt(
         `parens` holds a set of string leaf values immediately after which
         invisible parens should be put.
         """
-        normalize_invisible_parens(node, parens_after=parens)
+        normalize_invisible_parens(node, parens_after=parens, preview=self.mode.preview)
         for child in node.children:
             if is_name_token(child) and child.value in keywords:
                 yield from self.line()
@@ -141,7 +141,7 @@ def visit_stmt(
 
     def visit_match_case(self, node: Node) -> Iterator[Line]:
         """Visit either a match or case statement."""
-        normalize_invisible_parens(node, parens_after=set())
+        normalize_invisible_parens(node, parens_after=set(), preview=self.mode.preview)
 
         yield from self.line()
         for child in node.children:
@@ -802,7 +802,9 @@ def normalize_prefix(leaf: Leaf, *, inside_brackets: bool) -> None:
     leaf.prefix = ""
 
 
-def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
+def normalize_invisible_parens(
+    node: Node, parens_after: Set[str], *, preview: bool
+) -> None:
     """Make existing optional parentheses invisible or create new ones.
 
     `parens_after` is a set of string leaf values immediately after which parens
@@ -811,7 +813,7 @@ def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
     Standardizes on visible parentheses for single-element tuples, and keeps
     existing visible parentheses for other tuples and generator expressions.
     """
-    for pc in list_comments(node.prefix, is_endmarker=False):
+    for pc in list_comments(node.prefix, is_endmarker=False, preview=preview):
         if pc.value in FMT_OFF:
             # This `node` has a prefix with `# fmt: off`, don't mess with parens.
             return
@@ -820,7 +822,9 @@ def normalize_invisible_parens(node: Node, parens_after: Set[str]) -> None:
         # Fixes a bug where invisible parens are not properly stripped from
         # assignment statements that contain type annotations.
         if isinstance(child, Node) and child.type == syms.annassign:
-            normalize_invisible_parens(child, parens_after=parens_after)
+            normalize_invisible_parens(
+                child, parens_after=parens_after, preview=preview
+            )
 
         # Add parentheses around long tuple unpacking in assignments.
         if (


Test patch:
diff --git a/tests/data/comments8.py b/tests/data/comments8.py
new file mode 100644
index 0000000..a2030c2
--- /dev/null
+++ b/tests/data/comments8.py
@@ -0,0 +1,15 @@
+# The percent-percent comments are Spyder IDE cells.
+# Both `#%%`` and `# %%` are accepted, so `black` standardises
+# to the latter.
+
+#%%
+# %%
+
+# output
+
+# The percent-percent comments are Spyder IDE cells.
+# Both `#%%`` and `# %%` are accepted, so `black` standardises
+# to the latter.
+
+# %%
+# %%
diff --git a/tests/test_format.py b/tests/test_format.py
index 269bbac..667d5c1 100644
--- a/tests/test_format.py
+++ b/tests/test_format.py
@@ -75,6 +75,7 @@
     # string processing
     "cantfit",
     "comments7",
+    "comments8",
     "long_strings",
     "long_strings__edge_case",
     "long_strings__regression",