Repository: psf/black

Issue:
Black doesn't enforce two empty lines before functions on module level when they have a sticky leading comment
Operating system: macOS High Sierra
Python version: 3.6.5
Black version: 18.5b0
pycodestyle version: 2.4.0
Does also happen on master: yes

Should Black insert a newline between these variables and the function?

### pycodestyle complains about test.py but Black doesn't change it

```console
$ cat test.py
```
```python
from __future__ import print_function, unicode_literals

all_data = []
all_versions = set()

# Comment
def thing():
    pass
```
```console
$ pycodestyle test.py
test.py:7:1: E302 expected 2 blank lines, found 1
$ black test.py
/private/tmp/test.py already well formatted, good job.
```

### Both pycodestyle and Black are fine with test2.py

```console
$ cat test2.py
```
```python
from __future__ import print_function, unicode_literals

all_data = []
all_versions = set()


# Comment
def thing():
    pass
```
```console
$ pycodestyle test2.py
$ black test2.py
/private/tmp/test2.py already well formatted, good job.
```

### Black reformats test3.py, with no comment

```console
$ cat test3.py
```
```python
from __future__ import print_function, unicode_literals

all_data = []
all_versions = set()

def thing():
    pass
```
```console
$ pycodestyle test3.py
test3.py:6:1: E302 expected 2 blank lines, found 1
$ black test3.py
reformatted /private/tmp/test3.py
$ pycodestyle test3.py
$ cat test3.py
```
```python
from __future__ import print_function, unicode_literals

all_data = []
all_versions = set()


def thing():
    pass
```


Discussion:
Judging by the comments in [comments5.py](https://github.com/ambv/black/blob/master/tests/comments5.py#L61) it's not trivial to support this case with the current implementation.

@ambv, would the `EmptyLineTracker` need to have access to both the previous as well as the next line to be able to handle this case? 
@Sterbic, yes. The way that would work is to modify `format_str()` to buffer one line before appending it to `dst_contents`, and only doing that after the next line (or the end of the file) comes in. This complicates things a fair bit.

@hugovk, as a workaround in the mean time, add the extra newline yourself, Black will keep it (as you found out in test2.py).
Interesting, with give it a shot if I find some free time next week.
I was just going to post the same issue! Confirmed on my end too

```
_rmr_set_stimeout = rmr_c_lib.rmr_set_rtimeout
_rmr_set_stimeout.argtypes = [c_void_p, c_int]
_rmr_set_stimeout.restype = c_int
# extern int rmr_set_stimeout(void* vctx, int time)
def rmr_set_timeout(vctx, time):
    """
    Refer to the rmr C documentation for rmr_set_timeout
    """
    return _rmr_set_stimeout(vctx, time)
```

leads to `E302`; it doesn't change this code at all given the comment 


Can confirm that this is still an issue on main. Also even this code is untouched which isn't great:
```python
all_data = []
all_versions = set()
# Comment
def thing():
    pass

```

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -61,7 +61,7 @@
     unmask_cell,
 )
 from black.linegen import LN, LineGenerator, transform_line
-from black.lines import EmptyLineTracker, Line
+from black.lines import EmptyLineTracker, LinesBlock
 from black.mode import (
     FUTURE_FLAG_TO_FEATURE,
     VERSION_TO_FEATURES,
@@ -1075,7 +1075,7 @@ def f(
 
 def _format_str_once(src_contents: str, *, mode: Mode) -> str:
     src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
-    dst_contents = []
+    dst_blocks: List[LinesBlock] = []
     if mode.target_versions:
         versions = mode.target_versions
     else:
@@ -1084,22 +1084,25 @@ def _format_str_once(src_contents: str, *, mode: Mode) -> str:
 
     normalize_fmt_off(src_node, preview=mode.preview)
     lines = LineGenerator(mode=mode)
-    elt = EmptyLineTracker(is_pyi=mode.is_pyi)
-    empty_line = Line(mode=mode)
-    after = 0
+    elt = EmptyLineTracker(mode=mode)
     split_line_features = {
         feature
         for feature in {Feature.TRAILING_COMMA_IN_CALL, Feature.TRAILING_COMMA_IN_DEF}
         if supports_feature(versions, feature)
     }
+    block: Optional[LinesBlock] = None
     for current_line in lines.visit(src_node):
-        dst_contents.append(str(empty_line) * after)
-        before, after = elt.maybe_empty_lines(current_line)
-        dst_contents.append(str(empty_line) * before)
+        block = elt.maybe_empty_lines(current_line)
+        dst_blocks.append(block)
         for line in transform_line(
             current_line, mode=mode, features=split_line_features
         ):
-            dst_contents.append(str(line))
+            block.content_lines.append(str(line))
+    if dst_blocks:
+        dst_blocks[-1].after = 0
+    dst_contents = []
+    for block in dst_blocks:
+        dst_contents.extend(block.all_lines())
     return "".join(dst_contents)
 
 
diff --git a/src/black/lines.py b/src/black/lines.py
--- a/src/black/lines.py
+++ b/src/black/lines.py
@@ -448,6 +448,28 @@ def __bool__(self) -> bool:
         return bool(self.leaves or self.comments)
 
 
+@dataclass
+class LinesBlock:
+    """Class that holds information about a block of formatted lines.
+
+    This is introduced so that the EmptyLineTracker can look behind the standalone
+    comments and adjust their empty lines for class or def lines.
+    """
+
+    mode: Mode
+    previous_block: Optional["LinesBlock"]
+    original_line: Line
+    before: int = 0
+    content_lines: List[str] = field(default_factory=list)
+    after: int = 0
+
+    def all_lines(self) -> List[str]:
+        empty_line = str(Line(mode=self.mode))
+        return (
+            [empty_line * self.before] + self.content_lines + [empty_line * self.after]
+        )
+
+
 @dataclass
 class EmptyLineTracker:
     """Provides a stateful method that returns the number of potential extra
@@ -458,33 +480,55 @@ class EmptyLineTracker:
     are consumed by `maybe_empty_lines()` and included in the computation.
     """
 
-    is_pyi: bool = False
+    mode: Mode
     previous_line: Optional[Line] = None
-    previous_after: int = 0
+    previous_block: Optional[LinesBlock] = None
     previous_defs: List[int] = field(default_factory=list)
+    semantic_leading_comment: Optional[LinesBlock] = None
 
-    def maybe_empty_lines(self, current_line: Line) -> Tuple[int, int]:
+    def maybe_empty_lines(self, current_line: Line) -> LinesBlock:
         """Return the number of extra empty lines before and after the `current_line`.
 
         This is for separating `def`, `async def` and `class` with extra empty
         lines (two on module-level).
         """
         before, after = self._maybe_empty_lines(current_line)
+        previous_after = self.previous_block.after if self.previous_block else 0
         before = (
             # Black should not insert empty lines at the beginning
             # of the file
             0
             if self.previous_line is None
-            else before - self.previous_after
+            else before - previous_after
         )
-        self.previous_after = after
+        block = LinesBlock(
+            mode=self.mode,
+            previous_block=self.previous_block,
+            original_line=current_line,
+            before=before,
+            after=after,
+        )
+
+        # Maintain the semantic_leading_comment state.
+        if current_line.is_comment:
+            if self.previous_line is None or (
+                not self.previous_line.is_decorator
+                # `or before` means this comment already has an empty line before
+                and (not self.previous_line.is_comment or before)
+                and (self.semantic_leading_comment is None or before)
+            ):
+                self.semantic_leading_comment = block
+        elif not current_line.is_decorator:
+            self.semantic_leading_comment = None
+
         self.previous_line = current_line
-        return before, after
+        self.previous_block = block
+        return block
 
     def _maybe_empty_lines(self, current_line: Line) -> Tuple[int, int]:
         max_allowed = 1
         if current_line.depth == 0:
-            max_allowed = 1 if self.is_pyi else 2
+            max_allowed = 1 if self.mode.is_pyi else 2
         if current_line.leaves:
             # Consume the first leaf's extra newlines.
             first_leaf = current_line.leaves[0]
@@ -495,7 +539,7 @@ def _maybe_empty_lines(self, current_line: Line) -> Tuple[int, int]:
             before = 0
         depth = current_line.depth
         while self.previous_defs and self.previous_defs[-1] >= depth:
-            if self.is_pyi:
+            if self.mode.is_pyi:
                 assert self.previous_line is not None
                 if depth and not current_line.is_def and self.previous_line.is_def:
                     # Empty lines between attributes and methods should be preserved.
@@ -563,7 +607,7 @@ def _maybe_empty_lines_for_class_or_def(
             return 0, 0
 
         if self.previous_line.is_decorator:
-            if self.is_pyi and current_line.is_stub_class:
+            if self.mode.is_pyi and current_line.is_stub_class:
                 # Insert an empty line after a decorated stub class
                 return 0, 1
 
@@ -574,14 +618,27 @@ def _maybe_empty_lines_for_class_or_def(
         ):
             return 0, 0
 
+        comment_to_add_newlines: Optional[LinesBlock] = None
         if (
             self.previous_line.is_comment
             and self.previous_line.depth == current_line.depth
             and before == 0
         ):
-            return 0, 0
+            slc = self.semantic_leading_comment
+            if (
+                Preview.empty_lines_before_class_or_def_with_leading_comments
+                in current_line.mode
+                and slc is not None
+                and slc.previous_block is not None
+                and not slc.previous_block.original_line.is_class
+                and not slc.previous_block.original_line.opens_block
+                and slc.before <= 1
+            ):
+                comment_to_add_newlines = slc
+            else:
+                return 0, 0
 
-        if self.is_pyi:
+        if self.mode.is_pyi:
             if current_line.is_class or self.previous_line.is_class:
                 if self.previous_line.depth < current_line.depth:
                     newlines = 0
@@ -609,6 +666,13 @@ def _maybe_empty_lines_for_class_or_def(
                 newlines = 0
         else:
             newlines = 1 if current_line.depth else 2
+        if comment_to_add_newlines is not None:
+            previous_block = comment_to_add_newlines.previous_block
+            if previous_block is not None:
+                comment_to_add_newlines.before = (
+                    max(comment_to_add_newlines.before, newlines) - previous_block.after
+                )
+                newlines = 0
         return newlines, 0
 
 
diff --git a/src/black/mode.py b/src/black/mode.py
--- a/src/black/mode.py
+++ b/src/black/mode.py
@@ -150,6 +150,7 @@ class Preview(Enum):
     """Individual preview style features."""
 
     annotation_parens = auto()
+    empty_lines_before_class_or_def_with_leading_comments = auto()
     long_docstring_quotes_on_newline = auto()
     normalize_docstring_quotes_and_prefixes_properly = auto()
     one_element_subscript = auto()


Test patch:
diff --git a/tests/data/preview/comments9.py b/tests/data/preview/comments9.py
new file mode 100644
--- /dev/null
+++ b/tests/data/preview/comments9.py
@@ -0,0 +1,254 @@
+# Test for https://github.com/psf/black/issues/246.
+
+some = statement
+# This comment should be split from the statement above by two lines.
+def function():
+    pass
+
+
+some = statement
+# This multiline comments section
+# should be split from the statement
+# above by two lines.
+def function():
+    pass
+
+
+some = statement
+# This comment should be split from the statement above by two lines.
+async def async_function():
+    pass
+
+
+some = statement
+# This comment should be split from the statement above by two lines.
+class MyClass:
+    pass
+
+
+some = statement
+# This should be stick to the statement above
+
+# This should be split from the above by two lines
+class MyClassWithComplexLeadingComments:
+    pass
+
+
+class ClassWithDocstring:
+    """A docstring."""
+# Leading comment after a class with just a docstring
+class MyClassAfterAnotherClassWithDocstring:
+    pass
+
+
+some = statement
+# leading 1
+@deco1
+# leading 2
+# leading 2 extra
+@deco2(with_args=True)
+# leading 3
+@deco3
+# leading 4
+def decorated():
+    pass
+
+
+some = statement
+# leading 1
+@deco1
+# leading 2
+@deco2(with_args=True)
+
+# leading 3 that already has an empty line
+@deco3
+# leading 4
+def decorated_with_split_leading_comments():
+    pass
+
+
+some = statement
+# leading 1
+@deco1
+# leading 2
+@deco2(with_args=True)
+# leading 3
+@deco3
+
+# leading 4 that already has an empty line
+def decorated_with_split_leading_comments():
+    pass
+
+
+def main():
+    if a:
+        # Leading comment before inline function
+        def inline():
+            pass
+        # Another leading comment
+        def another_inline():
+            pass
+    else:
+        # More leading comments
+        def inline_after_else():
+            pass
+
+
+if a:
+    # Leading comment before "top-level inline" function
+    def top_level_quote_inline():
+        pass
+    # Another leading comment
+    def another_top_level_quote_inline_inline():
+        pass
+else:
+    # More leading comments
+    def top_level_quote_inline_after_else():
+        pass
+
+
+class MyClass:
+    # First method has no empty lines between bare class def.
+    # More comments.
+    def first_method(self):
+        pass
+
+
+# output
+
+
+# Test for https://github.com/psf/black/issues/246.
+
+some = statement
+
+
+# This comment should be split from the statement above by two lines.
+def function():
+    pass
+
+
+some = statement
+
+
+# This multiline comments section
+# should be split from the statement
+# above by two lines.
+def function():
+    pass
+
+
+some = statement
+
+
+# This comment should be split from the statement above by two lines.
+async def async_function():
+    pass
+
+
+some = statement
+
+
+# This comment should be split from the statement above by two lines.
+class MyClass:
+    pass
+
+
+some = statement
+# This should be stick to the statement above
+
+
+# This should be split from the above by two lines
+class MyClassWithComplexLeadingComments:
+    pass
+
+
+class ClassWithDocstring:
+    """A docstring."""
+
+
+# Leading comment after a class with just a docstring
+class MyClassAfterAnotherClassWithDocstring:
+    pass
+
+
+some = statement
+
+
+# leading 1
+@deco1
+# leading 2
+# leading 2 extra
+@deco2(with_args=True)
+# leading 3
+@deco3
+# leading 4
+def decorated():
+    pass
+
+
+some = statement
+
+
+# leading 1
+@deco1
+# leading 2
+@deco2(with_args=True)
+
+# leading 3 that already has an empty line
+@deco3
+# leading 4
+def decorated_with_split_leading_comments():
+    pass
+
+
+some = statement
+
+
+# leading 1
+@deco1
+# leading 2
+@deco2(with_args=True)
+# leading 3
+@deco3
+
+# leading 4 that already has an empty line
+def decorated_with_split_leading_comments():
+    pass
+
+
+def main():
+    if a:
+        # Leading comment before inline function
+        def inline():
+            pass
+
+        # Another leading comment
+        def another_inline():
+            pass
+
+    else:
+        # More leading comments
+        def inline_after_else():
+            pass
+
+
+if a:
+    # Leading comment before "top-level inline" function
+    def top_level_quote_inline():
+        pass
+
+    # Another leading comment
+    def another_top_level_quote_inline_inline():
+        pass
+
+else:
+    # More leading comments
+    def top_level_quote_inline_after_else():
+        pass
+
+
+class MyClass:
+    # First method has no empty lines between bare class def.
+    # More comments.
+    def first_method(self):
+        pass
diff --git a/tests/data/preview/remove_await_parens.py b/tests/data/preview/remove_await_parens.py
--- a/tests/data/preview/remove_await_parens.py
+++ b/tests/data/preview/remove_await_parens.py
@@ -80,6 +80,7 @@ async def main():
 # output
 import asyncio
 
+
 # Control example
 async def main():
     await asyncio.sleep(1)
diff --git a/tests/data/simple_cases/comments5.py b/tests/data/simple_cases/comments5.py
--- a/tests/data/simple_cases/comments5.py
+++ b/tests/data/simple_cases/comments5.py
@@ -58,9 +58,9 @@ def decorated1():
     ...
 
 
-# Note: crappy but inevitable.  The current design of EmptyLineTracker doesn't
-# allow this to work correctly.  The user will have to split those lines by
-# hand.
+# Note: this is fixed in
+# Preview.empty_lines_before_class_or_def_with_leading_comments.
+# In the current style, the user will have to split those lines by hand.
 some_instruction
 # This comment should be split from `some_instruction` by two lines but isn't.
 def g():
diff --git a/tests/data/simple_cases/docstring_no_extra_empty_line_before_eof.py b/tests/data/simple_cases/docstring_no_extra_empty_line_before_eof.py
new file mode 100644
--- /dev/null
+++ b/tests/data/simple_cases/docstring_no_extra_empty_line_before_eof.py
@@ -0,0 +1,4 @@
+# Make sure when the file ends with class's docstring,
+# It doesn't add extra blank lines.
+class ClassWithDocstring:
+    """A docstring."""