Repository: psf/black

Issue:
Black produced code that is not equivalent to the source
**Describe the bug** A clear and concise description of what the bug is.

**To Reproduce** Steps to reproduce the behavior:

1. Down file: [test.py.txt](https://github.com/psf/black/files/5611061/test.py.txt)
2. `black test.py`
3. Fails to format docstring
```
error: cannot format test.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report
 a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_8rv9ecj9.log
Oh no! 💥 💔 💥
1 file failed to reformat.
```
Error log: [blk_8rv9ecj9.log](https://github.com/psf/black/files/5611063/blk_8rv9ecj9.log)

**Expected behavior**
Successful formatting of docstring

**Environment (please complete the following information):**

- Version: 20.8b1
- OS and Python version: Ubuntu 18.04/Python 2.7

**Does this bug also happen on master?** 
yes


Discussion:
JelleZijlstra: Looks like we still need a news entry here, and @ichard26 may still want to review. The code otherwise looks good to me.
ichard26: Yeah, to be clear, I do want to do a review just haven't been able to do one yet.
ichard26: Oh, by the way, you can use special keywords in the PR description so GitHub will automatically close an issue upon merge. You can read more here: https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword. This makes sure any issue that should be closed are in fact closed (and I won't have to go proactively hunt for such issues!), keeping our very chaotic issue tracker ever so slightly less chaotic and scary :) 
CiderMan: Hi Guys, Is there anything preventing this PR from being merged? The bug has been reported a number of times so it would be good to get the fix onto master ASAP.

On a related note, does anyone know when we might get a new version pushed to PyPI? We can't (currently) roll out a new version unless it is on PyPI and there are a couple of bug fixes that are on master that we would really like to get...

Cheers!
JelleZijlstra: Could you add a news entry (https://github.com/psf/black/blob/master/CONTRIBUTING.md#news--changelog-requirement)? Apart from that, I'd be happy to merge this.
ichard26: To be clear, I want to rereview this before approving its merge. In terms of timeline, I should be able to get a review done around 10:40-ish, and if not, by the end of the day. Sorry for holding up the PR!

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index f911fdf..2999abf 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -52,6 +52,9 @@
 
 - Exclude `venv` directory by default (#1683)
 
+- Fixed "Black produced code that is not equivalent to the source" when formatting
+  Python 2 docstrings (#2037)
+
 #### _Packaging_
 
 - Self-contained native _Black_ binaries are now provided for releases via GitHub
diff --git a/src/black/__init__.py b/src/black/__init__.py
index efa82f4..5c9ab75 100644
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -6466,12 +6466,22 @@ def _stringify_ast(
             # Constant strings may be indented across newlines, if they are
             # docstrings; fold spaces after newlines when comparing. Similarly,
             # trailing and leading space may be removed.
+            # Note that when formatting Python 2 code, at least with Windows
+            # line-endings, docstrings can end up here as bytes instead of
+            # str so make sure that we handle both cases.
             if (
                 isinstance(node, ast.Constant)
                 and field == "value"
-                and isinstance(value, str)
+                and isinstance(value, (str, bytes))
             ):
-                normalized = re.sub(r" *\n[ \t]*", "\n", value).strip()
+                lineend = "\n" if isinstance(value, str) else b"\n"
+                # To normalize, we strip any leading and trailing space from
+                # each line...
+                stripped = [line.strip() for line in value.splitlines()]
+                normalized = lineend.join(stripped)  # type: ignore[attr-defined]
+                # ...and remove any blank lines at the beginning and end of
+                # the whole string
+                normalized = normalized.strip()
             else:
                 normalized = value
             yield f"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}"


Test patch:
diff --git a/tests/test_black.py b/tests/test_black.py
index c603233..201edfa 100644
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -1918,6 +1918,26 @@ def test_bpo_2142_workaround(self) -> None:
         actual = diff_header.sub(DETERMINISTIC_HEADER, actual)
         self.assertEqual(actual, expected)
 
+    def test_docstring_reformat_for_py27(self) -> None:
+        """
+        Check that stripping trailing whitespace from Python 2 docstrings
+        doesn't trigger a "not equivalent to source" error
+        """
+        source = (
+            b'def foo():\r\n    """Testing\r\n    Testing """\r\n    print "Foo"\r\n'
+        )
+        expected = 'def foo():\n    """Testing\n    Testing"""\n    print "Foo"\n'
+
+        result = CliRunner().invoke(
+            black.main,
+            ["-", "-q", "--target-version=py27"],
+            input=BytesIO(source),
+        )
+
+        self.assertEqual(result.exit_code, 0)
+        actual = result.output
+        self.assertFormatEqual(actual, expected)
+
 
 with open(black.__file__, "r", encoding="utf-8") as _bf:
     black_source_lines = _bf.readlines()