Repository: psf/black

Issue:
error: cannot format Tests/test_file_libtiff.py: (-1, 8)
**Describe the bug** A clear and concise description of what the bug is.

**To Reproduce** Steps to reproduce the behavior:

1. Take this file https://github.com/python-pillow/Pillow/blob/e54909b97c213278accc8176f1c52e403f8afcb6/Tests/test_file_libtiff.py
2. Run _Black_ on it with these arguments `--check Tests/test_file_libtiff.py`
3. See error

```console
$ black --version
black, version 19.10b1.dev141+g820f387
$ black --check Tests/test_file_libtiff.py
error: cannot format Tests/test_file_libtiff.py: (-1, 8)
Oh no! 💥 💔 💥
1 file would fail to reformat.
```

**Expected behavior** A clear and concise description of what you expected to happen.

File formats.

**Environment (please complete the following information):**

- Version: master
- OS and Python version: macOS/Python 3.8.3

**Does this bug also happen on master?** To answer this, you have two options:

Yes, see above.

Works with last release (the codebase is already formatted with that version):

```console
$ black --version
black, version 19.10b0
$ black --check Tests/test_file_libtiff.py
All done! ✨ 🍰 ✨
1 file would be left unchanged.
```

**Additional context** Add any other context about the problem here.

Looks like it's this part:

```python
                            assert (
                                c_float(val[0][0] / val[0][1]).value
                                == c_float(value[0][0] / value[0][1]).value
                            ), ("%s didn't roundtrip" % tag)
```

https://github.com/python-pillow/Pillow/blob/e54909b97c213278accc8176f1c52e403f8afcb6/Tests/test_file_libtiff.py#L171-L174

(We're about to drop Python 3.5 and adopt f-strings, it's replacement is handled fine:

```python
                            assert (
                                c_float(val[0][0] / val[0][1]).value
                                == c_float(value[0][0] / value[0][1]).value
                            ), f"{tag} didn't roundtrip"
```
)
error: cannot format Tests/test_file_libtiff.py: (-1, 8)
**Describe the bug** A clear and concise description of what the bug is.

**To Reproduce** Steps to reproduce the behavior:

1. Take this file https://github.com/python-pillow/Pillow/blob/e54909b97c213278accc8176f1c52e403f8afcb6/Tests/test_file_libtiff.py
2. Run _Black_ on it with these arguments `--check Tests/test_file_libtiff.py`
3. See error

```console
$ black --version
black, version 19.10b1.dev141+g820f387
$ black --check Tests/test_file_libtiff.py
error: cannot format Tests/test_file_libtiff.py: (-1, 8)
Oh no! 💥 💔 💥
1 file would fail to reformat.
```

**Expected behavior** A clear and concise description of what you expected to happen.

File formats.

**Environment (please complete the following information):**

- Version: master
- OS and Python version: macOS/Python 3.8.3

**Does this bug also happen on master?** To answer this, you have two options:

Yes, see above.

Works with last release (the codebase is already formatted with that version):

```console
$ black --version
black, version 19.10b0
$ black --check Tests/test_file_libtiff.py
All done! ✨ 🍰 ✨
1 file would be left unchanged.
```

**Additional context** Add any other context about the problem here.

Looks like it's this part:

```python
                            assert (
                                c_float(val[0][0] / val[0][1]).value
                                == c_float(value[0][0] / value[0][1]).value
                            ), ("%s didn't roundtrip" % tag)
```

https://github.com/python-pillow/Pillow/blob/e54909b97c213278accc8176f1c52e403f8afcb6/Tests/test_file_libtiff.py#L171-L174

(We're about to drop Python 3.5 and adopt f-strings, it's replacement is handled fine:

```python
                            assert (
                                c_float(val[0][0] / val[0][1]).value
                                == c_float(value[0][0] / value[0][1]).value
                            ), f"{tag} didn't roundtrip"
```
)


Discussion:
`git bisect` -> 544ea9c217cad9459d7b60665db787e94b52f93d in "Improve String Handling" (#1132)

Possibly duplicate of https://github.com/psf/black/issues/1597?
> Possibly duplicate of #1597?

Probably, but I don't trust myself to accurately know due to lack of familiarity with Black's formatting code so I'll someone else decide that.

FYI: Jelle created a PR (#1594) where in verbose mode, Black prints out the trackback on failure so it's a bit more clear that there was an ~`IndexError`~ `KeyError`  🙂.
I'm pretty sure this is the same underlying issue as #1597, but when we fix it we should make sure both cases are covered. Closing this one as a duplicate though.
> FYI: Jelle created a PR (#1594) where in verbose mode, Black prints out the trackback on failure so it's a bit more clear that there was an `IndexError` 🙂.

That's useful!

Same traceback:

```
Traceback (most recent call last):
  File "/Users/hugo/github/black/src/black/__init__.py", line 657, in reformat_one
    if changed is not Changed.CACHED and format_file_in_place(
  File "/Users/hugo/github/black/src/black/__init__.py", line 800, in format_file_in_place
    dst_contents = format_file_contents(src_contents, fast=fast, mode=mode)
  File "/Users/hugo/github/black/src/black/__init__.py", line 921, in format_file_contents
    dst_contents = format_str(src_contents, mode=mode)
  File "/Users/hugo/github/black/src/black/__init__.py", line 986, in format_str
    for line in transform_line(
  File "/Users/hugo/github/black/src/black/__init__.py", line 2740, in transform_line
    result.extend(
  File "/Users/hugo/github/black/src/black/__init__.py", line 2734, in transform_line
    for transformed_line in transform(line, features):
  File "/Users/hugo/github/black/src/black/__init__.py", line 2843, in __call__
    for line_result in self.do_transform(line, string_idx):
  File "/Users/hugo/github/black/src/black/__init__.py", line 3363, in do_transform
    append_leaves(new_line, line, LL[: string_idx - 1])
  File "/Users/hugo/github/black/src/black/__init__.py", line 4649, in append_leaves
    new_line.append(new_leaf)
  File "/Users/hugo/github/black/src/black/__init__.py", line 1434, in append
    self.bracket_tracker.mark(leaf)
  File "/Users/hugo/github/black/src/black/__init__.py", line 1300, in mark
    opening_bracket = self.bracket_match.pop((self.depth, leaf.type))
KeyError: (-1, 8)
```
This file now formats without errors with `master`.
`git bisect` -> 544ea9c217cad9459d7b60665db787e94b52f93d in "Improve String Handling" (#1132)

Possibly duplicate of https://github.com/psf/black/issues/1597?
> Possibly duplicate of #1597?

Probably, but I don't trust myself to accurately know due to lack of familiarity with Black's formatting code so I'll someone else decide that.

FYI: Jelle created a PR (#1594) where in verbose mode, Black prints out the trackback on failure so it's a bit more clear that there was an ~`IndexError`~ `KeyError`  🙂.
I'm pretty sure this is the same underlying issue as #1597, but when we fix it we should make sure both cases are covered. Closing this one as a duplicate though.
> FYI: Jelle created a PR (#1594) where in verbose mode, Black prints out the trackback on failure so it's a bit more clear that there was an `IndexError` 🙂.

That's useful!

Same traceback:

```
Traceback (most recent call last):
  File "/Users/hugo/github/black/src/black/__init__.py", line 657, in reformat_one
    if changed is not Changed.CACHED and format_file_in_place(
  File "/Users/hugo/github/black/src/black/__init__.py", line 800, in format_file_in_place
    dst_contents = format_file_contents(src_contents, fast=fast, mode=mode)
  File "/Users/hugo/github/black/src/black/__init__.py", line 921, in format_file_contents
    dst_contents = format_str(src_contents, mode=mode)
  File "/Users/hugo/github/black/src/black/__init__.py", line 986, in format_str
    for line in transform_line(
  File "/Users/hugo/github/black/src/black/__init__.py", line 2740, in transform_line
    result.extend(
  File "/Users/hugo/github/black/src/black/__init__.py", line 2734, in transform_line
    for transformed_line in transform(line, features):
  File "/Users/hugo/github/black/src/black/__init__.py", line 2843, in __call__
    for line_result in self.do_transform(line, string_idx):
  File "/Users/hugo/github/black/src/black/__init__.py", line 3363, in do_transform
    append_leaves(new_line, line, LL[: string_idx - 1])
  File "/Users/hugo/github/black/src/black/__init__.py", line 4649, in append_leaves
    new_line.append(new_leaf)
  File "/Users/hugo/github/black/src/black/__init__.py", line 1434, in append
    self.bracket_tracker.mark(leaf)
  File "/Users/hugo/github/black/src/black/__init__.py", line 1300, in mark
    opening_bracket = self.bracket_match.pop((self.depth, leaf.type))
KeyError: (-1, 8)
```
This file now formats without errors with `master`.

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -112,6 +112,10 @@ class InvalidInput(ValueError):
     """Raised when input source code fails all parse attempts."""
 
 
+class BracketMatchError(KeyError):
+    """Raised when an opening bracket is unable to be matched to a closing bracket."""
+
+
 T = TypeVar("T")
 E = TypeVar("E", bound=Exception)
 
@@ -1308,7 +1312,13 @@ def mark(self, leaf: Leaf) -> None:
         self.maybe_decrement_after_lambda_arguments(leaf)
         if leaf.type in CLOSING_BRACKETS:
             self.depth -= 1
-            opening_bracket = self.bracket_match.pop((self.depth, leaf.type))
+            try:
+                opening_bracket = self.bracket_match.pop((self.depth, leaf.type))
+            except KeyError as e:
+                raise BracketMatchError(
+                    "Unable to match a closing bracket to the following opening"
+                    f" bracket: {leaf}"
+                ) from e
             leaf.opening_bracket = opening_bracket
             if not leaf.value:
                 self.invisible.append(leaf)
@@ -3324,10 +3334,17 @@ def do_transform(self, line: Line, string_idx: int) -> Iterator[TResult[Line]]:
                 yield TErr(
                     "Will not strip parentheses which have comments attached to them."
                 )
+                return
 
         new_line = line.clone()
         new_line.comments = line.comments.copy()
-        append_leaves(new_line, line, LL[: string_idx - 1])
+        try:
+            append_leaves(new_line, line, LL[: string_idx - 1])
+        except BracketMatchError:
+            # HACK: I believe there is currently a bug somewhere in
+            # right_hand_split() that is causing brackets to not be tracked
+            # properly by a shared BracketTracker.
+            append_leaves(new_line, line, LL[: string_idx - 1], preformatted=True)
 
         string_leaf = Leaf(token.STRING, LL[string_idx].value)
         LL[string_idx - 1].remove()
@@ -4598,7 +4615,9 @@ def line_to_string(line: Line) -> str:
     return str(line).strip("\n")
 
 
-def append_leaves(new_line: Line, old_line: Line, leaves: List[Leaf]) -> None:
+def append_leaves(
+    new_line: Line, old_line: Line, leaves: List[Leaf], preformatted: bool = False
+) -> None:
     """
     Append leaves (taken from @old_line) to @new_line, making sure to fix the
     underlying Node structure where appropriate.
@@ -4614,7 +4633,7 @@ def append_leaves(new_line: Line, old_line: Line, leaves: List[Leaf]) -> None:
     for old_leaf in leaves:
         new_leaf = Leaf(old_leaf.type, old_leaf.value)
         replace_child(old_leaf, new_leaf)
-        new_line.append(new_leaf)
+        new_line.append(new_leaf, preformatted=preformatted)
 
         for comment_leaf in old_line.comments_after(old_leaf):
             new_line.append(comment_leaf, preformatted=True)


Test patch:
diff --git a/tests/data/long_strings__regression.py b/tests/data/long_strings__regression.py
--- a/tests/data/long_strings__regression.py
+++ b/tests/data/long_strings__regression.py
@@ -330,6 +330,25 @@ def who(self):
             % (i)
         )
 
+def A():
+    def B():
+        def C():
+            def D():
+                def E():
+                    def F():
+                        def G():
+                            assert (
+                                c_float(val[0][0] / val[0][1]).value
+                                == c_float(value[0][0] / value[0][1]).value
+                            ), "%s didn't roundtrip" % tag
+
+class xxxxxxxxxxxxxxxxxxxxx(xxxx.xxxxxxxxxxxxx):
+    def xxxxxxx_xxxxxx(xxxx):
+        assert xxxxxxx_xxxx in [
+            x.xxxxx.xxxxxx.xxxxx.xxxxxx,
+            x.xxxxx.xxxxxx.xxxxx.xxxx,
+        ], ("xxxxxxxxxxx xxxxxxx xxxx (xxxxxx xxxx) %x xxx xxxxx" % xxxxxxx_xxxx)
+
 # output
 
 
@@ -742,3 +761,26 @@ def who(self):
             r"for pid in $(ps aux | grep paster | grep -v grep | grep '\-%d' | awk"
             r" '{print $2}'); do kill $pid; done" % (i)
         )
+
+
+def A():
+    def B():
+        def C():
+            def D():
+                def E():
+                    def F():
+                        def G():
+                            assert (
+                                c_float(val[0][0] / val[0][1]).value
+                                == c_float(value[0][0] / value[0][1]).value
+                            ), "%s didn't roundtrip" % tag
+
+
+class xxxxxxxxxxxxxxxxxxxxx(xxxx.xxxxxxxxxxxxx):
+    def xxxxxxx_xxxxxx(xxxx):
+        assert xxxxxxx_xxxx in [
+            x.xxxxx.xxxxxx.xxxxx.xxxxxx,
+            x.xxxxx.xxxxxx.xxxxx.xxxx,
+        ], (
+            "xxxxxxxxxxx xxxxxxx xxxx (xxxxxx xxxx) %x xxx xxxxx" % xxxxxxx_xxxx
+        )