Repository: psf/black

Issue:
Cannot parse multiline f-string containing multiline string
<!--
Please make sure that the bug is not already fixed either in newer versions or the
current development version. To confirm this, you have three options:

1. Update Black's version if a newer release exists: `pip install -U black`
2. Use the online formatter at <https://black.vercel.app/?version=main>, which will use
   the latest main branch.
3. Or run _Black_ on your machine:
   - create a new virtualenv (make sure it's the same Python version);
   - clone this repository;
   - run `pip install -e .[d]`;
   - run `pip install -r test_requirements.txt`
   - make sure it's sane by running `python -m pytest`; and
   - run `black` like you did last time.
-->

**Describe the bug**

<!-- A clear and concise description of what the bug is. -->

Black fails to parse a file containing a multiline f-string which contains a multiline string within curly braces

**To Reproduce**

<!--
Minimal steps to reproduce the behavior with source code and Black's configuration.
-->

I created this MRC

```python
f"""{'''
'''}"""
```

And just run black on the file:

```sh
$ black file.py
```

The resulting error is:

> error: cannot format file.py: Cannot parse: 1:5: f"""{'''

**Expected behavior**

<!-- A clear and concise description of what you expected to happen. -->

Successfully parse as it did in version <= 24.4.0

**Environment**

<!-- Please complete the following information: -->

- Black's version: 24.4.2
- OS and Python version: Windows 11, Python 3.10, 3.11, 3.12

**Additional context**

<!-- Add any other context about the problem here. -->

Related to #4329 


Discussion:
```
% python3 -m blib2to3.pgen2.tokenize multil.py
1,0-1,4:	FSTRING_START	'f"""'
1,4-1,4:	FSTRING_MIDDLE	''
1,4-1,5:	LBRACE	'{'
1,5-2,0:	FSTRING_MIDDLE	"'''\n"
2,0-2,3:	FSTRING_END	"'''"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 1115, in <module>
    tokenize(open(sys.argv[1]).readline)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 281, in tokenize
    tokenize_loop(readline, tokeneater)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 288, in tokenize_loop
    for token_info in generate_tokens(readline):
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 680, in generate_tokens
    fstring_state.leave_fstring()
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 541, in leave_fstring
    assert state == STATE_MIDDLE
AssertionError
```

It incorrectly emits FSTRING_MIDDLE even though the inner string is inside braces and isn't an f-string.

cc @tusharsadhwani 

Resolving patch:
diff --git a/src/blib2to3/pgen2/tokenize.py b/src/blib2to3/pgen2/tokenize.py
--- a/src/blib2to3/pgen2/tokenize.py
+++ b/src/blib2to3/pgen2/tokenize.py
@@ -622,10 +622,10 @@ def generate_tokens(
                 spos = strstart
                 epos = (lnum, end)
                 tokenline = contline + line
-                if (
-                    fstring_state.current() == STATE_NOT_FSTRING
-                    and not is_fstring_start(token)
-                ):
+                if fstring_state.current() in (
+                    STATE_NOT_FSTRING,
+                    STATE_IN_BRACES,
+                ) and not is_fstring_start(token):
                     yield (STRING, token, spos, epos, tokenline)
                     endprog_stack.pop()
                     parenlev = parenlev_stack.pop()


Test patch:
diff --git a/tests/data/cases/pep_701.py b/tests/data/cases/pep_701.py
--- a/tests/data/cases/pep_701.py
+++ b/tests/data/cases/pep_701.py
@@ -125,6 +125,9 @@
 
 f'{{\\"kind\\":\\"ConfigMap\\",\\"metadata\\":{{\\"annotations\\":{{}},\\"name\\":\\"cluster-info\\",\\"namespace\\":\\"amazon-cloudwatch\\"}}}}'
 
+f"""{'''
+'''}"""
+
 # output
 
 x = f"foo"
@@ -252,3 +255,6 @@
 f"{1:{2}d}"
 
 f'{{\\"kind\\":\\"ConfigMap\\",\\"metadata\\":{{\\"annotations\\":{{}},\\"name\\":\\"cluster-info\\",\\"namespace\\":\\"amazon-cloudwatch\\"}}}}'
+
+f"""{'''
+'''}"""