Repository: psf/black

Issue:
Black produces code with invalid syntax by splitting `\n` in half
**Describe the bug**:
Black produces code with invalid syntax by splitting `\n` in half when reformatting string.

**To Reproduce**:
1. Put this code in a file:
```py
message = (
    f"1. Go to Google Developers Console and log in with your Google account.\n"
    "(https://console.developers.google.com/)\n"
    "2. You should be prompted to create a new project (name does not matter).\n"
    "3. Click on Enable APIs and Services at the top.\n"
    "4. In the list of APIs choose or search for YouTube Data API v3 and "
    "click on it. Choose Enable.\n"
    "5. Click on Credentials on the left navigation bar.\n"
    "6. Click on Create Credential at the top.\n"
    '7. At the top click the link for "API key".\n'
    "8. No application restrictions are needed. Click Create at the bottom.\n"
    "9. You now have a key to add to `[p]set api youtube api_key <your_api_key_here>`"
)
```
2. Run _Black_ on it with no arguments.
3. See error:
```
error: cannot format test.py: INTERNAL ERROR: Black produced invalid code: invalid syntax (<unknown>, line 2). Please report a bug on https://github.com/psf/black/issues.  This invalid output might be helpful: C:\Users\Jakub\AppData\Local\Temp\blk_6n5icmuc.log
```
Contents of the log file:
```diff
  File "c:\users\jakub\onedrive\dokumenty\red-discordbot\.venv\lib\site-packages\black\__init__.py", line 6037, in assert_equivalent
    dst_ast = parse_ast(dst)
  File "c:\users\jakub\onedrive\dokumenty\red-discordbot\.venv\lib\site-packages\black\__init__.py", line 5948, in parse_ast
    return ast27.parse(src)
  File "c:\users\jakub\onedrive\dokumenty\red-discordbot\.venv\lib\site-packages\typed_ast\ast27.py", line 50, in parse
    return _ast27.parse(source, filename, mode)
message = (
    f"1. Go to Google Developers Console and log in with your Google account.\n"
    f"(https://console.developers.google.com/)\"
    f"n2. You should be prompted to create a new project (name does not matter)."
    f"\n3. Click on Enable APIs and Services at the top"
    f".\n4. In the list of APIs choose or search for YouTube Data API v3 "
    f"and click on it. Choose Enab"
    f"le.\n5. Click on Credentials on the left navigation "
    f"bar.\n6. Click on Create Credential at the"
    f' top.\n7. At the top click the link for "API'
    f' key".\n8. No application restrictions are needed. Click Create at the '
    f"bottom.\n9. You now have a key to add to `[p]set api youtube api_key"
    f" <your_api_key_here>`"
)
```

**Expected behavior**
_Black_ shouldn't produce invalid syntax.

**Environment:**
- Version: master (commit 3461338)
- OS and Python version: Windows/Python 3.8.3

**Does this bug also happen on master?**:
It *only* happens on master, it doesn't happen on current release as there's no string reformatting in it yet. Online formatter doesn't error, but it does produce this output which is obviously invalid syntax:
```py
message = (
    f"1. Go to Google Developers Console and log in with your Google account.\n"
    f"(https://console.developers.google.com/)\"
    f"n2. You should be prompted to create a new project (name does not matter)."
    f"\n3. Click on Enable APIs and Services at the top"
    f".\n4. In the list of APIs choose or search for YouTube Data API v3 "
    f"and click on it. Choose Enab"
    f"le.\n5. Click on Credentials on the left navigation "
    f"bar.\n6. Click on Create Credential at the"
    f' top.\n7. At the top click the link for "API'
    f' key".\n8. No application restrictions are needed. Click Create at the '
    f"bottom.\n9. You now have a key to add to `[p]set api youtube api_key"
    f" <your_api_key_here>`"
)
```

**Additional context**
None

I'm sorry for the amount of issues but I tried to run Black's master on Cog-Creators/Red-DiscordBot's code base which resulted in quite a few of errors and also some undesired (by me) changes which I wanted to at least report even if they're not gonna be changed.


Discussion:
No longer crashes.
@JelleZijlstra It seems that it still does, it's just that it requires `--experimental-string-processing` flag which I believe was the default back when I reported this.

Contents of `test_black.py` (equivalent to the ones in issue description):
```
message = (
    f"1. Go to Google Developers Console and log in with your Google account.\n"
    "(https://console.developers.google.com/)\n"
    "2. You should be prompted to create a new project (name does not matter).\n"
    "3. Click on Enable APIs and Services at the top.\n"
    "4. In the list of APIs choose or search for YouTube Data API v3 and "
    "click on it. Choose Enable.\n"
    "5. Click on Credentials on the left navigation bar.\n"
    "6. Click on Create Credential at the top.\n"
    '7. At the top click the link for "API key".\n'
    "8. No application restrictions are needed. Click Create at the bottom.\n"
    "9. You now have a key to add to `[p]set api youtube api_key <your_api_key_here>`"
)
```

Commands I ran:
```
python -m venv test_black_venv
test_black_venv\Scripts\activate.bat
python -m pip install -U git+https://github.com/psf/black
black test_black.py --experimental-string-processing
```

Output of `black`:
```
error: cannot format test_black.py: INTERNAL ERROR: Black produced invalid code on pass 1: The requested source code has invalid Python 3 syntax.
If you are trying to format Python 2 files please reinstall Black with the 'python2' extra: `python3 -m pip install black[python2]`.. Please report a bug on https://github.com/psf/black/issues.  This invalid output might be helpful: C:\Users\Jakub\AppData\Local\Temp\blk_1sf1_qup.log
Oh no! 💥 💔 💥
1 file failed to reformat.
```

Contents of `blk_1sf1_qup.log`:
```
  File "c:\users\jakub\desktop\test_black_real_quick\lib\site-packages\black\__init__.py", line 994, in assert_equivalent
    dst_ast = parse_ast(dst)
  File "c:\users\jakub\desktop\test_black_real_quick\lib\site-packages\black\parsing.py", line 125, in parse_ast
    raise SyntaxError(
message = (
    f"1. Go to Google Developers Console and log in with your Google account.\n"
    f"(https://console.developers.google.com/)\"
    f"n2. You should be prompted to create a new project (name does not matter)."
    f"\n3. Click on Enable APIs and Services at the top"
    f".\n4. In the list of APIs choose or search for YouTube Data API v3 "
    f"and click on it. Choose Enab"
    f"le.\n5. Click on Credentials on the left navigation "
    f"bar.\n6. Click on Create Credential at the"
    f' top.\n7. At the top click the link for "API'
    f' key".\n8. No application restrictions are needed. Click Create at the '
    f"bottom.\n9. You now have a key to add to `[p]set api youtube api_key"
    f" <your_api_key_here>`"
)
```
Can confirm this still exists on main. Apologies for the erroneous closure!
> Apologies for the erroneous closure!

No worries, I was actually updating my other issues to include the information about the need of experimental flag back in August but somehow missed this one 😄 
I ran Black against the up-to-date codebase of the project to see if there are any other cases experimental string processing fails on. It seems that I found what appears to be another case for *probably* this bug.

Here it is:
```py
x = (
    "\N{BLACK RIGHT-POINTING TRIANGLE WITH DOUBLE VERTICAL BAR}\N{VARIATION SELECTOR-16}"
)
```

Resolving patch:
diff --git a/src/black/trans.py b/src/black/trans.py
--- a/src/black/trans.py
+++ b/src/black/trans.py
@@ -15,6 +15,7 @@
     List,
     Optional,
     Sequence,
+    Set,
     Tuple,
     TypeVar,
     Union,
@@ -1243,6 +1244,61 @@ def more_splits_should_be_made() -> bool:
             last_line.comments = line.comments.copy()
             yield Ok(last_line)
 
+    def _iter_nameescape_slices(self, string: str) -> Iterator[Tuple[Index, Index]]:
+        """
+        Yields:
+            All ranges of @string which, if @string were to be split there,
+            would result in the splitting of an \\N{...} expression (which is NOT
+            allowed).
+        """
+        # True - the previous backslash was unescaped
+        # False - the previous backslash was escaped *or* there was no backslash
+        previous_was_unescaped_backslash = False
+        it = iter(enumerate(string))
+        for idx, c in it:
+            if c == "\\":
+                previous_was_unescaped_backslash = not previous_was_unescaped_backslash
+                continue
+            if not previous_was_unescaped_backslash or c != "N":
+                previous_was_unescaped_backslash = False
+                continue
+            previous_was_unescaped_backslash = False
+
+            begin = idx - 1  # the position of backslash before \N{...}
+            for idx, c in it:
+                if c == "}":
+                    end = idx
+                    break
+            else:
+                # malformed nameescape expression?
+                # should have been detected by AST parsing earlier...
+                raise RuntimeError(f"{self.__class__.__name__} LOGIC ERROR!")
+            yield begin, end
+
+    def _iter_fexpr_slices(self, string: str) -> Iterator[Tuple[Index, Index]]:
+        """
+        Yields:
+            All ranges of @string which, if @string were to be split there,
+            would result in the splitting of an f-expression (which is NOT
+            allowed).
+        """
+        if "f" not in get_string_prefix(string).lower():
+            return
+
+        for match in re.finditer(self.RE_FEXPR, string, re.VERBOSE):
+            yield match.span()
+
+    def _get_illegal_split_indices(self, string: str) -> Set[Index]:
+        illegal_indices: Set[Index] = set()
+        iterators = [
+            self._iter_fexpr_slices(string),
+            self._iter_nameescape_slices(string),
+        ]
+        for it in iterators:
+            for begin, end in it:
+                illegal_indices.update(range(begin, end + 1))
+        return illegal_indices
+
     def _get_break_idx(self, string: str, max_break_idx: int) -> Optional[int]:
         """
         This method contains the algorithm that StringSplitter uses to
@@ -1272,40 +1328,15 @@ def _get_break_idx(self, string: str, max_break_idx: int) -> Optional[int]:
         assert is_valid_index(max_break_idx)
         assert_is_leaf_string(string)
 
-        _fexpr_slices: Optional[List[Tuple[Index, Index]]] = None
-
-        def fexpr_slices() -> Iterator[Tuple[Index, Index]]:
-            """
-            Yields:
-                All ranges of @string which, if @string were to be split there,
-                would result in the splitting of an f-expression (which is NOT
-                allowed).
-            """
-            nonlocal _fexpr_slices
-
-            if _fexpr_slices is None:
-                _fexpr_slices = []
-                for match in re.finditer(self.RE_FEXPR, string, re.VERBOSE):
-                    _fexpr_slices.append(match.span())
-
-            yield from _fexpr_slices
-
-        is_fstring = "f" in get_string_prefix(string).lower()
+        _illegal_split_indices = self._get_illegal_split_indices(string)
 
-        def breaks_fstring_expression(i: Index) -> bool:
+        def breaks_unsplittable_expression(i: Index) -> bool:
             """
             Returns:
                 True iff returning @i would result in the splitting of an
-                f-expression (which is NOT allowed).
+                unsplittable expression (which is NOT allowed).
             """
-            if not is_fstring:
-                return False
-
-            for (start, end) in fexpr_slices():
-                if start <= i < end:
-                    return True
-
-            return False
+            return i in _illegal_split_indices
 
         def passes_all_checks(i: Index) -> bool:
             """
@@ -1329,7 +1360,7 @@ def passes_all_checks(i: Index) -> bool:
                 is_space
                 and is_not_escaped
                 and is_big_enough
-                and not breaks_fstring_expression(i)
+                and not breaks_unsplittable_expression(i)
             )
 
         # First, we check all indices BELOW @max_break_idx.


Test patch:
diff --git a/tests/data/long_strings.py b/tests/data/long_strings.py
--- a/tests/data/long_strings.py
+++ b/tests/data/long_strings.py
@@ -207,6 +207,38 @@ def foo():
     " of it."
 )
 
+string_with_nameescape = (
+    "........................................................................ \N{LAO KO LA}"
+)
+
+string_with_nameescape = (
+    "........................................................................... \N{LAO KO LA}"
+)
+
+string_with_nameescape = (
+    "............................................................................ \N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    "...................................................................... \\\N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    "......................................................................... \\\N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    ".......................................................................... \\\N{LAO KO LA}"
+)
+
+string_with_escaped_nameescape = (
+    "........................................................................ \\N{LAO KO LA}"
+)
+
+string_with_escaped_nameescape = (
+    "........................................................................... \\N{LAO KO LA}"
+)
+
 
 # output
 
@@ -587,3 +619,43 @@ def foo():
     "This is a really long string that can't be merged because it has a likely pragma at the end"  # pylint: disable=some-pylint-check
     " of it."
 )
+
+string_with_nameescape = (
+    "........................................................................"
+    " \N{LAO KO LA}"
+)
+
+string_with_nameescape = (
+    "..........................................................................."
+    " \N{LAO KO LA}"
+)
+
+string_with_nameescape = (
+    "............................................................................"
+    " \N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    "......................................................................"
+    " \\\N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    "........................................................................."
+    " \\\N{LAO KO LA}"
+)
+
+string_with_nameescape_and_escaped_backslash = (
+    ".........................................................................."
+    " \\\N{LAO KO LA}"
+)
+
+string_with_escaped_nameescape = (
+    "........................................................................ \\N{LAO"
+    " KO LA}"
+)
+
+string_with_escaped_nameescape = (
+    "..........................................................................."
+    " \\N{LAO KO LA}"
+)
diff --git a/tests/data/long_strings__regression.py b/tests/data/long_strings__regression.py
--- a/tests/data/long_strings__regression.py
+++ b/tests/data/long_strings__regression.py
@@ -514,6 +514,10 @@ async def foo(self):
 
 x = F"This is a long string which contains an f-expr that should not split {{{[i for i in range(5)]}}}."
 
+x = (
+    "\N{BLACK RIGHT-POINTING TRIANGLE WITH DOUBLE VERTICAL BAR}\N{VARIATION SELECTOR-16}"
+)
+
 
 # output
 
@@ -1142,3 +1146,7 @@ async def foo(self):
     "This is a long string which contains an f-expr that should not split"
     f" {{{[i for i in range(5)]}}}."
 )
+
+x = (
+    "\N{BLACK RIGHT-POINTING TRIANGLE WITH DOUBLE VERTICAL BAR}\N{VARIATION SELECTOR-16}"
+)