CVE-2024-21503: Insufficient Information (NVD-CWE-noinfo)
value: Versions of the package black before 24.3.0 are vulnerable to Regular Expression Denial of Service (ReDoS) via the lines_with_leading_tabs_expanded function in the strings.py file. An attacker could exploit this vulnerability by crafting a malicious input that causes a denial of service.Exploiting this vulnerability is possible when running Black on untrusted input, or if you habitually put thousands of leading tab characters in your docstrings.
CWE description: There is insufficient information about the issue to classify it; details are unkown or unspecified.
Published: 2024-03-19T05:15Z
Repository: https://github.com/psf/black
Language: Python
Files changed: CHANGES.md, src/black/strings.py, tests/test_black.py
Commit date: 2024-03-15 12:06:12 -0700
Commit message: Fix catastrophic performance in lines_with_leading_tabs_expanded() (#4278)


Diff: diff --git a/CHANGES.md b/CHANGES.md
index c255c2a..ffd24fe 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -4,10 +4,18 @@
 
 ### Highlights
 
 <!-- Include any especially major or disruptive changes here -->
 
+This release is a milestone: it fixes Black's first CVE security vulnerability. If you
+run Black on untrusted input, or if you habitually put thousands of leading tab
+characters in your docstrings, you are strongly encouraged to upgrade immediately to fix
+[CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).
+
+This release also fixes a bug in Black's AST safety check that allowed Black to make
+incorrect changes to certain f-strings that are valid in Python 3.12 and higher.
+
 ### Stable style
 
 <!-- Changes that affect Black's stable style -->
 
 - Don't move comments along with delimiters, which could cause crashes (#4248)
@@ -34,11 +42,14 @@
 
 <!-- Changes to the parser or to version autodetection -->
 
 ### Performance
 
-<!-- Changes that improve Black's performance. -->
+- Fix catastrophic performance on docstrings that contain large numbers of leading tab
+  characters. This fixes
+  [CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).
+  (#4278)
 
 ### Output
 
 <!-- Changes to Black's terminal output and error messages -->
 
diff --git a/src/black/strings.py b/src/black/strings.py
index 0e0f968..baa8816 100644
--- a/src/black/strings.py
+++ b/src/black/strings.py
@@ -12,11 +12,10 @@
 
 STRING_PREFIX_CHARS: Final = "furbFURB"  # All possible string prefix characters.
 STRING_PREFIX_RE: Final = re.compile(
     r"^([" + STRING_PREFIX_CHARS + r"]*)(.*)$", re.DOTALL
 )
-FIRST_NON_WHITESPACE_RE: Final = re.compile(r"\s*\t+\s*(\S)")
 UNICODE_ESCAPE_RE: Final = re.compile(
     r"(?P<backslashes>\\+)(?P<body>"
     r"(u(?P<u>[a-fA-F0-9]{4}))"  # Character with 16-bit hex value xxxx
     r"|(U(?P<U>[a-fA-F0-9]{8}))"  # Character with 32-bit hex value xxxxxxxx
     r"|(x(?P<x>[a-fA-F0-9]{2}))"  # Character with hex value hh
@@ -49,22 +48,17 @@ def lines_with_leading_tabs_expanded(s: str) -> List[str]:
     Splits string into lines and expands only leading tabs (following the normal
     Python rules)
     """
     lines = []
     for line in s.splitlines():
-        # Find the index of the first non-whitespace character after a string of
-        # whitespace that includes at least one tab
-        match = FIRST_NON_WHITESPACE_RE.match(line)
-        if match:
-            first_non_whitespace_idx = match.start(1)
-
-            lines.append(
-                line[:first_non_whitespace_idx].expandtabs()
-                + line[first_non_whitespace_idx:]
-            )
-        else:
+        stripped_line = line.lstrip()
+        if not stripped_line or stripped_line == line:
             lines.append(line)
+        else:
+            prefix_length = len(line) - len(stripped_line)
+            prefix = line[:prefix_length].expandtabs()
+            lines.append(prefix + stripped_line)
     if s.endswith("\n"):
         lines.append("")
     return lines
 
 
diff --git a/tests/test_black.py b/tests/test_black.py
index 7088415..ecea4a0 100644
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -46,10 +46,11 @@
 from black.debug import DebugVisitor
 from black.mode import Mode, Preview
 from black.output import color_diff, diff
 from black.parsing import ASTSafetyError
 from black.report import Report
+from black.strings import lines_with_leading_tabs_expanded
 
 # Import other test classes
 from tests.util import (
     DATA_DIR,
     DEFAULT_MODE,
@@ -2039,10 +2040,21 @@ def test_line_ranges_in_pyproject_toml(self) -> None:
         assert result.stderr_bytes is not None
         assert (
             b"Cannot use line-ranges in the pyproject.toml file." in result.stderr_bytes
         )
 
+    def test_lines_with_leading_tabs_expanded(self) -> None:
+        # See CVE-2024-21503. Mostly test that this completes in a reasonable
+        # time.
+        payload = "\t" * 10_000
+        assert lines_with_leading_tabs_expanded(payload) == [payload]
+
+        tab = " " * 8
+        assert lines_with_leading_tabs_expanded("\tx") == [f"{tab}x"]
+        assert lines_with_leading_tabs_expanded("\t\tx") == [f"{tab}{tab}x"]
+        assert lines_with_leading_tabs_expanded("\tx\n  y") == [f"{tab}x", "  y"]
+
 
 class TestCaching:
     def test_get_cache_dir(
         self,
         tmp_path: Path,

Vulnerable code: <!-- Changes that improve Black's performance. -->
FIRST_NON_WHITESPACE_RE: Final = re.compile(r"\s*\t+\s*(\S)")
        # Find the index of the first non-whitespace character after a string of
        # whitespace that includes at least one tab
        match = FIRST_NON_WHITESPACE_RE.match(line)
        if match:
            first_non_whitespace_idx = match.start(1)

            lines.append(
                line[:first_non_whitespace_idx].expandtabs()
                + line[first_non_whitespace_idx:]
            )
        else:
Fixed code: This release is a milestone: it fixes Black's first CVE security vulnerability. If you
run Black on untrusted input, or if you habitually put thousands of leading tab
characters in your docstrings, you are strongly encouraged to upgrade immediately to fix
[CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).

This release also fixes a bug in Black's AST safety check that allowed Black to make
incorrect changes to certain f-strings that are valid in Python 3.12 and higher.

- Fix catastrophic performance on docstrings that contain large numbers of leading tab
  characters. This fixes
  [CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).
  (#4278)
        stripped_line = line.lstrip()
        if not stripped_line or stripped_line == line:
        else:
            prefix_length = len(line) - len(stripped_line)
            prefix = line[:prefix_length].expandtabs()
            lines.append(prefix + stripped_line)
from black.strings import lines_with_leading_tabs_expanded
    def test_lines_with_leading_tabs_expanded(self) -> None:
        # See CVE-2024-21503. Mostly test that this completes in a reasonable
        # time.
        payload = "\t" * 10_000
        assert lines_with_leading_tabs_expanded(payload) == [payload]

        tab = " " * 8
        assert lines_with_leading_tabs_expanded("\tx") == [f"{tab}x"]
        assert lines_with_leading_tabs_expanded("\t\tx") == [f"{tab}{tab}x"]
        assert lines_with_leading_tabs_expanded("\tx\n  y") == [f"{tab}x", "  y"]