Repository: sharkdp/bat (language: rust)

Issue:
Slowness/choppy with specific Python syntax
<!-- Hey there, thank you for creating an issue! -->

**Describe the bug you encountered:**

bat consumes 100% of one cpu and gives choppy output (~3 lines per second on my box) for a specific Python sample. Shell reproducer:

```shell
{ echo "foo={"; for i in {1..100}; do echo "                   #'a': '$i',"; done; } | bat --language python
```
This is sensitive to amount of whitespace within the echo. Adding or removing some might speed it up some, but is still choppy with some other amounts.

**What did you expect to happen instead?**

Normal smooth output.

**How did you install `bat`?**

<!-- apt-get, homebrew, GitHub release, etc. -->

git master

---

**bat version and environment**

<!--
In order to reproduce your issue, please add some information about the environment
in which you're running bat. To do this, run the full `bat` command that demonstrates
the bug, and attach the `--diagnostic` option:

    bat [other options and arguments…] --diagnostic

Finally, paste the Markdown output here. Please make sure that it does not reveal any
personal information.


If you are running bat 0.17.1 or older (where --diagnostic is not available), please
run the script at

    https://github.com/sharkdp/bat/blob/master/diagnostics/info.sh

(click "Raw" to get the actual source code) and paste the Markdown output here. If you
are on Windows, please let us know your bat version and your Windows version.
-->

#### Software version

bat 0.18.1 (35f3127-modified)

#### Operating system

Linux 5.8.0-55-generic

#### Command-line

```bash
bat --diagnostic 
```

#### Environment variables

```bash
SHELL=/bin/bash
PAGER=less
LESS=-iMR
BAT_PAGER='less -R'
BAT_CACHE_PATH=<not set>
BAT_CONFIG_PATH=<not set>
BAT_OPTS=<not set>
BAT_STYLE=<not set>
BAT_TABS=<not set>
BAT_THEME=Dracula
XDG_CONFIG_HOME=<not set>
XDG_CACHE_HOME=<not set>
COLORTERM=truecolor
NO_COLOR=<not set>
MANPAGER=<not set>
```

#### Config file

Could not read contents of '/home/scop/.config/bat/config': No such file or directory (os error 2).

#### Compile time information

- Profile: release
- Target triple: x86_64-unknown-linux-gnu
- Family: unix
- OS: linux
- Architecture: x86_64
- Pointer width: 64
- Endian: little
- CPU features: fxsr,sse,sse2
- Host: x86_64-unknown-linux-gnu

#### Less version

```
> less --version 
less 551 (GNU regular expressions)
Copyright (C) 1984-2019  Mark Nudelman

less comes with NO WARRANTY, to the extent permitted by law.
For information about the terms of redistribution,
see the file named README in the less distribution.
Home page: http://www.greenwoodsoftware.com/less
```



Discussion / hints:
Thanks a lot for the bug report!

I can confirm that it is slow. I also tried with the optional `fancy-regex` backend in `syntect`, but it is still slow.  A tip is to add `--pager=never` to the `bat` command in your reproducer to see the slowness live. Interestingly, while this is slow:

```python
foo={
                   #'a': '1',
                   #'a': '2',
                   #'a': '3',
                   #'a': '4',
                   #'a': '5',
                   ...
```

this (just removing the first `#`) makes the whole file fast:

```python
foo={
                   'a': '1',
                   #'a': '2',
                   #'a': '3',
                   #'a': '4',
                   #'a': '5',
                   ...

```

Yep, noticed myself too that various tiny changes to the input change the problem extent or remove it altogether.
I instrumented syntect with some debug printing of regex pattern timing, and found:

```
regex "(?=(?x:\n  \\s+                      # whitespace\n  | [urfb]*\"(?:\\\\.|[^\"])*\" # strings\n  | [urfb]*\'(?:\\\\.|[^\'])*\' # ^\n  | [\\d.ej]+               # numerics\n  | [+*/%@-] | // | and | or # operators\n  | (\\b[\\p{L}_][\\p{L}\\p{N}_]*\\b[ ]*\\.[ ]*)*\\b[\\p{L}_][\\p{L}\\p{N}_]*\\b               # a path\n)*:|\\s*\\*\\*)" took 131.704239ms to find a match
regex "(?=(?x:\n  \\s+                      # whitespace\n  | [urfb]*\"(?:\\\\.|[^\"])*\" # strings\n  | [urfb]*\'(?:\\\\.|[^\'])*\' # ^\n  | [\\d.ej]+               # numerics\n  | [+*/%@-] | // | and | or # operators\n  | (\\b[\\p{L}_][\\p{L}\\p{N}_]*\\b[ ]*\\.[ ]*)*\\b[\\p{L}_][\\p{L}\\p{N}_]*\\b               # a path\n)*[,}]|\\s*\\*)" took 134.816448ms to find a match
```

this corresponds with https://github.com/sublimehq/Packages/blob/09cb8000b383c2f32de6473f44fce7d43cb8772f/Python/Python.sublime-syntax#L57-L65, specifically https://github.com/sublimehq/Packages/blob/09cb8000b383c2f32de6473f44fce7d43cb8772f/Python/Python.sublime-syntax#L1013-L1025

I wouldn't be surprised to see this replaced with ST's new branch point functionality - which syntect doesn't support yet.
I think we may want to patch our Python.sublime-syntax file with a regex which performs better or just always scope/parse it as a "dictionary" instead of trying to determine whether it is a "set" or not as it likely makes little difference to highlighting.


Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 14e383cdbb..b15745c8eb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@
 
 ## Bugfixes
 
+- Python syntax highlighting no longer suffers from abysmal performance in specific scenarios. See #1688 (@keith-hall)
 
 ## Other
 
diff --git a/assets/patches/Python.sublime-syntax.patch b/assets/patches/Python.sublime-syntax.patch
new file mode 100644
index 0000000000..36a5089418
--- /dev/null
+++ b/assets/patches/Python.sublime-syntax.patch
@@ -0,0 +1,15 @@
+diff --git syntaxes/01_Packages/Python/Python.sublime-syntax syntaxes/01_Packages/Python/Python.sublime-syntax
+index 2acd86d8..86257f7b 100644
+--- syntaxes/01_Packages/Python/Python.sublime-syntax
++++ syntaxes/01_Packages/Python/Python.sublime-syntax
+@@ -988,10 +988,6 @@ contexts:
+         - match: \}
+           scope: punctuation.section.mapping-or-set.end.python
+           set: after-expression
+-        - match: (?={{simple_expression}}:|\s*\*\*)
+-          set: inside-dictionary
+-        - match: (?={{simple_expression}}[,}]|\s*\*)
+-          set: inside-set
+         - match: ','
+           scope: punctuation.separator.set.python
+           set: inside-set


Test patch: