Repository: executablebooks/mdit-py-plugins

Issue:
Infinite loop in `amsmath` extension for a simple input
### Describe the bug

**context**

The following markdown document, `bug.md` is a minimal version reproducing the issue (taken from a larger document):

```markdown
---
myst:
  enable_extensions: ["amsmath"]
---

> **Example**
>\begin{equation}A=
\begin{matrix}
-0.707 &  0.408 &  0.577 \\
-0.707 & -0.408 & -0.577 \\
-0.    & -0.816 &  0.577 
\end{matrix}
\end{equation}
```

(This renders properly in a markdown cell in Jupyter Lab.)
I try to convert it to HTML as follows:

```bash
myst-docutils-demo bug.md
```

**expectation**

To get some HTML output.

**bug**

The script hangs without error message, until I interrupt it with `ctrl+C`:

```python
Traceback (most recent call last):
  File "~/venv/bin/myst-docutils-demo", line 8, in <module>
    sys.exit(cli_html5_demo())
             ^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/myst_parser/parsers/docutils_.py", line 379, in cli_html5_demo
    publish_cmdline(
  File "~/venv/lib64/python3.11/site-packages/docutils/core.py", line 402, in publish_cmdline
    output = publisher.publish(
             ^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/docutils/core.py", line 234, in publish
    self.document = self.reader.read(self.source, self.parser,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/docutils/readers/__init__.py", line 70, in read
    self.parse()
  File "~/venv/lib64/python3.11/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File "~/venv/lib64/python3.11/site-packages/myst_parser/parsers/docutils_.py", line 318, in parse
    parser.render(inputstring)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/main.py", line 291, in render
    return self.renderer.render(self.parse(src, env), self.options, env)
                                ^^^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/markdown_it/main.py", line 276, in parse
    self.core.process(state)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/parser_core.py", line 45, in process
    rule(state)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/rules_core/block.py", line 13, in block
    state.md.block.parse(state.src, state.md, state.env, state.tokens)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/parser_block.py", line 110, in parse
    self.tokenize(state, state.line, state.lineMax)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/parser_block.py", line 85, in tokenize
    if rule(state, line, endLine, False):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/markdown_it/rules_block/blockquote.py", line 280, in blockquote
    state.md.block.tokenize(state, startLine, nextLine)
  File "~/venv/lib64/python3.11/site-packages/markdown_it/parser_block.py", line 85, in tokenize
    if rule(state, line, endLine, False):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/markdown_it/rules_block/fence.py", line 16, in fence
    if state.is_code_block(startLine):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/venv/lib64/python3.11/site-packages/markdown_it/rules_block/state_block.py", line 257, in is_code_block
    def is_code_block(self, line: int) -> bool:

KeyboardInterrupt
```

**problem**

I ran into a problem when trying to build a PDF document from several notebooks that have been created by several authors. This is fairly common pattern, and Jupyter can handle it, so it seems fair to expect that this should work.


### Reproduce the bug

The context above contains all the steps to reproduce the problem.

### List your environment

The command `jupyter-book --version` gives the following error:

```
usage: jupyter-book [-h] command
jupyter-book: error: the following arguments are required: command
```

`jupyter-book --version` gives:

```
Running Jupyter Book version: 0.6.5
```

Output of `pip list`:

```
Package                           Version
--------------------------------- ------------------
alabaster                         0.7.14
anyio                             4.3.0
appdirs                           1.4.4
argon2-cffi                       23.1.0
argon2-cffi-bindings              21.2.0
arrow                             1.3.0
asn1crypto                        1.5.1
astropy                           6.1.3
astropy-iers-data                 0.2024.9.2.0.33.23
asttokens                         2.4.1
async-lru                         2.0.4
atomicwrites                      1.4.1
attrs                             23.1.0
babel                             2.16.0
backcall                          0.2.0
backports.entry-points-selectable 1.2.0
backports.functools-lru-cache     1.6.6
beautifulsoup4                    4.12.2
beniget                           0.4.1
binaryornot                       0.4.4
bitarray                          2.8.2
bitstring                         4.1.2
bleach                            6.1.0
Bottleneck                        1.3.7
build                             1.2.1
CacheControl                      0.13.1
cachy                             0.3.0
certifi                           2024.8.30
cffi                              1.16.0
cfgv                              3.4.0
chardet                           5.2.0
charset-normalizer                3.3.2
cleo                              2.0.1
click                             8.1.7
cloudpickle                       3.0.0
colorama                          0.4.6
comm                              0.2.2
commonmark                        0.9.1
contourpy                         1.2.0
cookiecutter                      2.6.0
crashtest                         0.4.1
cryptography                      41.0.5
cycler                            0.12.1
Cython                            3.0.4
deap                              1.4.1
debugpy                           1.8.1
decorator                         5.1.1
defusedxml                        0.7.1
deprecation                       2.1.0
distlib                           0.3.7
distro                            1.8.0
docopt                            0.6.2
docutils                          0.21.2
doit                              0.36.0
dulwich                           0.21.6
ecdsa                             0.18.0
editables                         0.5
exceptiongroup                    1.1.3
execnet                           2.0.2
executing                         2.0.1
fastjsonschema                    2.19.1
filelock                          3.13.0
flit_core                         3.9.0
fonttools                         4.47.0
fqdn                              1.5.1
fsspec                            2023.10.0
furo                              2024.8.6
future                            0.18.3
gast                              0.5.4
glob2                             0.7
gmpy2                             2.1.5
greenlet                          3.0.3
h11                               0.14.0
html5lib                          1.1
httpcore                          1.0.5
httpx                             0.27.0
hypothesis                        6.90.0
identify                          2.6.0
idna                              3.8
imagesize                         1.4.1
importlib-metadata                6.8.0
importlib-resources               6.1.0
iniconfig                         2.0.0
intersphinx_registry              0.2408.13
intervaltree                      3.1.0
intreehooks                       1.0
ipaddress                         1.0.23
ipykernel                         6.29.4
ipympl                            0.9.4
ipytest                           0.14.2
ipython                           8.17.2
ipython-genutils                  0.2.0
ipywidgets                        8.1.2
isoduration                       20.11.0
jaraco.classes                    3.3.0
jax                               0.4.31
jaxlib                            0.4.31
jedi                              0.19.1
jeepney                           0.8.0
Jinja2                            3.1.4
joblib                            1.3.2
json5                             0.9.25
jsonpointer                       3.0.0
jsonschema                        4.18.0
jsonschema-specifications         2023.12.1
jupyter-book                      0.6.5
jupyter-cache                     1.0.0
jupyter_client                    8.6.1
jupyter_core                      5.7.2
jupyter-events                    0.10.0
jupyter-lsp                       2.2.5
jupyter_packaging                 0.12.3
jupyter_server                    2.14.0
jupyter_server_terminals          0.5.3
jupyterlab                        4.2.0
jupyterlab_pygments               0.3.0
jupyterlab_server                 2.27.1
jupyterlab_widgets                3.0.10
jupytext                          1.16.4
keyring                           24.2.0
keyrings.alt                      5.0.0
kiwisolver                        1.4.4
latexcodec                        3.0.0
liac-arff                         2.5.0
libsass                           0.23.0
lockfile                          0.12.2
lxml                              4.9.3
markdown-it-py                    3.0.0
MarkupSafe                        2.1.5
matplotlib                        3.8.2
matplotlib-inline                 0.1.6
mdit-py-plugins                   0.4.1
mdurl                             0.1.2
mistune                           3.0.2
ml-dtypes                         0.4.0
mock                              5.1.0
more-itertools                    10.1.0
mpmath                            1.3.0
msgpack                           1.0.7
myst-nb                           1.1.1
myst-parser                       4.0.0
nbclient                          0.10.0
nbconvert                         7.16.4
nbformat                          5.10.4
nest-asyncio                      1.6.0
netaddr                           0.9.0
netifaces                         0.11.0
nodeenv                           1.9.1
notebook_shim                     0.2.4
numexpr                           2.8.7
numpy                             1.26.2
numpydoc                          1.8.0
opt-einsum                        3.3.0
overrides                         7.7.0
packaging                         24.1
pandas                            2.1.3
pandocfilters                     1.5.1
parso                             0.8.3
pastel                            0.2.1
pathlib2                          2.3.7.post1
pathspec                          0.11.2
pbr                               5.11.1
pexpect                           4.8.0
pickleshare                       0.7.5
pillow                            10.2.0
pip                               23.3.2
pip-tools                         7.4.1
pkginfo                           1.9.6
platformdirs                      3.11.0
pluggy                            1.3.0
ply                               3.11
pooch                             1.8.0
pre-commit                        3.8.0
prometheus_client                 0.20.0
prompt-toolkit                    3.0.41
psutil                            5.9.6
ptyprocess                        0.7.0
pure-eval                         0.2.2
py                                1.11.0
py-expression-eval                0.3.14
pyasn1                            0.5.0
pybind11                          2.11.1
pybtex                            0.24.0
pybtex-docutils                   1.0.3
pycparser                         2.21
pycryptodome                      3.19.0
pydevtool                         0.3.0
pyerfa                            2.0.1.4
Pygments                          2.18.0
pylev                             1.4.0
PyNaCl                            1.5.0
pyparsing                         3.1.1
pyproject_hooks                   1.1.0
pyrsistent                        0.20.0
pytest                            7.4.3
pytest-xdist                      3.3.1
python-dateutil                   2.8.2
python-json-logger                2.0.7
python-slugify                    8.0.4
pythran                           0.14.0
pytoml                            0.1.21
pytz                              2023.3.post1
PyYAML                            6.0.1
pyzmq                             25.1.2
rapidfuzz                         2.15.2
referencing                       0.35.1
regex                             2023.10.3
requests                          2.32.3
requests-toolbelt                 1.0.0
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rich                              13.6.0
rich-click                        1.7.0
rpds-py                           0.18.1
ruamel.yaml                       0.18.6
ruamel.yaml.clib                  0.2.8
ruff                              0.6.3
scandir                           1.10.0
schemdraw                         0.19
scipy                             1.11.4
SecretStorage                     3.3.3
semantic-version                  2.10.0
Send2Trash                        1.8.3
setuptools                        74.1.2
setuptools-scm                    8.0.4
shellingham                       1.5.4
simplegeneric                     0.8.1
simplejson                        3.19.2
six                               1.16.0
sniffio                           1.3.1
snowballstemmer                   2.2.0
sortedcontainers                  2.4.0
soupsieve                         2.5
Sphinx                            8.0.2
sphinx-autobuild                  2024.9.3
sphinx-basic-ng                   1.0.0b2
sphinx-bootstrap-theme            0.8.1
sphinx-codeautolink               0.15.2
sphinx-copybutton                 0.5.2
sphinxcontrib-applehelp           2.0.0
sphinxcontrib-bibtex              2.6.2
sphinxcontrib-devhelp             2.0.0
sphinxcontrib-htmlhelp            2.1.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              2.0.0
sphinxcontrib-serializinghtml     2.0.0
sphinxcontrib-svg2pdfconverter    1.2.2
sphinxcontrib-websupport          2.0.0
SQLAlchemy                        2.0.34
stack-data                        0.6.3
starlette                         0.38.4
sympy                             1.12
tabulate                          0.9.0
terminado                         0.18.1
text-unidecode                    1.3
threadpoolctl                     3.2.0
tinycss2                          1.3.0
toml                              0.10.2
tomli                             2.0.1
tomli_w                           1.0.0
tomlkit                           0.12.1
tornado                           6.4
tqdm                              4.66.5
traitlets                         5.13.0
types-python-dateutil             2.9.0.20240906
typing_extensions                 4.8.0
tzdata                            2023.3
ujson                             5.8.0
uri-template                      1.3.0
urllib3                           2.2.2
uvicorn                           0.30.6
versioneer                        0.29
virtualenv                        20.24.6
watchfiles                        0.24.0
wcwidth                           0.2.8
webcolors                         24.8.0
webencodings                      0.5.1
websocket-client                  1.8.0
websockets                        13.0.1
wheel                             0.41.2
widgetsnbextension                4.0.10
xlrd                              2.0.1
zipfile36                         0.1.3
zipp                              3.17.0
```

Hints from issue comments:
@tovrstra: Found it: the `amsmath` extension erroneously lists the `pmatrix`, `bmatrix`, `Bmatrix`, `vmatrix` and `Vmatrix` as top-level equation environments. Removing them from the `ENVIRONMENTS` list fixes the issue.@chrisjsewell: @tovrstra you really went for the nuclear option with #118 😅

But that is not the fix, I can confirm that I can recreate the loop with only this (you have to have the middle newline)

```
>\begin{equation}

\end{equation}
```

Solution patch:
diff --git a/mdit_py_plugins/amsmath/__init__.py b/mdit_py_plugins/amsmath/__init__.py
index 0aed5c1..0712618 100644
--- a/mdit_py_plugins/amsmath/__init__.py
+++ b/mdit_py_plugins/amsmath/__init__.py
@@ -54,7 +54,7 @@ ENVIRONMENTS = [
 # whose total width is the actual width of the contents;
 # thus they can be used as a component in a containing expression
 
-RE_OPEN = re.compile(r"\\begin\{(" + "|".join(ENVIRONMENTS) + r")([\*]?)\}")
+RE_OPEN = r"\\begin\{(" + "|".join(ENVIRONMENTS) + r")([\*]?)\}"
 
 
 def amsmath_plugin(
@@ -95,47 +95,60 @@ def amsmath_plugin(
     md.add_render_rule("amsmath", render_amsmath_block)
 
 
-def match_environment(string: str) -> None | tuple[str, str, int]:
-    match_open = RE_OPEN.match(string)
-    if not match_open:
-        return None
-    environment = match_open.group(1)
-    numbered = match_open.group(2)
-    match_close = re.search(
-        r"\\end\{" + environment + numbered.replace("*", r"\*") + "\\}", string
-    )
-    if not match_close:
-        return None
-    return (environment, numbered, match_close.end())
-
-
 def amsmath_block(
     state: StateBlock, startLine: int, endLine: int, silent: bool
 ) -> bool:
+    # note the code principally follows the logic in markdown_it/rules_block/fence.py,
+    # except that:
+    # (a) it allows for closing tag on same line as opening tag
+    # (b) it does not allow for opening tag without closing tag (i.e. no auto-closing)
+
     if is_code_block(state, startLine):
         return False
 
-    begin = state.bMarks[startLine] + state.tShift[startLine]
+    # does the first line contain the beginning of an amsmath environment
+    first_start = state.bMarks[startLine] + state.tShift[startLine]
+    first_end = state.eMarks[startLine]
+    first_text = state.src[first_start:first_end]
 
-    outcome = match_environment(state.src[begin:])
-    if not outcome:
+    if not (match_open := re.match(RE_OPEN, first_text)):
         return False
-    environment, numbered, endpos = outcome
-    endpos += begin
-
-    line = startLine
-    while line < endLine:
-        if endpos >= state.bMarks[line] and endpos <= state.eMarks[line]:
-            # line for end of block math found ...
-            state.line = line + 1
+
+    # construct the closing tag
+    environment = match_open.group(1)
+    numbered = match_open.group(2)
+    closing = rf"\end{{{match_open.group(1)}{match_open.group(2)}}}"
+
+    # start looking for the closing tag, including the current line
+    nextLine = startLine - 1
+
+    while True:
+        nextLine += 1
+        if nextLine >= endLine:
+            # reached the end of the block without finding the closing tag
+            return False
+
+        next_start = state.bMarks[nextLine] + state.tShift[nextLine]
+        next_end = state.eMarks[nextLine]
+        if next_start < first_end and state.sCount[nextLine] < state.blkIndent:
+            # non-empty line with negative indent should stop the list:
+            # - \begin{align}
+            #  test
+            return False
+
+        if state.src[next_start:next_end].rstrip().endswith(closing):
+            # found the closing tag
             break
-        line += 1
+
+    state.line = nextLine + 1
 
     if not silent:
         token = state.push("amsmath", "math", 0)
         token.block = True
-        token.content = state.src[begin:endpos]
+        token.content = state.getLines(
+            startLine, state.line, state.sCount[startLine], False
+        )
         token.meta = {"environment": environment, "numbered": numbered}
-        token.map = [startLine, line]
+        token.map = [startLine, nextLine]
 
     return True


Test patch:
diff --git a/tests/fixtures/amsmath.md b/tests/fixtures/amsmath.md
index ed83488..0bea6ab 100644
--- a/tests/fixtures/amsmath.md
+++ b/tests/fixtures/amsmath.md
@@ -11,6 +11,15 @@ a = 1
 </div>
 .
 
+equation environment on one line:
+.
+\begin{equation}a = 1\end{equation}
+.
+<div class="math amsmath">
+\begin{equation}a = 1\end{equation}
+</div>
+.
+
 equation* environment:
 .
 \begin{equation*}
@@ -181,13 +190,43 @@ equation environment, in list:
 <li>
 <div class="math amsmath">
 \begin{equation}
-  a = 1
-  \end{equation}
+a = 1
+\end{equation}
 </div>
 </li>
 </ul>
 .
 
+equation environment, in block quote:
+.
+> \begin{matrix}
+> -0.707 &  0.408 &  0.577 \\
+> -0.707 & -0.408 & -0.577 \\
+> -0.    & -0.816 &  0.577
+> \end{matrix}
+
+> \begin{equation}
+a = 1
+\end{equation}
+.
+<blockquote>
+<div class="math amsmath">
+\begin{matrix}
+-0.707 &amp;  0.408 &amp;  0.577 \\
+-0.707 &amp; -0.408 &amp; -0.577 \\
+-0.    &amp; -0.816 &amp;  0.577
+\end{matrix}
+</div>
+</blockquote>
+<blockquote>
+<div class="math amsmath">
+\begin{equation}
+a = 1
+\end{equation}
+</div>
+</blockquote>
+.
+
 `alignat` environment and HTML escaping
 .
 \begin{alignat}{3}
@@ -242,7 +281,7 @@ Indented by 4 spaces, DISABLE-CODEBLOCKS
 .
 <div class="math amsmath">
 \begin{equation}
-    a = 1
-    \end{equation}
+a = 1
+\end{equation}
 </div>
 .