Repository: pulldown-cmark/pulldown-cmark (rust)

Issue:
[regression in 0.10] Consecutive footnote refs fail to parse correctly
Consider the following markdown:

```
Lorem ipsum.[^a][^b]

[^a]: Foo
[^b]: Bar
```

Which on GitHub, renders thus:

Lorem ipsum.[^a][^b]

[^a]: Foo
[^b]: Bar

With pulldown-cmark **v0.9.6**, these were correctly parsed as two consecutive footnote references:

```
➜  cargo run -- --events --enable-footnotes < in.md
   Compiling pulldown-cmark v0.9.6 (/home/zoni/Workspace/pulldown-cmark)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.61s
     Running `/home/zoni/Workspace/pulldown-cmark/target/debug/pulldown-cmark --events --enable-footnotes`
0..21: Start(Paragraph)
0..12: Text(Borrowed("Lorem ipsum."))
12..16: FootnoteReference(Borrowed("a"))         // <--- footnote ref a
16..20: FootnoteReference(Borrowed("b"))         // <--- footnote ref b
0..21: End(Paragraph)
22..32: Start(FootnoteDefinition(Borrowed("a")))
28..31: Start(Paragraph)
28..31: Text(Borrowed("Foo"))
31..32: SoftBreak
28..31: End(Paragraph)
22..32: End(FootnoteDefinition(Borrowed("a")))
32..42: Start(FootnoteDefinition(Borrowed("b")))
38..42: Start(Paragraph)
38..41: Text(Borrowed("Bar"))
38..42: End(Paragraph)
32..42: End(FootnoteDefinition(Borrowed("b")))
EOF
```

With pulldown-cmark **0.11** (`v0.11.0-44-g4e9bc2c`),  this fails to resolve both footnote references correctly. Notably, it fails on both the **old**:

```
➜  cargo run -- --events --enable-old-footnotes < in.md
   Compiling pulldown-cmark v0.11.0 (/home/zoni/Workspace/pulldown-cmark/pulldown-cmark)
   Compiling pulldown-cmark-escape v0.11.0 (/home/zoni/Workspace/pulldown-cmark/pulldown-cmark-escape)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.63s
     Running `/home/zoni/Workspace/pulldown-cmark/target/debug/pulldown-cmark --events --enable-old-footnotes`
0..21: Start(Paragraph)
0..12: Text(Borrowed("Lorem ipsum."))
12..13: Text(Borrowed("["))                    // <---- footnote ref a incorrectly parsed as text
13..15: Text(Borrowed("^a"))
15..16: Text(Borrowed("]"))
16..20: FootnoteReference(Borrowed("b"))       // <---- footnote ref b
0..21: End(Paragraph)
22..42: Start(FootnoteDefinition(Borrowed("a")))
28..42: Start(Paragraph)
28..31: Text(Borrowed("Foo"))
31..32: SoftBreak
32..36: FootnoteReference(Borrowed("b"))
36..41: Text(Borrowed(": Bar"))
28..42: End(Paragraph)
22..42: End(FootnoteDefinition)
EOF
```

As well as the **new** (GFM) footnote parser:

```
➜  cargo run -- --events --enable-footnotes < in.md
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running `/home/zoni/Workspace/pulldown-cmark/target/debug/pulldown-cmark --events --enable-footnotes`
0..21: Start(Paragraph)
0..12: Text(Borrowed("Lorem ipsum."))
12..13: Text(Borrowed("["))                    // <---- footnote ref a incorrectly parsed as text
13..15: Text(Borrowed("^a"))
15..16: Text(Borrowed("]"))
16..20: FootnoteReference(Borrowed("b"))       // <---- footnote ref b
0..21: End(Paragraph)
22..32: Start(FootnoteDefinition(Borrowed("a")))
28..32: Start(Paragraph)
28..31: Text(Borrowed("Foo"))
28..32: End(Paragraph)
22..32: End(FootnoteDefinition)
32..42: Start(FootnoteDefinition(Borrowed("b")))
38..42: Start(Paragraph)
38..41: Text(Borrowed("Bar"))
38..42: End(Paragraph)
32..42: End(FootnoteDefinition)
EOF
```

(Putting a space between `[^a]` and `[^b]` will produce correct events. The problem exists when two or more footnotes are squished up together. Then, only the last one is parsed correctly)

Pull request description:
Fix confusing bug with back-to-back footnotes
Fixes #927
Follows up on #773

Patch:
diff --git a/pulldown-cmark/specs/footnotes.txt b/pulldown-cmark/specs/footnotes.txt
index dd341d9..df48269 100644
--- a/pulldown-cmark/specs/footnotes.txt
+++ b/pulldown-cmark/specs/footnotes.txt
@@ -396,11 +396,8 @@ Test case for the relationship between link references and footnotes.
 
 GitHub writes
 `My [cmark-gfm]<sup class="footnote-reference"><a href="#examples">1</a></sup>.`
-in the first paragraph. This seems like a bug, but [markdown-it] and [commonmark-hs]
-both do it, too, so it's probably not worth it to be different here.
-
-[markdown-it]: https://markdown-it.github.io/#md3=%7B%22source%22%3A%22%5B%5Efoo%5D%5B%5D%5Cn%5Cn%5B%5Efoo%5D%5Bbaz%5D%5Cn%5Cn%5Bbaz%5D%5B%5Efoo%5D%5Cn%5Cn%5B%5Efoo%5D%3A%20bar%5Cn%5Cn%5Bbaz%5D%3A%20https%3A%2F%2Frust-lang.org%22%2C%22defaults%22%3A%7B%22html%22%3Afalse%2C%22xhtmlOut%22%3Afalse%2C%22breaks%22%3Afalse%2C%22langPrefix%22%3A%22language-%22%2C%22linkify%22%3Atrue%2C%22typographer%22%3Atrue%2C%22_highlight%22%3Atrue%2C%22_strict%22%3Afalse%2C%22_view%22%3A%22html%22%7D%7D
-[commonmark-hs]: https://pandoc.org/try/?params=%7B%22text%22%3A%22%5B%5Efoo%5D%5B%5D%5Cn%5Cn%5B%5Efoo%5D%5Bbaz%5D%5Cn%5Cn%5Bbaz%5D%5B%5Efoo%5D%5Cn%5Cn%5B%5Efoo%5D%3A+bar%5Cn%5Cn%5Bbaz%5D%3A+https%3A%2F%2Frust-lang.org%22%2C%22to%22%3A%22html5%22%2C%22from%22%3A%22commonmark_x%22%2C%22standalone%22%3Afalse%2C%22embed-resources%22%3Afalse%2C%22table-of-contents%22%3Afalse%2C%22number-sections%22%3Afalse%2C%22citeproc%22%3Afalse%2C%22html-math-method%22%3A%22plain%22%2C%22wrap%22%3A%22auto%22%2C%22highlight-style%22%3Anull%2C%22files%22%3A%7B%7D%2C%22template%22%3Anull%7D
+in the first paragraph. This seems like a bug, and it's easier to fix it than
+to remain compatible with it <https://github.com/pulldown-cmark/pulldown-cmark/pull/929>.
 
 ```````````````````````````````` example
 My [cmark-gfm][^c].
@@ -423,7 +420,7 @@ My [otherlink[^c]].
 
 [otherlink[^c]]: https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702
 .
-<p>My [cmark-gfm]<sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
+<p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a> <sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
@@ -646,3 +643,37 @@ footnote [^quux]</p>
 <p>x</p>
 </div>
 ````````````````````````````````
+
+Multiple footnotes should be able without needing whitespace in between.
+
+```````````````````````````````` example
+Lorem ipsum.[^a][^b]
+
+[^a]: Foo
+[^b]: Bar
+.
+<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup><sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+````````````````````````````````
+
+This does work already:
+
+```````````````````````````````` example
+Lorem ipsum.[^a] [^b]
+
+[^a]: Foo
+[^b]: Bar
+.
+<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup> <sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+````````````````````````````````
diff --git a/pulldown-cmark/specs/old_footnotes.txt b/pulldown-cmark/specs/old_footnotes.txt
index 22aded4..fc3b2f7 100644
--- a/pulldown-cmark/specs/old_footnotes.txt
+++ b/pulldown-cmark/specs/old_footnotes.txt
@@ -161,3 +161,37 @@ Second check for footnotes:
 .
 <p>[Reference to footnotes A<sup class="footnote-reference"><a href="#1">1</a></sup>, B<sup class="footnote-reference"><a href="#2">2</a></sup> and C<sup class="footnote-reference"><a href="#3">3</a></sup>.</p><div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup><p>Footnote A.</p></div><div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup><p>Footnote B.</p></div><div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup><p>Footnote C.</p></div>
 ````````````````````````````````
+
+Multiple footnotes should be able without needing whitespace in between.
+
+```````````````````````````````` example
+Lorem ipsum.[^a][^b]
+
+[^a]: Foo
+[^b]: Bar
+.
+<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup><sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+````````````````````````````````
+
+This does work already:
+
+```````````````````````````````` example
+Lorem ipsum.[^a] [^b]
+
+[^a]: Foo
+[^b]: Bar
+.
+<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup> <sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+````````````````````````````````
diff --git a/pulldown-cmark/specs/regression.txt b/pulldown-cmark/specs/regression.txt
index 2101bb0..0a2bf3d 100644
--- a/pulldown-cmark/specs/regression.txt
+++ b/pulldown-cmark/specs/regression.txt
@@ -1681,7 +1681,7 @@ ISSUE 772
 .
 <p><sup class="footnote-reference"><a href="#foo">1</a></sup>[]</p>
 <p><a href="https://rust-lang.org">^foo</a></p>
-<p>[baz]<sup class="footnote-reference"><a href="#foo">1</a></sup></p>
+<p><a href="https://rust-lang.org">baz</a><sup class="footnote-reference"><a href="#foo">1</a></sup></p>
 <div class="footnote-definition" id="foo"><sup class="footnote-definition-label">1</sup>
 <p>bar</p>
 </div>
diff --git a/pulldown-cmark/src/parse.rs b/pulldown-cmark/src/parse.rs
index 063b2ef..7705bdc 100644
--- a/pulldown-cmark/src/parse.rs
+++ b/pulldown-cmark/src/parse.rs
@@ -660,13 +660,12 @@ impl<'input, F: BrokenLinkCallback<'input>> Parser<'input, F> {
                                 // [shortcut]
                                 //
                                 // [shortcut]: /blah
-                                RefScan::Failed => {
+                                RefScan::Failed | RefScan::UnexpectedFootnote => {
                                     if !could_be_ref {
                                         continue;
                                     }
                                     (next, LinkType::Shortcut)
                                 }
-                                RefScan::UnexpectedFootnote => continue,
                             };
 
                             // FIXME: references and labels are mixed in the naming of variables
@@ -677,7 +676,7 @@ impl<'input, F: BrokenLinkCallback<'input>> Parser<'input, F> {
                                 RefScan::LinkLabel(l, end_ix) => {
                                     Some((ReferenceLabel::Link(l), end_ix))
                                 }
-                                RefScan::Collapsed(..) | RefScan::Failed => {
+                                RefScan::Collapsed(..) | RefScan::Failed | RefScan::UnexpectedFootnote => {
                                     // No label? maybe it is a shortcut reference
                                     let label_start = self.tree[tos.node].item.end - 1;
                                     let label_end = self.tree[cur_ix].item.end;
@@ -690,7 +689,6 @@ impl<'input, F: BrokenLinkCallback<'input>> Parser<'input, F> {
                                     .map(|(ix, label)| (label, label_start + ix))
                                     .filter(|(_, end)| *end == label_end)
                                 }
-                                RefScan::UnexpectedFootnote => continue,
                             };
 
                             let id = match &label {


Test patch:
diff --git a/pulldown-cmark/tests/suite/footnotes.rs b/pulldown-cmark/tests/suite/footnotes.rs
index ab79433..5d4e906 100644
--- a/pulldown-cmark/tests/suite/footnotes.rs
+++ b/pulldown-cmark/tests/suite/footnotes.rs
@@ -418,7 +418,7 @@ My [otherlink[^c]].
 
 [otherlink[^c]]: https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702
 "##;
-    let expected = r##"<p>My [cmark-gfm]<sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
+    let expected = r##"<p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a><sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
 <p>My <a href="https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702">cmark-gfm</a> <sup class="footnote-reference"><a href="#c">1</a></sup>.</p>
@@ -646,3 +646,41 @@ footnote [^quux]</p>
 
     test_markdown_html(original, expected, false, false, false);
 }
+
+#[test]
+fn footnotes_test_25() {
+    let original = r##"Lorem ipsum.[^a][^b]
+
+[^a]: Foo
+[^b]: Bar
+"##;
+    let expected = r##"<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup><sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+"##;
+
+    test_markdown_html(original, expected, false, false, false);
+}
+
+#[test]
+fn footnotes_test_26() {
+    let original = r##"Lorem ipsum.[^a] [^b]
+
+[^a]: Foo
+[^b]: Bar
+"##;
+    let expected = r##"<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup> <sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+"##;
+
+    test_markdown_html(original, expected, false, false, false);
+}
diff --git a/pulldown-cmark/tests/suite/old_footnotes.rs b/pulldown-cmark/tests/suite/old_footnotes.rs
index 9924f9f..80ec6ef 100644
--- a/pulldown-cmark/tests/suite/old_footnotes.rs
+++ b/pulldown-cmark/tests/suite/old_footnotes.rs
@@ -175,3 +175,41 @@ fn old_footnotes_test_9() {
 
     test_markdown_html(original, expected, false, false, false);
 }
+
+#[test]
+fn old_footnotes_test_10() {
+    let original = r##"Lorem ipsum.[^a][^b]
+
+[^a]: Foo
+[^b]: Bar
+"##;
+    let expected = r##"<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup><sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+"##;
+
+    test_markdown_html(original, expected, false, false, false);
+}
+
+#[test]
+fn old_footnotes_test_11() {
+    let original = r##"Lorem ipsum.[^a] [^b]
+
+[^a]: Foo
+[^b]: Bar
+"##;
+    let expected = r##"<p>Lorem ipsum.<sup class="footnote-reference"><a href="#a">1</a></sup> <sup class="footnote-reference"><a href="#b">2</a></sup></p>
+<div class="footnote-definition" id="a"><sup class="footnote-definition-label">1</sup>
+<p>Foo</p>
+</div>
+<div class="footnote-definition" id="b"><sup class="footnote-definition-label">2</sup>
+<p>Bar</p>
+</div>
+"##;
+
+    test_markdown_html(original, expected, false, false, false);
+}
diff --git a/pulldown-cmark/tests/suite/regression.rs b/pulldown-cmark/tests/suite/regression.rs
index 954bc83..3123631 100644
--- a/pulldown-cmark/tests/suite/regression.rs
+++ b/pulldown-cmark/tests/suite/regression.rs
@@ -1966,7 +1966,7 @@ fn regression_test_124() {
 "##;
     let expected = r##"<p><sup class="footnote-reference"><a href="#foo">1</a></sup>[]</p>
 <p><a href="https://rust-lang.org">^foo</a></p>
-<p>[baz]<sup class="footnote-reference"><a href="#foo">1</a></sup></p>
+<p><a href="https://rust-lang.org">baz</a><sup class="footnote-reference"><a href="#foo">1</a></sup></p>
 <div class="footnote-definition" id="foo"><sup class="footnote-definition-label">1</sup>
 <p>bar</p>
 </div>