Repository: zulip/zulip

Issue:
linkifiers: Fix Markdown processor handling of topics that should have multiple linkifier topic_links.
As reported in https://chat.zulip.org/#narrow/stream/9-issues/topic/Multiple.20topic.20links.3F/near/1348089, it appears that we've regressed support for computing multiple topic_links generated by linkifiers in the server. Here's a test case:

```
diff --git a/zerver/tests/test_markdown.py b/zerver/tests/test_markdown.py
index 7b945f3187..4ab4e7a750 100644
--- a/zerver/tests/test_markdown.py
+++ b/zerver/tests/test_markdown.py
@@ -1344,6 +1344,16 @@ class MarkdownTest(ZulipTestCase):
             ],
         )

+        # BUG: This should render topic_links for both issue IDs.
+        msg.set_topic_name("Two issues: #444 #555")
+        converted_topic = topic_links(realm.id, msg.topic_name())
+        self.assertEqual(
+            converted_topic,
+            [
+                {"url": "https://trac.example.com/ticket/444", "text": "#444"},
+            ],
+        )
+
         RealmFilter(
             realm=realm,
             pattern=r"#(?P<id>[a-zA-Z]+-[0-9]+)",
```


Discussion:
The prepared regex (by `prepare_linkifier_pattern`) matches `" #444  "` with spaces on both sides, after which it cannot also match `" #555"` because the space has already been used.
So yes, this is a regression from db934be0646e3002287c72ba2154a904a4dc42b5
Without using any sort of look-around, both regexes need to mach that in-betweeen space, in order to know that the pattern is well-bounded.  re2 doesn't support `\G`, which would be a way to work around the second regex trying to match it.

We can try using `\b` at the end, which is effectively a cheating look-ahead?

Resolving patch:
diff --git a/zerver/lib/markdown/__init__.py b/zerver/lib/markdown/__init__.py
--- a/zerver/lib/markdown/__init__.py
+++ b/zerver/lib/markdown/__init__.py
@@ -2365,9 +2365,21 @@ def topic_links(linkifiers_key: int, topic_name: str) -> List[Dict[str, str]]:
             # here on an invalid regex would spam the logs with every
             # message sent; simply move on.
             continue
-        for m in pattern.finditer(topic_name):
+        pos = 0
+        while pos < len(topic_name):
+            m = pattern.search(topic_name, pos)
+            if m is None:
+                break
+
             match_details = m.groupdict()
             match_text = match_details[OUTER_CAPTURE_GROUP]
+
+            # Adjust the start point of the match for the next
+            # iteration -- we rewind the non-word character at the
+            # end, if there was one, so a potential next match can
+            # also use it.
+            pos = m.end() - len(match_details[AFTER_CAPTURE_GROUP])
+
             # We format the linkifier's url string using the matched text.
             # Also, we include the matched text in the response, so that our clients
             # don't have to implement any logic of their own to get back the text.


Test patch:
diff --git a/zerver/tests/test_markdown.py b/zerver/tests/test_markdown.py
--- a/zerver/tests/test_markdown.py
+++ b/zerver/tests/test_markdown.py
@@ -1322,13 +1322,13 @@ def test_realm_patterns(self) -> None:
 
         flush_per_request_caches()
 
-        content = "We should fix #224 and #115, but not issue#124 or #1124z or [trac #15](https://trac.example.com/ticket/16) today."
+        content = "We should fix #224 #336 #446 and #115, but not issue#124 or #1124z or [trac #15](https://trac.example.com/ticket/16) today."
         converted = markdown_convert(content, message_realm=realm, message=msg)
         converted_topic = topic_links(realm.id, msg.topic_name())
 
         self.assertEqual(
             converted.rendered_content,
-            '<p>We should fix <a href="https://trac.example.com/ticket/224">#224</a> and <a href="https://trac.example.com/ticket/115">#115</a>, but not issue#124 or #1124z or <a href="https://trac.example.com/ticket/16">trac #15</a> today.</p>',
+            '<p>We should fix <a href="https://trac.example.com/ticket/224">#224</a> <a href="https://trac.example.com/ticket/336">#336</a> <a href="https://trac.example.com/ticket/446">#446</a> and <a href="https://trac.example.com/ticket/115">#115</a>, but not issue#124 or #1124z or <a href="https://trac.example.com/ticket/16">trac #15</a> today.</p>',
         )
         self.assertEqual(
             converted_topic, [{"url": "https://trac.example.com/ticket/444", "text": "#444"}]
@@ -1344,6 +1344,17 @@ def test_realm_patterns(self) -> None:
             ],
         )
 
+        msg.set_topic_name("#444 #555 #666")
+        converted_topic = topic_links(realm.id, msg.topic_name())
+        self.assertEqual(
+            converted_topic,
+            [
+                {"url": "https://trac.example.com/ticket/444", "text": "#444"},
+                {"url": "https://trac.example.com/ticket/555", "text": "#555"},
+                {"url": "https://trac.example.com/ticket/666", "text": "#666"},
+            ],
+        )
+
         RealmFilter(
             realm=realm,
             pattern=r"#(?P<id>[a-zA-Z]+-[0-9]+)",
@@ -1474,7 +1485,7 @@ def test_multiple_matching_realm_patterns(self) -> None:
             converted.rendered_content,
             '<p>We should fix <a href="https://trac.example.com/ticket/ABC-123">ABC-123</a> or <a href="https://trac.example.com/ticket/16">trac ABC-123</a> today.</p>',
         )
-        # Both the links should be generated in topics.
+        # But both the links should be generated in topics.
         self.assertEqual(
             converted_topic,
             [