Repository: zulip/zulip

Issue:
Linkifier validation -- non-existant group name breaks rendering 
With the following pattern and replacement:
`#(?P<id>[0-9]+)` / `https://github.com/zulip/zulip/issues/%(bogus)s`

...sending messages with `#123` results in:
```
2020-10-06 19:14:23.691 ERR  [] Exception in Markdown parser; input (sanitized) was: '#xxx'
 (message unknown)
Traceback (most recent call last):
  File "/srv/zulip/zerver/lib/markdown/__init__.py", line 2344, in do_convert
    rendered_content = timeout(5, _md_engine.convert, content)
  File "/srv/zulip/zerver/lib/timeout.py", line 90, in timeout
    raise thread.exc_info[1].with_traceback(thread.exc_info[2])
  File "/srv/zulip/zerver/lib/timeout.py", line 50, in run
    self.result = func(*args, **kwargs)
  File "/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py", line 267, in convert
    newRoot = treeprocessor.run(root)
  File "/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/treeprocessors.py", line 370, in run
    self.__handleInline(text), child
  File "/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/treeprocessors.py", line 131, in __handleInline
    self.inlinePatterns[patternIndex], data, patternIndex, startIndex
  File "/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/treeprocessors.py", line 285, in __applyPattern
    node = pattern.handleMatch(match)
  File "/srv/zulip/zerver/lib/markdown/__init__.py", line 1634, in handleMatch
    self.format_string % m.groupdict(),
KeyError: 'bogus'
```

We should at very least not fail to send the message -- especially since there are already such in existance.  But we should also prevent the creation of the linkifier in the first place, so people aren't confused why they silently aren't linking.


Discussion:
@aero31aero can you help investigate this?  

I think #13398 might be what we need to validate this more correctly.  I think most correctly we should just block creating such an invalid linkifier, and purge any that have been already created.
@timabbott yes, the first commit of #13398 fixes this. I'll test and amend the commit message to close this issue and rebase the PR while I'm at it.
[This is also possible in topic names](https://sentry.io/share/issue/655e20a2bd4f467cafa4f9c0dd17ad00/).  The effects of this are worse if it's in a topic name, as it breaks for more than just a single message.
Let's see if we can get #13398 integrated.
@timabbott can I work on this issue ? Or is this dependant on the PR #13398.
I think ideally we should make a PR that extracts the first commit of #13398 as a separate PR addressing the comments related to that commit; that would allow us to close out this, since #13398 as a whole has a decent number of unresolved comments.

Resolving patch:
diff --git a/zerver/migrations/0330_linkifier_pattern_validator.py b/zerver/migrations/0330_linkifier_pattern_validator.py
new file mode 100644
--- /dev/null
+++ b/zerver/migrations/0330_linkifier_pattern_validator.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.1.8 on 2021-04-18 15:36
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("zerver", "0329_remove_realm_allow_community_topic_editing"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="realmfilter",
+            name="pattern",
+            field=models.TextField(),
+        ),
+    ]
diff --git a/zerver/models.py b/zerver/models.py
--- a/zerver/models.py
+++ b/zerver/models.py
@@ -11,6 +11,7 @@
     Dict,
     List,
     Optional,
+    Pattern,
     Sequence,
     Set,
     Tuple,
@@ -918,7 +919,7 @@ def flush_realm_emoji(sender: Any, **kwargs: Any) -> None:
 post_delete.connect(flush_realm_emoji, sender=RealmEmoji)
 
 
-def filter_pattern_validator(value: str) -> None:
+def filter_pattern_validator(value: str) -> Pattern[str]:
     regex = re.compile(r"^(?:(?:[\w\-#_= /:]*|[+]|[!])(\(\?P<\w+>.+\)))+$")
     error_msg = _("Invalid linkifier pattern.  Valid characters are {}.").format(
         "[ a-zA-Z_#=/:+!-]",
@@ -928,11 +929,13 @@ def filter_pattern_validator(value: str) -> None:
         raise ValidationError(error_msg)
 
     try:
-        re.compile(value)
+        pattern = re.compile(value)
     except re.error:
         # Regex is invalid
         raise ValidationError(error_msg)
 
+    return pattern
+
 
 def filter_format_validator(value: str) -> None:
     regex = re.compile(r"^([\.\/:a-zA-Z0-9#_?=&;~-]+%\(([a-zA-Z0-9_-]+)\)s)+[/a-zA-Z0-9#_?=&;~-]*$")
@@ -948,12 +951,54 @@ class RealmFilter(models.Model):
 
     id: int = models.AutoField(auto_created=True, primary_key=True, verbose_name="ID")
     realm: Realm = models.ForeignKey(Realm, on_delete=CASCADE)
-    pattern: str = models.TextField(validators=[filter_pattern_validator])
+    pattern: str = models.TextField()
     url_format_string: str = models.TextField(validators=[URLValidator(), filter_format_validator])
 
     class Meta:
         unique_together = ("realm", "pattern")
 
+    def clean(self) -> None:
+        """Validate whether the set of parameters in the URL Format string
+        match the set of parameters in the regular expression.
+
+        Django's `full_clean` calls `clean_fields` followed by `clean` method
+        and stores all ValidationErrors from all stages to return as JSON.
+        """
+
+        # Extract variables present in the pattern
+        pattern = filter_pattern_validator(self.pattern)
+        group_set = set(pattern.groupindex.keys())
+
+        # Extract variables used in the URL format string.  Note that
+        # this regex will incorrectly reject patterns that attempt to
+        # escape % using %%.
+        found_group_set: Set[str] = set()
+        group_match_regex = r"%\((?P<group_name>[^()]+)\)s"
+        for m in re.finditer(group_match_regex, self.url_format_string):
+            group_name = m.group("group_name")
+            found_group_set.add(group_name)
+
+        # Report patterns missing in linkifier pattern.
+        missing_in_pattern_set = found_group_set - group_set
+        if len(missing_in_pattern_set) > 0:
+            name = list(missing_in_pattern_set)[0]
+            raise ValidationError(
+                _("Group %(name)r in URL format string is not present in linkifier pattern."),
+                params={"name": name},
+            )
+
+        missing_in_url_set = group_set - found_group_set
+        # Report patterns missing in URL format string.
+        if len(missing_in_url_set) > 0:
+            # We just report the first missing pattern here. Users can
+            # incrementally resolve errors if there are multiple
+            # missing patterns.
+            name = list(missing_in_url_set)[0]
+            raise ValidationError(
+                _("Group %(name)r in linkifier pattern is not present in URL format string."),
+                params={"name": name},
+            )
+
     def __str__(self) -> str:
         return f"<RealmFilter({self.realm.string_id}): {self.pattern} {self.url_format_string}>"
 


Test patch:
diff --git a/frontend_tests/puppeteer_tests/realm-linkifier.ts b/frontend_tests/puppeteer_tests/realm-linkifier.ts
--- a/frontend_tests/puppeteer_tests/realm-linkifier.ts
+++ b/frontend_tests/puppeteer_tests/realm-linkifier.ts
@@ -47,9 +47,9 @@ async function test_add_invalid_linkifier_pattern(page: Page): Promise<void> {
     });
     await page.click("form.admin-linkifier-form button.button");
 
-    await page.waitForSelector("div#admin-linkifier-pattern-status", {visible: true});
+    await page.waitForSelector("div#admin-linkifier-status", {visible: true});
     assert.strictEqual(
-        await common.get_text_from_selector(page, "div#admin-linkifier-pattern-status"),
+        await common.get_text_from_selector(page, "div#admin-linkifier-status"),
         "Failed: Invalid linkifier pattern.  Valid characters are [ a-zA-Z_#=/:+!-].",
     );
 }
@@ -88,7 +88,7 @@ async function test_edit_invalid_linkifier(page: Page): Promise<void> {
     });
     await page.click(".submit-linkifier-info-change");
 
-    const edit_linkifier_pattern_status_selector = "div#edit-linkifier-pattern-status";
+    const edit_linkifier_pattern_status_selector = "div#edit-linkifier-status";
     await page.waitForSelector(edit_linkifier_pattern_status_selector, {visible: true});
     const edit_linkifier_pattern_status = await common.get_text_from_selector(
         page,
diff --git a/zerver/tests/test_realm_linkifiers.py b/zerver/tests/test_realm_linkifiers.py
--- a/zerver/tests/test_realm_linkifiers.py
+++ b/zerver/tests/test_realm_linkifiers.py
@@ -94,6 +94,41 @@ def test_create(self) -> None:
         self.assert_json_success(result)
         self.assertIsNotNone(re.match(data["pattern"], "!123"))
 
+        # This block of tests is for mismatches between field sets
+        data["pattern"] = r"ZUL-(?P<id>\d+)"
+        data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_error(
+            result, "Group 'hello' in URL format string is not present in linkifier pattern."
+        )
+
+        data["pattern"] = r"ZUL-(?P<id>\d+)-(?P<hello>\d+)"
+        data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_error(
+            result, "Group 'id' in linkifier pattern is not present in URL format string."
+        )
+
+        data["pattern"] = r"ZULZ-(?P<hello>\d+)-(?P<world>\d+)"
+        data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s/%(world)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_success(result)
+
+        data["pattern"] = r"ZUL-(?P<id>\d+)-(?P<hello>\d+)-(?P<world>\d+)"
+        data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_error(
+            result, "Group 'id' in linkifier pattern is not present in URL format string."
+        )
+
+        # BUG: In theory, this should be valid, since %% should be a
+        # valid escaping method. It's unlikely someone actually wants
+        # to do this, though.
+        data["pattern"] = r"ZUL-(?P<id>\d+)"
+        data["url_format_string"] = r"https://realm.com/my_realm_filter/%%(ignored)s/%(id)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_error(result, "Invalid URL format string.")
+
         data["pattern"] = r"(?P<org>[a-zA-Z0-9_-]+)/(?P<repo>[a-zA-Z0-9_-]+)#(?P<id>[0-9]+)"
         data["url_format_string"] = "https://github.com/%(org)s/%(repo)s/issue/%(id)s"
         result = self.client_post("/json/realm/filters", info=data)