Repository: zulip/zulip

Issue:
Allow more characters (`%`, `!`, `+`, `[`, `]`, `@`) in linkifier urls
https://chat.zulip.org/#narrow/stream/19-documentation/topic/linkifier/near/967173

We cannot use the % character in linkifier format strings currently. We should either document this or fix this.

Example: `https://trac.example.com/epic/%(num)s/hello%25world` is an invalid format string.


Discussion:
I also can't use a literal `@` character. This makes URLs with an `@` in them impossible to output from a linkifier, which basically makes it impossible to linkify URLs of the form `https://observablehq.com/@jrus/sinebar`.

While we are at it apparently @ can't be used in the match pattern either. Ideally I'd like to be able to make a linkifier for  
`@jrus/sinebar` → https://observablehq.com/@jrus/sinebar  

Currently trying to put a `%` or `@` into the URL format string results in `Failed: Invalid URL format string.`
Here is the spec: https://url.spec.whatwg.org

> The **URL code points** are ASCII alphanumeric, U+0021 (!), U+0024 ($), U+0026 (&), U+0027 ('), U+0028 LEFT PARENTHESIS, U+0029 RIGHT PARENTHESIS, U+002A (*), U+002B (+), U+002C (,), U+002D (-), U+002E (.), U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+003F (?), U+0040 (@), U+005F (_), U+007E (~), and code points in the range U+00A0 to U+10FFFD, inclusive, excluding surrogates and noncharacters.

> The **fragment percent-encode set** is the C0 control percent-encode set and U+0020 SPACE, U+0022 ("), U+003C (<), U+003E (>), and U+0060 (`).

> The **query percent-encode set** is the C0 control percent-encode set and U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>).

Whereas the current validator supports only: `[/a-zA-Z0-9#_?=&;~-]*`, which does not match the above.
https://github.com/zulip/zulip/blob/master/zerver/models.py#L751-L769

* * *

Perhaps `[/a-zA-Z0-9#_?=&;~-]*` could be changed to ``[-!$&'()*+,./:;=?@_~\w"#<>`%]*`` or the like.

Spaces should also probably be allowed in output URLs, but if necessary people could be forced to encode them as `%20`.
I've encountered the same issue with bang (`!`) characters, which are used in Icinga2 resource URLs. I don't understand why a second validation is needed when the URL submitted is already run through the standard `URLValidator`. Many valid characters are needlessly blocked...
Another requested character is `.`: https://chat.zulip.org/#narrow/stream/9-issues/topic/linkifier.20with.20period/near/1264973

(Though I agree with @jcharaoui that we probably shouldn't be applying a regex like this at all.)


Resolving patch:
diff --git a/zerver/lib/markdown/__init__.py b/zerver/lib/markdown/__init__.py
--- a/zerver/lib/markdown/__init__.py
+++ b/zerver/lib/markdown/__init__.py
@@ -1795,7 +1795,22 @@ def __init__(
         options.log_errors = False
 
         compiled_re2 = re2.compile(prepare_linkifier_pattern(source_pattern), options=options)
-        self.format_string = format_string
+
+        # Find percent-encoded bytes and escape them from the python
+        # interpolation.  That is:
+        #     %(foo)s -> %(foo)s
+        #     %%      -> %%
+        #     %ab     -> %%ab
+        #     %%ab    -> %%ab
+        #     %%%ab   -> %%%%ab
+        #
+        # We do this here, rather than before storing, to make edits
+        # to the underlying linkifier more straightforward, and
+        # because JS does not have a real formatter.
+        self.format_string = re.sub(
+            r"(?<!%)(%%)*%([a-fA-F0-9][a-fA-F0-9])", r"\1%%\2", format_string
+        )
+
         super().__init__(compiled_re2, md)
 
     def handleMatch(  # type: ignore[override] # supertype incompatible with supersupertype
diff --git a/zerver/migrations/0368_alter_realmfilter_url_format_string.py b/zerver/migrations/0368_alter_realmfilter_url_format_string.py
new file mode 100644
--- /dev/null
+++ b/zerver/migrations/0368_alter_realmfilter_url_format_string.py
@@ -0,0 +1,20 @@
+# Generated by Django 3.2.8 on 2021-10-20 23:42
+
+from django.db import migrations, models
+
+from zerver.models import filter_format_validator
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("zerver", "0367_scimclient"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="realmfilter",
+            name="url_format_string",
+            field=models.TextField(validators=[filter_format_validator]),
+        ),
+    ]
diff --git a/zerver/models.py b/zerver/models.py
--- a/zerver/models.py
+++ b/zerver/models.py
@@ -1120,10 +1120,36 @@ def filter_pattern_validator(value: str) -> Pattern[str]:
 
 
 def filter_format_validator(value: str) -> None:
-    regex = re.compile(r"^([\.\/:a-zA-Z0-9#_?=&;~-]+%\(([a-zA-Z0-9_-]+)\)s)+[/a-zA-Z0-9#_?=&;~-]*$")
+    """Verifies URL-ness, and then %(foo)s.
+
+    URLValidator is assumed to catch anything which is malformed as a
+    URL; the regex then verifies the format-string pieces.
+    """
+
+    URLValidator()(value)
+
+    regex = re.compile(
+        r"""
+            ^
+            (
+              [^%]                        # Any non-percent,
+            |                             #   OR...
+              % (                         # A %, which can mean:
+                  \( [a-zA-Z0-9_-]+ \) s  #   Interpolation group
+                |                         #     OR
+                  %                       #   %%, which is an escaped %
+                |                         #     OR
+                  [0-9a-fA-F][0-9a-fA-F]  #   URL percent-encoded bytes, which we
+                                          #   special-case in markdown translation
+                )
+            )+                            # Those happen one or more times
+            $
+        """,
+        re.VERBOSE,
+    )
 
     if not regex.match(value):
-        raise ValidationError(_("Invalid URL format string."))
+        raise ValidationError(_("Invalid format string in URL."))
 
 
 class RealmFilter(models.Model):
@@ -1134,7 +1160,7 @@ class RealmFilter(models.Model):
     id: int = models.AutoField(auto_created=True, primary_key=True, verbose_name="ID")
     realm: Realm = models.ForeignKey(Realm, on_delete=CASCADE)
     pattern: str = models.TextField()
-    url_format_string: str = models.TextField(validators=[URLValidator(), filter_format_validator])
+    url_format_string: str = models.TextField(validators=[filter_format_validator])
 
     class Meta:
         unique_together = ("realm", "pattern")


Test patch:
diff --git a/frontend_tests/puppeteer_tests/realm-linkifier.ts b/frontend_tests/puppeteer_tests/realm-linkifier.ts
--- a/frontend_tests/puppeteer_tests/realm-linkifier.ts
+++ b/frontend_tests/puppeteer_tests/realm-linkifier.ts
@@ -105,10 +105,7 @@ async function test_edit_invalid_linkifier(page: Page): Promise<void> {
         page,
         edit_linkifier_format_status_selector,
     );
-    assert.strictEqual(
-        edit_linkifier_format_status,
-        "Failed: Enter a valid URL.,Invalid URL format string.",
-    );
+    assert.strictEqual(edit_linkifier_format_status, "Failed: Enter a valid URL.");
 
     await page.click(".close-modal-btn");
     await page.waitForSelector("#dialog_widget_modal", {hidden: true});
diff --git a/zerver/tests/test_markdown.py b/zerver/tests/test_markdown.py
--- a/zerver/tests/test_markdown.py
+++ b/zerver/tests/test_markdown.py
@@ -1401,6 +1401,20 @@ def assert_conversion(content: str, should_have_converted: bool = True) -> None:
             ],
         )
 
+        # Test URI escaping
+        RealmFilter(
+            realm=realm,
+            pattern=r"url-(?P<id>[0-9]+)",
+            url_format_string="https://example.com/%%%ba/%(id)s",
+        ).save()
+        msg = Message(sender=self.example_user("hamlet"))
+        content = "url-123 is well-escaped"
+        converted = markdown_convert(content, message_realm=realm, message=msg)
+        self.assertEqual(
+            converted.rendered_content,
+            '<p><a href="https://example.com/%%ba/123">url-123</a> is well-escaped</p>',
+        )
+
     def test_multiple_matching_realm_patterns(self) -> None:
         realm = get_realm("zulip")
         url_format_string = r"https://trac.example.com/ticket/%(id)s"
diff --git a/zerver/tests/test_realm_linkifiers.py b/zerver/tests/test_realm_linkifiers.py
--- a/zerver/tests/test_realm_linkifiers.py
+++ b/zerver/tests/test_realm_linkifiers.py
@@ -1,7 +1,9 @@
 import re
 
+from django.core.exceptions import ValidationError
+
 from zerver.lib.test_classes import ZulipTestCase
-from zerver.models import RealmFilter
+from zerver.models import RealmFilter, filter_format_validator
 
 
 class RealmFilterTest(ZulipTestCase):
@@ -43,7 +45,9 @@ def test_create(self) -> None:
         data["pattern"] = r"ZUL-(?P<id>\d+)"
         data["url_format_string"] = "https://realm.com/my_realm_filter/"
         result = self.client_post("/json/realm/filters", info=data)
-        self.assert_json_error(result, "Invalid URL format string.")
+        self.assert_json_error(
+            result, "Group 'id' in linkifier pattern is not present in URL format string."
+        )
 
         data["url_format_string"] = "https://realm.com/my_realm_filter/#hashtag/%(id)s"
         result = self.client_post("/json/realm/filters", info=data)
@@ -117,13 +121,15 @@ def test_create(self) -> None:
             result, "Group 'id' in linkifier pattern is not present in URL format string."
         )
 
-        # BUG: In theory, this should be valid, since %% should be a
-        # valid escaping method. It's unlikely someone actually wants
-        # to do this, though.
-        data["pattern"] = r"ZUL-(?P<id>\d+)"
+        data["pattern"] = r"ZUL-ESCAPE-(?P<id>\d+)"
         data["url_format_string"] = r"https://realm.com/my_realm_filter/%%(ignored)s/%(id)s"
         result = self.client_post("/json/realm/filters", info=data)
-        self.assert_json_error(result, "Invalid URL format string.")
+        self.assert_json_success(result)
+
+        data["pattern"] = r"ZUL-URI-(?P<id>\d+)"
+        data["url_format_string"] = "https://example.com/%ba/%(id)s"
+        result = self.client_post("/json/realm/filters", info=data)
+        self.assert_json_success(result)
 
         data["pattern"] = r"(?P<org>[a-zA-Z0-9_-]+)/(?P<repo>[a-zA-Z0-9_-]+)#(?P<id>[0-9]+)"
         data["url_format_string"] = "https://github.com/%(org)s/%(repo)s/issue/%(id)s"
@@ -200,3 +206,46 @@ def test_update(self) -> None:
         data["url_format_string"] = "https://realm.com/my_realm_filter/%(id)s"
         result = self.client_patch(f"/json/realm/filters/{linkifier_id + 1}", info=data)
         self.assert_json_error(result, "Linkifier not found.")
+
+    def test_valid_urls(self) -> None:
+        valid_urls = [
+            "http://example.com/",
+            "https://example.com/",
+            "https://user:password@example.com/",
+            "https://example.com/@user/thing",
+            "https://example.com/!path",
+            "https://example.com/foo.bar",
+            "https://example.com/foo[bar]",
+            "https://example.com/%(foo)s",
+            "https://example.com/%(foo)s%(bars)s",
+            "https://example.com/%(foo)s/and/%(bar)s",
+            "https://example.com/?foo=%(foo)s",
+            "https://example.com/%ab",
+            "https://example.com/%ba",
+            "https://example.com/%21",
+            "https://example.com/words%20with%20spaces",
+            "https://example.com/back%20to%20%(back)s",
+            "https://example.com/encoded%2fwith%2fletters",
+            "https://example.com/encoded%2Fwith%2Fupper%2Fcase%2Fletters",
+            "https://example.com/%%",
+            "https://example.com/%%(",
+            "https://example.com/%%()",
+            "https://example.com/%%(foo",
+            "https://example.com/%%(foo)",
+            "https://example.com/%%(foo)s",
+        ]
+        for url in valid_urls:
+            filter_format_validator(url)
+
+        invalid_urls = [
+            "file:///etc/passwd",
+            "data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==",
+            "https://example.com/%(foo)",
+            "https://example.com/%()s",
+            "https://example.com/%4!",
+            "https://example.com/%(foo",
+            "https://example.com/%2(foo)s",
+        ]
+        for url in invalid_urls:
+            with self.assertRaises(ValidationError):
+                filter_format_validator(url)