Repository: django/django
Issue category: Performance Issue

Stop URL system checks from compiling regular expressions
Description
		
(last modified by Adam Johnson)		
Continuing my project to optimize the system checks, I found some good optimizations under django.core.checks.urls.check_url_config(), which showed up as quite expensive in profiling.
Looking down the call tree, it seems the most expensive part of this process is compiling the each URL pattern’s regular expression. This is unnecessary work though, as the checks only need *uncompiled* regular expression patterns. Using the compiled versions “undoes” the lazy-compile optimization that LocaleRegexDescriptor was created for in #27453 / 6e222dae5636f875c19ec66f730a4241abe33faa, at least for any process that runs checks.
The checks were fetching the uncompiled pattern with self.regex.pattern, which makse LocaleRegexDescriptor compile the pattern only to then read the uncompiled pattern from ​its pattern attribute.
Additionally, RoutePattern was calling _route_to_regex() twice to fetch its two result variables in different places: once in __init__() and again in _compile() (in the non-translated case). This function has non-trivial cost so avoiding double execution is worth it.
Before optimization stats:
check_url_config took 67ms, or ~10% of the time for checks.
LocaleRegexDescriptor.__get__() showed 965 calls taking ~60ms, ~9% of the total runtime of checks.
re.compile() showed 741 calls for 94ms.
_route_to_regex() had 1900 calls taking 18ms (~2.6% of the total runtime).
After optimization:
check_url_config() took 5ms, ~0.9% of the new total runtime.
The calls to LocaleRegexDescriptor.__get__ are gone.
re.compile() drops to 212 calls from other sites, for a total of 51ms.
_route_to_regex() drops to the expected 950 calls, taking half the time at 9ms.
(I also tried benchmarking with django-asv but got inconclusive results where change was within the error margins.)


Functions to edit:
django/urls/resolvers.py:LocaleRegexDescriptor.__init__
django/urls/resolvers.py:LocaleRegexDescriptor.__get__
django/urls/resolvers.py:CheckURLMixin._check_pattern_startswith_slash
django/urls/resolvers.py:RegexPattern
django/urls/resolvers.py:RegexPattern._check_include_trailing_dollar
django/urls/resolvers.py:RegexPattern._compile
django/urls/resolvers.py:_route_to_regex
django/urls/resolvers.py:RoutePattern.__init__
django/urls/resolvers.py:RoutePattern
django/urls/resolvers.py:RoutePattern._compile
Functions added: django/urls/resolvers.py:LocaleRegexDescriptor._compile
django/urls/resolvers.py:RegexPattern
django/urls/resolvers.py:LocaleRegexRouteDescriptor
django/urls/resolvers.py:LocaleRegexRouteDescriptor.__get__
django/urls/resolvers.py:RoutePattern

Patch:
diff --git a/django/urls/resolvers.py b/django/urls/resolvers.py
index 5f9941dd65e8..1b26aed8c112 100644
--- a/django/urls/resolvers.py
+++ b/django/urls/resolvers.py
@@ -128,9 +128,6 @@ def get_ns_resolver(ns_pattern, resolver, converters):
 
 
 class LocaleRegexDescriptor:
-    def __init__(self, attr):
-        self.attr = attr
-
     def __get__(self, instance, cls=None):
         """
         Return a compiled regular expression based on the active language.
@@ -140,15 +137,23 @@ def __get__(self, instance, cls=None):
         # As a performance optimization, if the given regex string is a regular
         # string (not a lazily-translated string proxy), compile it once and
         # avoid per-language compilation.
-        pattern = getattr(instance, self.attr)
+        pattern = instance._regex
         if isinstance(pattern, str):
-            instance.__dict__["regex"] = instance._compile(pattern)
+            instance.__dict__["regex"] = self._compile(pattern)
             return instance.__dict__["regex"]
         language_code = get_language()
         if language_code not in instance._regex_dict:
-            instance._regex_dict[language_code] = instance._compile(str(pattern))
+            instance._regex_dict[language_code] = self._compile(str(pattern))
         return instance._regex_dict[language_code]
 
+    def _compile(self, regex):
+        try:
+            return re.compile(regex)
+        except re.error as e:
+            raise ImproperlyConfigured(
+                f'"{regex}" is not a valid regular expression: {e}'
+            ) from e
+
 
 class CheckURLMixin:
     def describe(self):
@@ -164,12 +169,11 @@ def _check_pattern_startswith_slash(self):
         """
         Check that the pattern does not begin with a forward slash.
         """
-        regex_pattern = self.regex.pattern
         if not settings.APPEND_SLASH:
             # Skip check as it can be useful to start a URL pattern with a slash
             # when APPEND_SLASH=False.
             return []
-        if regex_pattern.startswith(("/", "^/", "^\\/")) and not regex_pattern.endswith(
+        if self._regex.startswith(("/", "^/", "^\\/")) and not self._regex.endswith(
             "/"
         ):
             warning = Warning(
@@ -186,7 +190,7 @@ def _check_pattern_startswith_slash(self):
 
 
 class RegexPattern(CheckURLMixin):
-    regex = LocaleRegexDescriptor("_regex")
+    regex = LocaleRegexDescriptor()
 
     def __init__(self, regex, name=None, is_endpoint=False):
         self._regex = regex
@@ -219,8 +223,7 @@ def check(self):
         return warnings
 
     def _check_include_trailing_dollar(self):
-        regex_pattern = self.regex.pattern
-        if regex_pattern.endswith("$") and not regex_pattern.endswith(r"\$"):
+        if self._regex.endswith("$") and not self._regex.endswith(r"\$"):
             return [
                 Warning(
                     "Your URL pattern {} uses include with a route ending with a '$'. "
@@ -232,15 +235,6 @@ def _check_include_trailing_dollar(self):
         else:
             return []
 
-    def _compile(self, regex):
-        """Compile and return the given regular expression."""
-        try:
-            return re.compile(regex)
-        except re.error as e:
-            raise ImproperlyConfigured(
-                '"%s" is not a valid regular expression: %s' % (regex, e)
-            ) from e
-
     def __str__(self):
         return str(self._regex)
 
@@ -250,7 +244,7 @@ def __str__(self):
 )
 
 
-def _route_to_regex(route, is_endpoint=False):
+def _route_to_regex(route, is_endpoint):
     """
     Convert a path pattern into a regular expression. Return the regular
     expression and a dictionary mapping the capture names to the converters.
@@ -296,15 +290,36 @@ def _route_to_regex(route, is_endpoint=False):
     return "".join(parts), converters
 
 
+class LocaleRegexRouteDescriptor:
+    def __get__(self, instance, cls=None):
+        """
+        Return a compiled regular expression based on the active language.
+        """
+        if instance is None:
+            return self
+        # As a performance optimization, if the given route is a regular string
+        # (not a lazily-translated string proxy), compile it once and avoid
+        # per-language compilation.
+        if isinstance(instance._route, str):
+            instance.__dict__["regex"] = re.compile(instance._regex)
+            return instance.__dict__["regex"]
+        language_code = get_language()
+        if language_code not in instance._regex_dict:
+            instance._regex_dict[language_code] = re.compile(
+                _route_to_regex(str(instance._route), instance._is_endpoint)[0]
+            )
+        return instance._regex_dict[language_code]
+
+
 class RoutePattern(CheckURLMixin):
-    regex = LocaleRegexDescriptor("_route")
+    regex = LocaleRegexRouteDescriptor()
 
     def __init__(self, route, name=None, is_endpoint=False):
         self._route = route
+        self._regex, self.converters = _route_to_regex(str(route), is_endpoint)
         self._regex_dict = {}
         self._is_endpoint = is_endpoint
         self.name = name
-        self.converters = _route_to_regex(str(route), is_endpoint)[1]
 
     def match(self, path):
         match = self.regex.search(path)
@@ -356,9 +371,6 @@ def _check_pattern_unmatched_angle_brackets(self):
             warnings.append(Warning(msg % (self.describe(), "<"), id="urls.W010"))
         return warnings
 
-    def _compile(self, route):
-        return re.compile(_route_to_regex(route, self._is_endpoint)[0])
-
     def __str__(self):
         return str(self._route)
 

Test patch: diff --git a/tests/i18n/patterns/locale/en/LC_MESSAGES/django.mo b/tests/i18n/patterns/locale/en/LC_MESSAGES/django.mo
index ec7644b504c3..b1f63b103106 100644
Binary files a/tests/i18n/patterns/locale/en/LC_MESSAGES/django.mo and b/tests/i18n/patterns/locale/en/LC_MESSAGES/django.mo differ
diff --git a/tests/i18n/patterns/locale/en/LC_MESSAGES/django.po b/tests/i18n/patterns/locale/en/LC_MESSAGES/django.po
index 9a14a80ceb8f..ac98eb5f08aa 100644
--- a/tests/i18n/patterns/locale/en/LC_MESSAGES/django.po
+++ b/tests/i18n/patterns/locale/en/LC_MESSAGES/django.po
@@ -7,31 +7,53 @@ msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2011-06-15 11:33+0200\n"
+"POT-Creation-Date: 2024-03-01 21:18+0000\n"
 "PO-Revision-Date: 2011-06-14 16:16+0100\n"
 "Last-Translator: Jannis Leidel <jannis@leidel.info>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
+"Language: \n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"Language: \n"
 
 #: urls/default.py:11
-msgid "^translated/$"
+#, fuzzy
+#| msgid "^translated/$"
+msgid "translated/"
 msgstr "^translated/$"
 
 #: urls/default.py:12
+#, fuzzy
+#| msgid "^translated/$"
+msgid "^translated-regex/$"
+msgstr "^translated/$"
+
+#: urls/default.py:14
 msgid "^translated/(?P<slug>[\\w-]+)/$"
 msgstr "^translated/(?P<slug>[\\w-]+)/$"
 
-#: urls/default.py:17
+#: urls/default.py:25
+msgid "^with-arguments/(?P<argument>[\\w-]+)/(?:(?P<optional>[\\w-]+).html)?$"
+msgstr ""
+
+#: urls/default.py:29
 msgid "^users/$"
 msgstr "^users/$"
 
-#: urls/default.py:18 urls/wrong.py:7
+#: urls/default.py:31 urls/wrong.py:7
 msgid "^account/"
 msgstr "^account/"
 
 #: urls/namespace.py:9 urls/wrong_namespace.py:10
 msgid "^register/$"
 msgstr "^register/$"
+
+#: urls/namespace.py:10
+msgid "^register-without-slash$"
+msgstr ""
+
+#: urls/namespace.py:11
+#, fuzzy
+#| msgid "^register/$"
+msgid "register-as-path/"
+msgstr "^register/$"
diff --git a/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.mo b/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.mo
index 5eac50466cb7..544bfdbfc664 100644
Binary files a/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.mo and b/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.mo differ
diff --git a/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.po b/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.po
index a938e3371dea..aa58c506d562 100644
--- a/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.po
+++ b/tests/i18n/patterns/locale/nl/LC_MESSAGES/django.po
@@ -7,29 +7,37 @@ msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2011-06-15 11:33+0200\n"
+"POT-Creation-Date: 2024-03-01 21:18+0000\n"
 "PO-Revision-Date: 2011-06-14 16:16+0100\n"
 "Last-Translator: Jannis Leidel <jannis@leidel.info>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
+"Language: \n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"Language: \n"
 "Plural-Forms: nplurals=2; plural=(n != 1)\n"
 
 #: urls/default.py:11
-msgid "^translated/$"
-msgstr "^vertaald/$"
+msgid "translated/"
+msgstr "vertaald/"
 
 #: urls/default.py:12
+msgid "^translated-regex/$"
+msgstr "^vertaald-regex/$"
+
+#: urls/default.py:14
 msgid "^translated/(?P<slug>[\\w-]+)/$"
 msgstr "^vertaald/(?P<slug>[\\w-]+)/$"
 
-#: urls/default.py:17
+#: urls/default.py:25
+msgid "^with-arguments/(?P<argument>[\\w-]+)/(?:(?P<optional>[\\w-]+).html)?$"
+msgstr ""
+
+#: urls/default.py:29
 msgid "^users/$"
 msgstr "^gebruikers/$"
 
-#: urls/default.py:18 urls/wrong.py:7
+#: urls/default.py:31 urls/wrong.py:7
 msgid "^account/"
 msgstr "^profiel/"
 
@@ -37,6 +45,10 @@ msgstr "^profiel/"
 msgid "^register/$"
 msgstr "^registreren/$"
 
-#: urls/namespace.py:12
+#: urls/namespace.py:10
+msgid "^register-without-slash$"
+msgstr ""
+
+#: urls/namespace.py:11
 msgid "register-as-path/"
 msgstr "registreren-als-pad/"
diff --git a/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.mo b/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.mo
index 1d7b346c278c..8e36cb206409 100644
Binary files a/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.mo and b/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.mo differ
diff --git a/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.po b/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.po
index fd3388e4b013..464d14bc1f75 100644
--- a/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.po
+++ b/tests/i18n/patterns/locale/pt_BR/LC_MESSAGES/django.po
@@ -7,32 +7,50 @@ msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2011-06-15 11:34+0200\n"
+"POT-Creation-Date: 2024-03-01 21:18+0000\n"
 "PO-Revision-Date: 2011-06-14 16:17+0100\n"
 "Last-Translator: Jannis Leidel <jannis@leidel.info>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
+"Language: \n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"Language: \n"
 "Plural-Forms: nplurals=2; plural=(n > 1)\n"
 
 #: urls/default.py:11
-msgid "^translated/$"
-msgstr "^traduzidos/$"
+msgid "translated/"
+msgstr "traduzidos/"
 
 #: urls/default.py:12
+msgid "^translated-regex/$"
+msgstr "^traduzidos-regex/$"
+
+#: urls/default.py:14
 msgid "^translated/(?P<slug>[\\w-]+)/$"
 msgstr "^traduzidos/(?P<slug>[\\w-]+)/$"
 
-#: urls/default.py:17
+#: urls/default.py:25
+msgid "^with-arguments/(?P<argument>[\\w-]+)/(?:(?P<optional>[\\w-]+).html)?$"
+msgstr ""
+
+#: urls/default.py:29
 msgid "^users/$"
 msgstr "^usuarios/$"
 
-#: urls/default.py:18 urls/wrong.py:7
+#: urls/default.py:31 urls/wrong.py:7
 msgid "^account/"
 msgstr "^conta/"
 
 #: urls/namespace.py:9 urls/wrong_namespace.py:10
 msgid "^register/$"
 msgstr "^registre-se/$"
+
+#: urls/namespace.py:10
+msgid "^register-without-slash$"
+msgstr ""
+
+#: urls/namespace.py:11
+#, fuzzy
+#| msgid "^register/$"
+msgid "register-as-path/"
+msgstr "^registre-se/$"
diff --git a/tests/i18n/patterns/tests.py b/tests/i18n/patterns/tests.py
index e2fee904b149..bd329e69f8e7 100644
--- a/tests/i18n/patterns/tests.py
+++ b/tests/i18n/patterns/tests.py
@@ -134,6 +134,9 @@ class URLTranslationTests(URLTestCaseBase):
     def test_no_prefix_translated(self):
         with translation.override("en"):
             self.assertEqual(reverse("no-prefix-translated"), "/translated/")
+            self.assertEqual(
+                reverse("no-prefix-translated-regex"), "/translated-regex/"
+            )
             self.assertEqual(
                 reverse("no-prefix-translated-slug", kwargs={"slug": "yeah"}),
                 "/translated/yeah/",
@@ -141,6 +144,7 @@ def test_no_prefix_translated(self):
 
         with translation.override("nl"):
             self.assertEqual(reverse("no-prefix-translated"), "/vertaald/")
+            self.assertEqual(reverse("no-prefix-translated-regex"), "/vertaald-regex/")
             self.assertEqual(
                 reverse("no-prefix-translated-slug", kwargs={"slug": "yeah"}),
                 "/vertaald/yeah/",
@@ -148,6 +152,9 @@ def test_no_prefix_translated(self):
 
         with translation.override("pt-br"):
             self.assertEqual(reverse("no-prefix-translated"), "/traduzidos/")
+            self.assertEqual(
+                reverse("no-prefix-translated-regex"), "/traduzidos-regex/"
+            )
             self.assertEqual(
                 reverse("no-prefix-translated-slug", kwargs={"slug": "yeah"}),
                 "/traduzidos/yeah/",
@@ -180,7 +187,7 @@ def test_translate_url_utility(self):
                 "/nl/profiel/registreren-als-pad/",
             )
             self.assertEqual(translation.get_language(), "en")
-            # URL with parameters.
+            # re_path() URL with parameters.
             self.assertEqual(
                 translate_url("/en/with-arguments/regular-argument/", "nl"),
                 "/nl/with-arguments/regular-argument/",
@@ -191,6 +198,11 @@ def test_translate_url_utility(self):
                 ),
                 "/nl/with-arguments/regular-argument/optional.html",
             )
+            # path() URL with parameter.
+            self.assertEqual(
+                translate_url("/en/path-with-arguments/regular-argument/", "nl"),
+                "/nl/path-with-arguments/regular-argument/",
+            )
 
         with translation.override("nl"):
             self.assertEqual(translate_url("/nl/gebruikers/", "en"), "/en/users/")
diff --git a/tests/i18n/patterns/urls/default.py b/tests/i18n/patterns/urls/default.py
index 418e9f568568..090b92eeca66 100644
--- a/tests/i18n/patterns/urls/default.py
+++ b/tests/i18n/patterns/urls/default.py
@@ -8,7 +8,8 @@
 urlpatterns = [
     path("not-prefixed/", view, name="not-prefixed"),
     path("not-prefixed-include/", include("i18n.patterns.urls.included")),
-    re_path(_(r"^translated/$"), view, name="no-prefix-translated"),
+    path(_("translated/"), view, name="no-prefix-translated"),
+    re_path(_(r"^translated-regex/$"), view, name="no-prefix-translated-regex"),
     re_path(
         _(r"^translated/(?P<slug>[\w-]+)/$"),
         view,
@@ -25,6 +26,11 @@
         view,
         name="with-arguments",
     ),
+    path(
+        _("path-with-arguments/<str:argument>/"),
+        view,
+        name="path-with-arguments",
+    ),
     re_path(_(r"^users/$"), view, name="users"),
     re_path(
         _(r"^account/"), include("i18n.patterns.urls.namespace", namespace="account")