Repository: wagtail/wagtail

Issue:
Rich text external-to-internal link converter not working when using a non-root path for `wagtail_serve`
### Issue Summary

Entering an Internal link in the External Link for Rich Text Editor isn't converting to an internal link.

### Steps to Reproduce

1. in settings.py enable Link Conversion by setting
-  WAGTAILADMIN_EXTERNAL_LINK_CONVERSION = "confirm"  
        or
-  WAGTAILADMIN_EXTERNAL_LINK_CONVERSION = "all"
2. Open a page in wagtail to edit
3. Add a paragraph tag, link some text using the link builder
![image](https://github.com/wagtail/wagtail/assets/23039742/09f50e2e-ad22-4728-ab93-eca4d5a1f43d)

I have tried a relative link and a fully qualified URL in my local, feature, and production environments, but the link isn't converting.

- /the-ascent/credit-cards/best-hotel-credit-cards/

**To get this functionality to work locally,** 
I had to hack the **.venv/lib/python3.11/site-packages/wagtail/admin/views/chooser.py**  file to remove the first instance of things in the components when doing a page match. 

``` python

                    components = [
                        component for component in url.split("/") if component
                    ]
                    if "localhost" in settings.WAGTAILADMIN_BASE_URL:
                        components = components[1:]
                    route = Site.objects.get(pk=pk).root_page.specific.route(
                        request,
                        components,
                    )
```

if the setting is ="confirm", 
I'd expect to get the popup:
![image](https://github.com/wagtail/wagtail/assets/23039742/96150939-b5cb-4aaa-9c82-dfc8f0a847a6)


If it's "all, I'd expect the next time I'd edit the link it would be automatically set to "Internal Link" and the page that it's currently set to as the highlighted option. **NOTE this is not what is highlighted**, its parent page or the parent page of the page I'm linking from is what is highlighted:
![image](https://github.com/wagtail/wagtail/assets/23039742/9d348b30-f129-4379-892d-927e7f225307)


-   I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: (no)

### Technical details

-   Python version:3.11.1
-   Django version: 5.0.3
-   Wagtail version: 6.1.2
-   Browser version: Chrome on Mac 125.0.6422.113



Discussion:
squash-labs[bot]: <h2>Manage this branch in <a href="https://app.squash.io/accounts/">Squash</a></h2>Test this branch here: <a href='https://laymonagefix-external-link-con-7tbvh.squash.io' target='_blank'>https://laymonagefix-external-link-con-7tbvh.squash.io</a><br/>

Patch:
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index 8329e4b57b..ce2f1dac80 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -13,6 +13,7 @@ Changelog
  * Skip `ChooseParentView` if only one possible valid parent page availale (Matthias Brück)
  * Add `copy_for_translation_done` signal when a page is copied for translation (Arnar Tumi Þorsteinsson)
  * Remove reduced opacity for draft page title in listings (Inju Michorius)
+ * Adopt more compact representation for StreamField definitions in migrations (Matt Westcott)
  * Fix: Make `WAGTAILIMAGES_CHOOSER_PAGE_SIZE` setting functional again (Rohit Sharma)
  * Fix: Enable `richtext` template tag to convert lazy translation values (Benjamin Bach)
  * Fix: Ensure permission labels on group permissions page are translated where available (Matt Westcott)
@@ -24,6 +25,9 @@ Changelog
  * Fix: Allow renditions of `.ico` images (Julie Rymer)
  * Fix: Handle choice groups as dictionaries in active filters (Sébastien Corbin)
  * Fix: Add separators when displaying multiple error messages on a StructBlock (Kyle Bayliss)
+ * Fix: Specify `verbose_name` on `TranslatableMixin.locale` so that it is translated when used as a label (Romein van Buren)
+ * Fix: Disallow null characters in API filter values (Jochen Wersdörfer)
+ * Fix: Fix image preview when Willow optimizers are enabled (Alex Tomkins)
  * Docs: Remove duplicate section on frontend caching proxies from performance page (Jake Howard)
  * Docs: Document `restriction_type` field on PageViewRestriction (Shlomo Markowitz)
  * Docs: Document Wagtail's bug bounty policy (Jake Howard)
@@ -376,6 +380,13 @@ Changelog
  * Maintenance: Remove support for Django 4.1 and below (Sage Abdullah)
 
 
+5.2.6 (xx.xx.xxxx) - IN DEVELOPMENT
+~~~~~~~~~~~~~~~~~~
+
+ * Fix: Fix image preview when Willow optimizers are enabled (Alex Tomkins)
+ * Maintenance: Remove django-pattern-library upper bound in testing dependencies (Sage Abdullah)
+
+
 5.2.5 (01.05.2024)
 ~~~~~~~~~~~~~~~~~~
 
diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
index 04b90111e9..b5566ecc54 100644
--- a/CONTRIBUTORS.md
+++ b/CONTRIBUTORS.md
@@ -824,6 +824,7 @@
 * Roel Koper
 * Inju Michorius
 * Jeroen de Vries
+* Romein van Buren
 
 ## Translators
 
diff --git a/docs/reference/settings.md b/docs/reference/settings.md
index b0c2fea429..c0a7671cd6 100644
--- a/docs/reference/settings.md
+++ b/docs/reference/settings.md
@@ -205,6 +205,8 @@ WAGTAILADMIN_EXTERNAL_LINK_CONVERSION = 'exact'
 
 Customize Wagtail's behavior when an internal page url is entered in the external link chooser. Possible values for this setting are `'all'`, `'exact'`, `'confirm`, or `''`. The default, `'all'`, means that Wagtail will automatically convert submitted urls that exactly match page urls to the corresponding internal links. If the url is an inexact match - for example, the submitted url has query parameters - then Wagtail will confirm the conversion with the user. `'exact'` means that any inexact matches will be left as external urls, and the confirmation step will be skipped. `'confirm'` means that every link conversion will be confirmed with the user, even if the match is exact. `''` means that Wagtail will not attempt to convert any urls entered to internal page links.
 
+If the url is relative, Wagtail will not convert the link if there are more than one {class}`~wagtail.models.Site` instances. This is to avoid accidentally matching coincidentally named pages on different sites.
+
 (wagtail_date_time_formats)=
 
 ### `WAGTAIL_DATE_FORMAT`, `WAGTAIL_DATETIME_FORMAT`, `WAGTAIL_TIME_FORMAT`
diff --git a/docs/releases/5.2.6.md b/docs/releases/5.2.6.md
new file mode 100644
index 0000000000..d47a0dd4de
--- /dev/null
+++ b/docs/releases/5.2.6.md
@@ -0,0 +1,21 @@
+# Wagtail 5.2.6 release notes - IN DEVELOPMENT
+
+_Unreleased_
+
+```{contents}
+---
+local:
+depth: 1
+---
+```
+
+## What's new
+
+### Bug fixes
+
+ * Fix image preview when Willow optimizers are enabled (Alex Tomkins)
+
+
+### Maintenance
+
+ * Remove django-pattern-library upper bound in testing dependencies (Sage Abdullah)
diff --git a/docs/releases/6.2.md b/docs/releases/6.2.md
index 82c1455ff6..67579056ab 100644
--- a/docs/releases/6.2.md
+++ b/docs/releases/6.2.md
@@ -23,6 +23,7 @@ depth: 1
  * Skip `ChooseParentView` if only one possible valid parent page availale (Matthias Brück)
  * Add `copy_for_translation_done` signal when a page is copied for translation (Arnar Tumi Þorsteinsson)
  * Remove reduced opacity for draft page title in listings (Inju Michorius)
+ * Adopt more compact representation for StreamField definitions in migrations (Matt Westcott)
 
 
 ### Bug fixes
@@ -38,6 +39,9 @@ depth: 1
  * Allow renditions of `.ico` images (Julie Rymer)
  * Fix the rendering of grouped choices when using ChoiceFilter in combination with choices (Sébastien Corbin)
  * Add separators when displaying multiple error messages on a StructBlock (Kyle Bayliss)
+ * Specify `verbose_name` on `TranslatableMixin.locale` so that it is translated when used as a label (Romein van Buren)
+ * Disallow null characters in API filter values (Jochen Wersdörfer)
+ * Fix image preview when Willow optimizers are enabled (Alex Tomkins)
 
 
 ### Documentation
diff --git a/docs/releases/index.rst b/docs/releases/index.rst
index cbee02acf3..415d8c13f6 100644
--- a/docs/releases/index.rst
+++ b/docs/releases/index.rst
@@ -15,6 +15,7 @@ Release notes
    6.0.2
    6.0.1
    6.0
+   5.2.6
    5.2.5
    5.2.4
    5.2.3
diff --git a/wagtail/admin/views/chooser.py b/wagtail/admin/views/chooser.py
index 920c361127..8ec9dde8b9 100644
--- a/wagtail/admin/views/chooser.py
+++ b/wagtail/admin/views/chooser.py
@@ -1,4 +1,5 @@
 import re
+from collections import defaultdict
 from urllib.parse import parse_qs, quote, urlencode, urlsplit
 
 from django.conf import settings
@@ -6,6 +7,7 @@ from django.core.paginator import InvalidPage, Paginator
 from django.http import Http404
 from django.shortcuts import get_object_or_404
 from django.template.response import TemplateResponse
+from django.urls import NoReverseMatch
 from django.urls.base import reverse
 from django.utils.translation import gettext_lazy as _
 from django.views.generic.base import View
@@ -629,29 +631,57 @@ class ExternalLinkView(BaseLinkFormView):
             if sites is None:
                 sites = Site.get_site_root_paths()
 
+            try:
+                # The serve view might not be routed to the root path of the domain,
+                # e.g. /pages/, so we need to account for the path to the serve view
+                serve_path = reverse("wagtail_serve", args=("",))
+            except NoReverseMatch:
+                serve_path = None
+
             match_relative_paths = submitted_url.startswith("/") and len(sites) == 1
             # We should only match relative urls if there's only a single site
             # Otherwise this could get very annoying accidentally matching coincidentally
             # named pages on different sites
 
+            possible_sites = defaultdict(list)
+
             if match_relative_paths:
-                possible_sites = [
-                    (pk, url_without_query) for pk, path, url, language_code in sites
-                ]
+                for pk, path, url, language_code in sites:
+                    possible_sites[pk].append(url_without_query)
+
+                    # If the submitted URL is prefixed with the serve path,
+                    # also consider it without the serve path so we can match
+                    # the page using Page.route()
+                    if serve_path and url_without_query.startswith(serve_path):
+                        possible_sites[pk].append(
+                            url_without_query[len(serve_path) - 1 :]
+                        )
             else:
-                possible_sites = [
-                    (pk, url_without_query[len(url) :])
-                    for pk, path, url, language_code in sites
-                    if submitted_url.startswith(url)
-                ]
+                for pk, path, url, language_code in sites:
+                    if not submitted_url.startswith(url):
+                        continue
+                    possible_sites[pk].append(url_without_query[len(url) :])
+
+                    # If the submitted URL is prefixed with the serve path,
+                    # also consider it without the serve path so we can match
+                    # the page using Page.route()
+                    if serve_path and url_without_query.startswith(url + serve_path):
+                        possible_sites[pk].append(
+                            url_without_query[len(url) + len(serve_path) - 1 :]
+                        )
 
             # Loop over possible sites to identify a page match
-            for pk, url in possible_sites:
-                try:
-                    route = Site.objects.get(pk=pk).root_page.specific.route(
-                        request,
-                        [component for component in url.split("/") if component],
-                    )
+            for pk, possible_urls in possible_sites.items():
+                site = Site.objects.select_related("root_page").get(pk=pk)
+                root_page = site.root_page.specific
+                for url in possible_urls:
+                    try:
+                        route = root_page.route(
+                            request,
+                            [component for component in url.split("/") if component],
+                        )
+                    except Http404:
+                        continue
 
                     matched_page = route.page.specific
 
@@ -704,9 +734,6 @@ class ExternalLinkView(BaseLinkFormView):
                             },
                         )
 
-                except Http404:
-                    continue
-
             # Otherwise, with no internal matches, fall back to an external url
             return self.render_chosen_response(result)
         else:  # form invalid
diff --git a/wagtail/api/v2/filters.py b/wagtail/api/v2/filters.py
index 4038a6727a..e9b54e17c3 100644
--- a/wagtail/api/v2/filters.py
+++ b/wagtail/api/v2/filters.py
@@ -46,6 +46,12 @@ class FieldsFilter(BaseFilterBackend):
                         % (value, field_name, str(e))
                     )
 
+                if "\x00" in str(value):
+                    raise BadRequestError(
+                        "field filter error. null characters are not allowed for %s"
+                        % field_name
+                    )
+
                 if isinstance(field, TaggableManager):
                     for tag in value.split(","):
                         queryset = queryset.filter(**{field_name + "__name": tag})
diff --git a/wagtail/blocks/base.py b/wagtail/blocks/base.py
index 984349243c..0ec04799f8 100644
--- a/wagtail/blocks/base.py
+++ b/wagtail/blocks/base.py
@@ -97,6 +97,17 @@ class Block(metaclass=BaseBlock):
 
         self.label = self.meta.label or ""
 
+    @classmethod
+    def construct_from_lookup(cls, lookup, *args, **kwargs):
+        """
+        See `wagtail.blocks.definition_lookup.BlockDefinitionLookup`.
+        Construct a block instance from the provided arguments, using the given BlockDefinitionLookup
+        object to perform any necessary lookups.
+        """
+        # In the base implementation, no lookups take place - args / kwargs are passed
+        # on to the constructor as-is
+        return cls(*args, **kwargs)
+
     def set_name(self, name):
         self.name = name
         if not self.meta.label:
@@ -376,7 +387,11 @@ class Block(metaclass=BaseBlock):
         """
         return False
 
-    def deconstruct(self):
+    @cached_property
+    def canonical_module_path(self):
+        """
+        Return the module path string that should be used to refer to this block in migrations.
+        """
         # adapted from django.utils.deconstruct.deconstructible
         module_name = self.__module__
         name = self.__class__.__name__
@@ -394,16 +409,29 @@ class Block(metaclass=BaseBlock):
         # if the module defines a DECONSTRUCT_ALIASES dictionary, see if the class has an entry in there;
         # if so, use that instead of the real path
         try:
-            path = module.DECONSTRUCT_ALIASES[self.__class__]
+            return module.DECONSTRUCT_ALIASES[self.__class__]
         except (AttributeError, KeyError):
-            path = f"{module_name}.{name}"
+            return f"{module_name}.{name}"
 
+    def deconstruct(self):
         return (
-            path,
+            self.canonical_module_path,
             self._constructor_args[0],
             self._constructor_args[1],
         )
 
+    def deconstruct_with_lookup(self, lookup):
+        """
+        Like `deconstruct`, but with a `wagtail.blocks.definition_lookup.BlockDefinitionLookupBuilder`
+        object available so that any block instances within the definition can be added to the lookup
+        table to obtain an ID (potentially shared with other matching block definitions, thus reducing
+        the overall definition size) to be used in place of the block. The resulting deconstructed form
+        returned here can then be restored into a block object using `Block.construct_from_lookup`.
+        """
+        # In the base implementation, no substitutions happen, so we ignore the lookup and just call
+        # deconstruct
+        return self.deconstruct()
+
     def __eq__(self, other):
         """
         Implement equality on block objects so that two blocks with matching definitions are considered
@@ -411,14 +439,9 @@ class Block(metaclass=BaseBlock):
         attributes identified in MUTABLE_META_ATTRIBUTES, so checking these along with the result of
         deconstruct (which captures the constructor arguments) is sufficient to identify (valid) differences.
 
-        This was originally necessary as a workaround for https://code.djangoproject.com/ticket/24340
-        in Django <1.9; the deep_deconstruct function used to detect changes for migrations did not
-        recurse into the block lists, and left them as Block instances. This __eq__ method therefore
-        came into play when identifying changes within migrations.
-
-        As of Django >=1.9, this *probably* isn't required any more. However, it may be useful in
-        future as a way of identifying blocks that can be re-used within StreamField definitions
-        (https://github.com/wagtail/wagtail/issues/4298#issuecomment-367656028).
+        This was implemented as a workaround for a Django <1.9 bug and is quite possibly not used by Wagtail
+        any more, but has been retained as it provides a sensible definition of equality (and there's no
+        reason to break it).
         """
 
         if not isinstance(other, Block):
diff --git a/wagtail/blocks/definition_lookup.py b/wagtail/blocks/definition_lookup.py
new file mode 100644
index 0000000000..bb65cea614
--- /dev/null
+++ b/wagtail/blocks/definition_lookup.py
@@ -0,0 +1,85 @@
+from collections import defaultdict
+from importlib import import_module
+
+
+class BlockDefinitionLookup:
+    """
+    A utility for constructing StreamField Block objects in migrations, starting from
+    a compact representation that avoids repeating the same definition whenever a
+    block is re-used in multiple places over the block definition tree.
+
+    The underlying data is a dict of block definitions, such as:
+    ```
+    {
+        0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+        1: ("wagtail.blocks.RichTextBlock", [], {}),
+        2: ("wagtail.blocks.StreamBlock", [
+            [
+                ("heading", 0),
+                ("paragraph", 1),
+            ],
+        ], {}),
+    }
+    ```
+
+    where each definition is a tuple of (module_path, args, kwargs) similar to that
+    returned by `deconstruct` - with the difference that any block objects appearing
+    in args / kwargs may be substituted with an index into the lookup table that
+    points to that block's definition. Any block class that wants to support such
+    substitutions should implement a static/class method
+    `construct_from_lookup(lookup, *args, **kwargs)`, where `lookup` is
+    the `BlockDefinitionLookup` instance. The method should return a block instance
+    constructed from the provided arguments (after performing any lookups).
+    """
+
+    def __init__(self, blocks):
+        self.blocks = blocks
+        self.block_classes = {}
+
+    def get_block(self, index):
+        path, args, kwargs = self.blocks[index]
+        try:
+            cls = self.block_classes[path]
+        except KeyError:
+            module_name, class_name = path.rsplit(".", 1)
+            module = import_module(module_name)
+            cls = self.block_classes[path] = getattr(module, class_name)
+
+        return cls.construct_from_lookup(self, *args, **kwargs)
+
+
+class BlockDefinitionLookupBuilder:
+    """
+    Helper for constructing the lookup data used by BlockDefinitionLookup
+    """
+
+    def __init__(self):
+        self.blocks = []
+
+        # Lookup table mapping the deconstructed tuple forms of blocks (as obtained from
+        # `block.deconstruct_with_lookup`) to their index in the `blocks` list. These
+        # tuples can be compared for equality, but not hashed, so we cannot use them as
+        # dict keys; instead, we index them on the first tuple element (the module path)
+        # and maintain a list of (index, deconstructed_tuple) pairs for each one.
+        self.block_indexes_by_type = defaultdict(list)
+
+    def add_block(self, block):
+        """
+        Add a block to the lookup table, returning an index that can be used to refer to it
+        """
+        deconstructed = block.deconstruct_with_lookup(self)
+
+        # Check if we've already seen this block definition
+        block_indexes = self.block_indexes_by_type[deconstructed[0]]
+        for index, existing_deconstructed in block_indexes:
+            if existing_deconstructed == deconstructed:
+                return index
+
+        # If not, add it to the lookup table
+        index = len(self.blocks)
+        self.blocks.append(deconstructed)
+        block_indexes.append((index, deconstructed))
+        return index
+
+    def get_lookup_as_dict(self):
+        return dict(enumerate(self.blocks))
diff --git a/wagtail/blocks/list_block.py b/wagtail/blocks/list_block.py
index 5801adae86..cb73f79ddb 100644
--- a/wagtail/blocks/list_block.py
+++ b/wagtail/blocks/list_block.py
@@ -151,6 +151,12 @@ class ListBlock(Block):
             # Default to a list consisting of one empty (i.e. default-valued) child item
             self.meta.default = [self.child_block.get_default()]
 
+    @classmethod
+    def construct_from_lookup(cls, lookup, child_block, **kwargs):
+        if isinstance(child_block, int):
+            child_block = lookup.get_block(child_block)
+        return cls(child_block, **kwargs)
+
     def value_from_datadict(self, data, files, prefix):
         count = int(data["%s-count" % prefix])
         child_blocks_with_indexes = []
@@ -396,6 +402,12 @@ class ListBlock(Block):
         errors.extend(self.child_block.check(**kwargs))
         return errors
 
+    def deconstruct_with_lookup(self, lookup):
+        path, args, kwargs = super().deconstruct_with_lookup(lookup)
+        if isinstance(args[0], Block):
+            args = (lookup.add_block(args[0]),)
+        return path, args, kwargs
+
     class Meta:
         # No icon specified here, because that depends on the purpose that the
         # block is being used for. Feel encouraged to specify an icon in your
diff --git a/wagtail/blocks/stream_block.py b/wagtail/blocks/stream_block.py
index 4a603d64da..860712b583 100644
--- a/wagtail/blocks/stream_block.py
+++ b/wagtail/blocks/stream_block.py
@@ -89,6 +89,14 @@ class BaseStreamBlock(Block):
                 block.set_name(name)
                 self.child_blocks[name] = block
 
+    @classmethod
+    def construct_from_lookup(cls, lookup, child_blocks, **kwargs):
+        if child_blocks:
+            child_blocks = [
+                (name, lookup.get_block(index)) for name, index in child_blocks
+            ]
+        return cls(child_blocks, **kwargs)
+
     def empty_value(self, raw_text=None):
         return StreamValue(self, [], raw_text=raw_text)
 
@@ -433,6 +441,17 @@ class BaseStreamBlock(Block):
         kwargs = self._constructor_kwargs
         return (path, args, kwargs)
 
+    def deconstruct_with_lookup(self, lookup):
+        path = "wagtail.blocks.StreamBlock"
+        args = [
+            [
+                (name, lookup.add_block(block))
+                for name, block in self.child_blocks.items()
+            ]
+        ]
+        kwargs = self._constructor_kwargs
+        return (path, args, kwargs)
+
     def check(self, **kwargs):
         errors = super().check(**kwargs)
         for name, child_block in self.child_blocks.items():
diff --git a/wagtail/blocks/struct_block.py b/wagtail/blocks/struct_block.py
index 8d88a3b6c6..2ff7962f02 100644
--- a/wagtail/blocks/struct_block.py
+++ b/wagtail/blocks/struct_block.py
@@ -119,6 +119,14 @@ class BaseStructBlock(Block):
                 block.set_name(name)
                 self.child_blocks[name] = block
 
+    @classmethod
+    def construct_from_lookup(cls, lookup, child_blocks, **kwargs):
+        if child_blocks:
+            child_blocks = [
+                (name, lookup.get_block(index)) for name, index in child_blocks
+            ]
+        return cls(child_blocks, **kwargs)
+
     def get_default(self):
         """
         Any default value passed in the constructor or self.meta is going to be a dict
@@ -312,6 +320,17 @@ class BaseStructBlock(Block):
         kwargs = self._constructor_kwargs
         return (path, args, kwargs)
 
+    def deconstruct_with_lookup(self, lookup):
+        path = "wagtail.blocks.StructBlock"
+        args = [
+            [
+                (name, lookup.add_block(block))
+                for name, block in self.child_blocks.items()
+            ]
+        ]
+        kwargs = self._constructor_kwargs
+        return (path, args, kwargs)
+
     def check(self, **kwargs):
         errors = super().check(**kwargs)
         for name, child_block in self.child_blocks.items():
diff --git a/wagtail/contrib/typed_table_block/blocks.py b/wagtail/contrib/typed_table_block/blocks.py
index d8d1a5ad35..7304e1d9d6 100644
--- a/wagtail/contrib/typed_table_block/blocks.py
+++ b/wagtail/contrib/typed_table_block/blocks.py
@@ -87,6 +87,14 @@ class BaseTypedTableBlock(Block):
                 block.set_name(name)
                 self.child_blocks[name] = block
 
+    @classmethod
+    def construct_from_lookup(cls, lookup, child_blocks, **kwargs):
+        if child_blocks:
+            child_blocks = [
+                (name, lookup.get_block(index)) for name, index in child_blocks
+            ]
+        return cls(child_blocks, **kwargs)
+
     def value_from_datadict(self, data, files, prefix):
         caption = data["%s-caption" % prefix]
 
@@ -259,6 +267,17 @@ class BaseTypedTableBlock(Block):
         kwargs = self._constructor_kwargs
         return (path, args, kwargs)
 
+    def deconstruct_with_lookup(self, lookup):
+        path = "wagtail.contrib.typed_table_block.blocks.TypedTableBlock"
+        args = [
+            [
+                (name, lookup.add_block(block))
+                for name, block in self.child_blocks.items()
+            ]
+        ]
+        kwargs = self._constructor_kwargs
+        return (path, args, kwargs)
+
     def check(self, **kwargs):
         errors = super().check(**kwargs)
         for name, child_block in self.child_blocks.items():
diff --git a/wagtail/fields.py b/wagtail/fields.py
index 915f4144e9..7d2f60bb40 100644
--- a/wagtail/fields.py
+++ b/wagtail/fields.py
@@ -5,8 +5,13 @@ from django.core.validators import MaxLengthValidator
 from django.db import models
 from django.db.models.fields.json import KeyTransform
 from django.utils.encoding import force_str
+from django.utils.functional import cached_property
 
 from wagtail.blocks import Block, BlockField, StreamBlock, StreamValue
+from wagtail.blocks.definition_lookup import (
+    BlockDefinitionLookup,
+    BlockDefinitionLookupBuilder,
+)
 from wagtail.rich_text import (
     RichTextMaxLengthValidator,
     extract_references_from_rich_text,
@@ -82,34 +87,68 @@ class Creator:
 
 
 class StreamField(models.Field):
-    def __init__(self, block_types, use_json_field=True, **kwargs):
-        # use_json_field no longer has any effect but is recognised to support historical
-        # migrations
+    def __init__(self, block_types, use_json_field=True, block_lookup=None, **kwargs):
+        """
+        Construct a StreamField.
+
+        :param block_types: Either a list of block types that are allowed in this StreamField
+            (as a list of tuples of block name and block instance) or a StreamBlock to use as
+            the top level block (as a block instance or class).
+        :param use_json_field: Ignored, but retained for compatibility with historical migrations.
+        :param block_lookup: Used in migrations to provide a more compact block definition -
+            see ``wagtail.blocks.definition_lookup.BlockDefinitionLookup``. If passed, ``block_types``
+            can contain integer indexes into this lookup table, in place of actual block instances.
+        """
 
         # extract kwargs that are to be passed on to the block, not handled by super
-        block_opts = {}
+        self.block_opts = {}
         for arg in ["min_num", "max_num", "block_counts", "collapsed"]:
             if arg in kwargs:
-                block_opts[arg] = kwargs.pop(arg)
+                self.block_opts[arg] = kwargs.pop(arg)
 
         # for a top-level block, the 'blank' kwarg (defaulting to False) always overrides the
         # block's own 'required' meta attribute, even if not passed explicitly; this ensures
         # that the field and block have consistent definitions
-        block_opts["required"] = not kwargs.get("blank", False)
+        self.block_opts["required"] = not kwargs.get("blank", False)
+
+        # Store the `block_types` and `block_lookup` arguments to be handled in the `stream_block`
+        # property
+        self.block_types_arg = block_types
+        self.block_lookup = block_lookup
 
         super().__init__(**kwargs)
 
-        if isinstance(block_types, Block):
+    @cached_property
+    def stream_block(self):
+        has_block_lookup = self.block_lookup is not None
+        if has_block_lookup:
+            lookup = BlockDefinitionLookup(self.block_lookup)
+
+        if isinstance(self.block_types_arg, Block):
             # use the passed block as the top-level block
-            self.stream_block = block_types
-        elif isinstance(block_types, type):
+            block = self.block_types_arg
+        elif isinstance(self.block_types_arg, int) and has_block_lookup:
+            # retrieve block from lookup table to use as the top-level block
+            block = lookup.get_block(self.block_types_arg)
+        elif isinstance(self.block_types_arg, type):
             # block passed as a class - instantiate it
-            self.stream_block = block_types()
+            block = self.block_types_arg()
         else:
-            # construct a top-level StreamBlock from the list of block types
-            self.stream_block = StreamBlock(block_types)
+            # construct a top-level StreamBlock from the list of block types.
+            # If an integer is found in place of a block instance, and block_lookup is
+            # provided, it will be replaced with the corresponding block definition.
+            child_blocks = []
+
+            for name, child_block in self.block_types_arg:
+                if isinstance(child_block, int) and has_block_lookup:
+                    child_blocks.append((name, lookup.get_block(child_block)))
+                else:
+                    child_blocks.append((name, child_block))
+
+            block = StreamBlock(child_blocks)
 
-        self.stream_block.set_meta_options(block_opts)
+        block.set_meta_options(self.block_opts)
+        return block
 
     @property
     def json_field(self):
@@ -126,8 +165,13 @@ class StreamField(models.Field):
 
     def deconstruct(self):
         name, path, _, kwargs = super().deconstruct()
-        block_types = list(self.stream_block.child_blocks.items())
+        lookup = BlockDefinitionLookupBuilder()
+        block_types = [
+            (name, lookup.add_block(block))
+            for name, block in self.stream_block.child_blocks.items()
+        ]
         args = [block_types]
+        kwargs["block_lookup"] = lookup.get_lookup_as_dict()
         return name, path, args, kwargs
 
     def to_python(self, value):
diff --git a/wagtail/images/views/images.py b/wagtail/images/views/images.py
index 1187df09c2..c818a847d4 100644
--- a/wagtail/images/views/images.py
+++ b/wagtail/images/views/images.py
@@ -1,8 +1,9 @@
 import os
+from tempfile import SpooledTemporaryFile
 
 from django.conf import settings
 from django.core.exceptions import PermissionDenied
-from django.http import HttpResponse, JsonResponse
+from django.http import FileResponse, HttpResponse, JsonResponse
 from django.shortcuts import get_object_or_404, redirect
 from django.template.response import TemplateResponse
 from django.urls import reverse
@@ -291,8 +292,11 @@ def preview(request, image_id, filter_spec):
     image = get_object_or_404(get_image_model(), id=image_id)
 
     try:
-        response = HttpResponse()
-        image = Filter(spec=filter_spec).run(image, response)
+        # Temporary image needs to be an instance that Willow can run optimizers on
+        temp_image = SpooledTemporaryFile(max_size=settings.FILE_UPLOAD_MAX_MEMORY_SIZE)
+        image = Filter(spec=filter_spec).run(image, temp_image)
+        temp_image.seek(0)
+        response = FileResponse(temp_image)
         response["Content-Type"] = "image/" + image.format_name
         return response
     except InvalidFilterSpecError:
diff --git a/wagtail/migrations/0094_alter_page_locale.py b/wagtail/migrations/0094_alter_page_locale.py
new file mode 100644
index 0000000000..de28c6a030
--- /dev/null
+++ b/wagtail/migrations/0094_alter_page_locale.py
@@ -0,0 +1,19 @@
+# Generated by Django 5.0.6 on 2024-06-21 06:47
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('wagtailcore', '0093_uploadedfile'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='page',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+    ]
diff --git a/wagtail/models/i18n.py b/wagtail/models/i18n.py
index 1fedade98c..36d21edf45 100644
--- a/wagtail/models/i18n.py
+++ b/wagtail/models/i18n.py
@@ -9,6 +9,7 @@ from django.db.models.signals import pre_save
 from django.dispatch import receiver
 from django.utils import translation
 from django.utils.encoding import force_str
+from django.utils.translation import gettext_lazy as _
 from modelcluster.fields import ParentalKey
 
 from wagtail.actions.copy_for_translation import CopyForTranslationAction
@@ -177,7 +178,11 @@ class Locale(models.Model):
 class TranslatableMixin(models.Model):
     translation_key = models.UUIDField(default=uuid.uuid4, editable=False)
     locale = models.ForeignKey(
-        Locale, on_delete=models.PROTECT, related_name="+", editable=False
+        Locale,
+        on_delete=models.PROTECT,
+        related_name="+",
+        editable=False,
+        verbose_name=_("locale"),
     )
     locale.wagtail_reference_index_ignore = True
 


Test patch:
diff --git a/wagtail/admin/tests/test_page_chooser.py b/wagtail/admin/tests/test_page_chooser.py
index 836055699b..d054f8b227 100644
--- a/wagtail/admin/tests/test_page_chooser.py
+++ b/wagtail/admin/tests/test_page_chooser.py
@@ -724,6 +724,7 @@ class TestChooserExternalLink(WagtailTestUtils, TestCase):
         self.assertEqual(
             response_json["external"]["url"], "http://localhost/about?test=1"
         )
+        self.assertEqual(response_json["internal"]["url"], "/about/")
         self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
 
     def test_convert_relative_external_link_to_internal_link(self):
@@ -752,6 +753,7 @@ class TestChooserExternalLink(WagtailTestUtils, TestCase):
 
         self.assertEqual(response_json["result"]["url"], url)
         self.assertEqual(response_json["result"]["title"], title)
+        self.assertNotIn("id", response_json["result"])
 
     @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="exact")
     def test_no_confirm_external_to_internal_link_when_exact(self):
@@ -767,6 +769,7 @@ class TestChooserExternalLink(WagtailTestUtils, TestCase):
 
         self.assertEqual(response_json["result"]["url"], url)
         self.assertEqual(response_json["result"]["title"], title)
+        self.assertNotIn("id", response_json["result"])
 
     @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="confirm")
     def test_convert_external_link_to_internal_link_with_confirm_setting(self):
@@ -785,9 +788,308 @@ class TestChooserExternalLink(WagtailTestUtils, TestCase):
         self.assertEqual(response_json["step"], "confirm_external_to_internal")
 
         self.assertEqual(response_json["external"]["url"], url)
+        self.assertEqual(response_json["internal"]["url"], "/about/")
         self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
 
 
+@override_settings(ROOT_URLCONF="wagtail.test.headless_urls")
+class TestChooserExternalLinkWithNoServePath(TestChooserExternalLink):
+    def test_convert_external_to_internal_link(self):
+        # Normally this should be converted without any confirmation, but since
+        # the serve path is not registered, the page route will resolve to None,
+        # so the user should be asked to confirm the conversion.
+        # As a result, this test is now identical to
+        # test_convert_external_link_to_internal_link_with_confirm_setting
+        response = self.post(
+            {
+                "external-link-chooser-url": "http://localhost/about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+        self.assertEqual(response_json["external"]["url"], "http://localhost/about/")
+        self.assertIsNone(response_json["internal"]["url"])
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_external_link_with_query_parameters_to_internal_link(self):
+        response = self.post(
+            {
+                "external-link-chooser-url": "http://localhost/about?test=1",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+
+        # Query parameters will get stripped, so the user should get asked to confirm the conversion
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+
+        self.assertEqual(
+            response_json["external"]["url"], "http://localhost/about?test=1"
+        )
+        # The serve path is not registered, so the page route will resolve to None
+        self.assertIsNone(response_json["internal"]["url"])
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_relative_external_link_to_internal_link(self):
+        response = self.post(
+            {
+                "external-link-chooser-url": "/about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        # As with test_convert_external_to_internal_link, normally this doesn't
+        # require confirmation, but since the serve path is not registered, the
+        # full URL of the page is None, thus triggering the confirmation
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+        self.assertEqual(response_json["external"]["url"], "/about/")
+        # The serve path is not registered, so the page route will resolve to None
+        self.assertIsNone(response_json["internal"]["url"])
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="confirm")
+    def test_convert_external_link_to_internal_link_with_confirm_setting(self):
+        url = "http://localhost/about/"
+        response = self.post(
+            {
+                "external-link-chooser-url": url,
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+
+        # The serve path is not registered so the URL can never be identical,
+        # which means the setting doesn't matter in this case, but we keep the
+        # test case anyway to ensure that the confirmation step is triggered
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+
+        self.assertEqual(response_json["external"]["url"], url)
+        self.assertIsNone(response_json["internal"]["url"])
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+
+@override_settings(ROOT_URLCONF="wagtail.test.non_root_urls")
+class TestChooserExternalLinkWithNonRootServePath(TestChooserExternalLink):
+    prefix = "site/"
+
+    def test_convert_external_to_internal_link(self):
+        # Legacy behaviour: when using a non-root serve path, entering a full
+        # URL without the serve path will trigger the conversion.
+        # Normally this should be converted without any confirmation, but since
+        # the actual URL will include the serve path (thus different from the
+        # input URL), the user should be asked to confirm.
+        response = self.post(
+            {
+                "external-link-chooser-url": "http://localhost/about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+        self.assertEqual(response_json["external"]["url"], "http://localhost/about/")
+        self.assertEqual(response_json["internal"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_external_to_internal_link_with_serve_path(self):
+        # https://github.com/wagtail/wagtail/issues/11996
+        # New behaviour: when using a non-root serve path, entering a full
+        # URL with the serve path will trigger the conversion. Previously, this
+        # would have been considered an external link since the logic does not
+        # take the serve path into account, but now it should be correctly
+        # converted to an internal link (without needing confirmation as the
+        # input URL will be an exact match to the page's full URL).
+
+        # Warm up the cache
+        response = self.post(
+            {
+                "external-link-chooser-url": f"http://localhost/{self.prefix}about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        with self.assertNumQueries(11):
+            response = self.post(
+                {
+                    "external-link-chooser-url": f"http://localhost/{self.prefix}about/",
+                    "external-link-chooser-link_text": "about",
+                }
+            )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "external_link_chosen")
+        self.assertEqual(response_json["result"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["result"]["id"], self.internal_page.pk)
+
+    def test_convert_external_link_with_query_parameters_to_internal_link(self):
+        # Legacy behaviour: when using a non-root serve path, entering a full
+        # URL without the serve path will trigger the conversion. The user
+        # should be asked to confirm the conversion as the input URL will be
+        # different from the page's full URL (as the serve path is added and
+        # the query parameters are removed).
+        response = self.post(
+            {
+                "external-link-chooser-url": "http://localhost/about?test=1",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+
+        # Query parameters will get stripped, so the user should get asked to confirm the conversion
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+
+        self.assertEqual(
+            response_json["external"]["url"], "http://localhost/about?test=1"
+        )
+        self.assertEqual(response_json["internal"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_external_link_with_query_parameters_to_internal_link_with_serve_path(
+        self
+    ):
+        # https://github.com/wagtail/wagtail/issues/11996
+        # New behaviour: when using a non-root serve path, entering a full
+        # URL with the serve path will trigger the conversion. Previously, this
+        # would have been considered an external link since the logic does not
+        # take the serve path into account, but now it should be correctly
+        # converted to an internal link. It still needs a confirmation as the
+        # input URL includes query parameters that will be stripped when matching
+        # it with the page's full URL.
+        response = self.post(
+            {
+                "external-link-chooser-url": f"http://localhost/{self.prefix}about?test=1",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+
+        # Query parameters will get stripped, so the user should get asked to confirm the conversion
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+
+        self.assertEqual(
+            response_json["external"]["url"],
+            f"http://localhost/{self.prefix}about?test=1",
+        )
+        self.assertEqual(response_json["internal"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_relative_external_link_to_internal_link(self):
+        # Legacy behaviour: when using a non-root serve path, entering a relative
+        # URL without the serve path will trigger the conversion.
+        # Normally this should be converted without any confirmation, but since
+        # the actual URL will include the serve path (thus different from the
+        # input URL), the user should be asked to confirm.
+        response = self.post(
+            {
+                "external-link-chooser-url": "/about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+        self.assertEqual(response_json["external"]["url"], "/about/")
+        self.assertEqual(response_json["internal"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+    def test_convert_relative_external_link_to_internal_link_with_serve_path(self):
+        # https://github.com/wagtail/wagtail/issues/11996
+        # New behaviour: when using a non-root serve path, entering a relative
+        # URL with the serve path will trigger the conversion. Previously, this
+        # would have been considered an external link since the logic does not
+        # take the serve path into account, but now it should be correctly
+        # converted to an internal link (without needing confirmation as the
+        # input URL will be an exact match to the page's relative URL).
+        response = self.post(
+            {
+                "external-link-chooser-url": f"/{self.prefix}about/",
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "external_link_chosen")
+        self.assertEqual(response_json["result"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["result"]["id"], self.internal_page.pk)
+
+    @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="")
+    def test_no_conversion_external_to_internal_link_when_disabled_with_serve_path(
+        self
+    ):
+        url = f"http://localhost/{self.prefix}about/"
+        title = "about"
+        response = self.post(
+            {"external-link-chooser-url": url, "external-link-chooser-link_text": title}
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        self.assertEqual(response_json["step"], "external_link_chosen")
+
+        self.assertEqual(response_json["result"]["url"], url)
+        self.assertEqual(response_json["result"]["title"], title)
+        self.assertNotIn("id", response_json["result"])
+
+    @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="exact")
+    def test_no_confirm_external_to_internal_link_when_exact_with_serve_path(self):
+        url = f"http://localhost/{self.prefix}about?test=1"
+        title = "about"
+        response = self.post(
+            {"external-link-chooser-url": url, "external-link-chooser-link_text": title}
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+        # Query parameters will get stripped, so this link should be left as an external url with the 'exact' setting
+        self.assertEqual(response_json["step"], "external_link_chosen")
+
+        self.assertEqual(response_json["result"]["url"], url)
+        self.assertEqual(response_json["result"]["title"], title)
+        self.assertNotIn("id", response_json["result"])
+
+    @override_settings(WAGTAILADMIN_EXTERNAL_LINK_CONVERSION="confirm")
+    def test_convert_external_link_to_internal_link_with_confirm_setting(self):
+        url = f"http://localhost/{self.prefix}about/"
+        response = self.post(
+            {
+                "external-link-chooser-url": url,
+                "external-link-chooser-link_text": "about",
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+        response_json = json.loads(response.content.decode())
+
+        # The url is identical, but the conversion setting is set to 'confirm'
+        # so the user should get asked to confirm the conversion
+        self.assertEqual(response_json["step"], "confirm_external_to_internal")
+
+        self.assertEqual(response_json["external"]["url"], url)
+        self.assertEqual(response_json["internal"]["url"], f"/{self.prefix}about/")
+        self.assertEqual(response_json["internal"]["id"], self.internal_page.pk)
+
+
+@override_settings(
+    WAGTAIL_I18N_ENABLED=True,
+    ROOT_URLCONF="wagtail.test.urls_multilang",
+)
+class TestChooserExternalLinkWithI18n(TestChooserExternalLinkWithNonRootServePath):
+    prefix = "en/"
+
+
+@override_settings(
+    WAGTAIL_I18N_ENABLED=True,
+    ROOT_URLCONF="wagtail.test.urls_multilang_non_root",
+)
+class TestChooserExternalLinkWithI18nNonRoot(
+    TestChooserExternalLinkWithNonRootServePath
+):
+    prefix = "en/site/"
+
+
 class TestChooserAnchorLink(WagtailTestUtils, TestCase):
     def setUp(self):
         self.login()
diff --git a/wagtail/api/v2/tests/test_pages.py b/wagtail/api/v2/tests/test_pages.py
index cc84ba1f6a..57b38f3f5b 100644
--- a/wagtail/api/v2/tests/test_pages.py
+++ b/wagtail/api/v2/tests/test_pages.py
@@ -683,6 +683,16 @@ class TestPageListing(WagtailTestUtils, TestCase):
             },
         )
 
+    def test_slug_field_containing_null_bytes_gives_error(self):
+        response = self.get_response(slug="\0")
+        content = json.loads(response.content.decode("UTF-8"))
+
+        self.assertEqual(response.status_code, 400)
+        self.assertEqual(
+            content,
+            {"message": "field filter error. null characters are not allowed for slug"},
+        )
+
     # CHILD OF FILTER
 
     def test_child_of_filter(self):
diff --git a/wagtail/contrib/typed_table_block/tests.py b/wagtail/contrib/typed_table_block/tests.py
index cd8d5a7d4f..c89b28a1fb 100644
--- a/wagtail/contrib/typed_table_block/tests.py
+++ b/wagtail/contrib/typed_table_block/tests.py
@@ -4,6 +4,7 @@ from django.test import TestCase
 
 from wagtail import blocks
 from wagtail.blocks.base import get_error_json_data
+from wagtail.blocks.definition_lookup import BlockDefinitionLookup
 from wagtail.blocks.struct_block import StructBlockValidationError
 from wagtail.contrib.typed_table_block.blocks import (
     TypedTable,
@@ -257,3 +258,40 @@ class TestTableBlock(TestCase):
                 "blockErrors": {1: {2: {"messages": ["This field is required."]}}},
             },
         )
+
+
+class TestBlockDefinitionLookup(TestCase):
+    def test_block_lookup(self):
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: (
+                    "wagtail.blocks.ChoiceBlock",
+                    [],
+                    {
+                        "choices": [
+                            ("be", "Belgium"),
+                            ("fr", "France"),
+                            ("nl", "Netherlands"),
+                        ]
+                    },
+                ),
+                2: (
+                    "wagtail.contrib.typed_table_block.blocks.TypedTableBlock",
+                    [
+                        [
+                            ("text", 0),
+                            ("country", 1),
+                        ],
+                    ],
+                    {},
+                ),
+            }
+        )
+        struct_block = lookup.get_block(2)
+        self.assertIsInstance(struct_block, TypedTableBlock)
+        text_block = struct_block.child_blocks["text"]
+        self.assertIsInstance(text_block, blocks.CharBlock)
+        self.assertTrue(text_block.required)
+        country_block = struct_block.child_blocks["country"]
+        self.assertIsInstance(country_block, blocks.ChoiceBlock)
diff --git a/wagtail/images/tests/test_admin_views.py b/wagtail/images/tests/test_admin_views.py
index 47ca175b13..e89dd091bd 100644
--- a/wagtail/images/tests/test_admin_views.py
+++ b/wagtail/images/tests/test_admin_views.py
@@ -1,6 +1,7 @@
 import datetime
 import json
 import urllib
+from unittest.mock import patch
 
 from django.conf import settings
 from django.contrib.auth.models import Group, Permission
@@ -14,6 +15,8 @@ from django.utils.encoding import force_str
 from django.utils.html import escape, escapejs
 from django.utils.http import RFC3986_SUBDELIMS, urlencode
 from django.utils.safestring import mark_safe
+from willow.optimizers.base import OptimizerBase
+from willow.registry import registry
 
 from wagtail.admin.admin_url_finder import AdminURLFinder
 from wagtail.images import get_image_model
@@ -3535,6 +3538,35 @@ class TestPreviewView(WagtailTestUtils, TestCase):
         self.assertEqual(response.status_code, 200)
         self.assertEqual(response["Content-Type"], "image/png")
 
+    def test_preview_with_optimizer(self):
+        """
+        Test that preview works with optimizers
+
+        Willow optimizers require
+        """
+
+        class DummyOptimizer(OptimizerBase):
+            library_name = "dummy"
+            image_format = "png"
+
+            @classmethod
+            def check_library(cls):
+                return True
+
+            @classmethod
+            def process(cls, file_path: str):
+                pass
+
+        # Get the image
+        with patch.object(registry, "_registered_optimizers", [DummyOptimizer]):
+            response = self.client.get(
+                reverse("wagtailimages:preview", args=(self.image.id, "fill-800x600"))
+            )
+
+        # Check response
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response["Content-Type"], "image/png")
+
     def test_get_invalid_filter_spec(self):
         """
         Test that an invalid filter spec returns a 400 response
diff --git a/wagtail/test/i18n/migrations/0003_alter_clusterabletestmodel_locale_and_more.py b/wagtail/test/i18n/migrations/0003_alter_clusterabletestmodel_locale_and_more.py
new file mode 100644
index 0000000000..17538c27b8
--- /dev/null
+++ b/wagtail/test/i18n/migrations/0003_alter_clusterabletestmodel_locale_and_more.py
@@ -0,0 +1,40 @@
+# Generated by Django 5.0.6 on 2024-06-21 06:47
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('i18n', '0002_clusterabletestmodel_clusterabletestmodelchild_clusterabletestmodeltranslatablechild'),
+        ('wagtailcore', '0094_alter_page_locale'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='clusterabletestmodel',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='clusterabletestmodeltranslatablechild',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='testchildobject',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='testmodel',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='testnonparentalchildobject',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+    ]
diff --git a/wagtail/test/snippets/migrations/0012_alter_translatablesnippet_locale.py b/wagtail/test/snippets/migrations/0012_alter_translatablesnippet_locale.py
new file mode 100644
index 0000000000..00b545f7ef
--- /dev/null
+++ b/wagtail/test/snippets/migrations/0012_alter_translatablesnippet_locale.py
@@ -0,0 +1,20 @@
+# Generated by Django 5.0.6 on 2024-06-21 06:47
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('snippetstests', '0011_nonautocompletesearchablesnippet'),
+        ('wagtailcore', '0094_alter_page_locale'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='translatablesnippet',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+    ]
diff --git a/wagtail/test/testapp/migrations/0039_alter_eventcategory_locale_and_more.py b/wagtail/test/testapp/migrations/0039_alter_eventcategory_locale_and_more.py
new file mode 100644
index 0000000000..494e8b5239
--- /dev/null
+++ b/wagtail/test/testapp/migrations/0039_alter_eventcategory_locale_and_more.py
@@ -0,0 +1,45 @@
+# Generated by Django 5.0.6 on 2024-06-21 06:47
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('tests', '0038_sociallink'),
+        ('wagtailcore', '0094_alter_page_locale'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='eventcategory',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='eventpagecarouselitem',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='eventpagerelatedlink',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='eventpagespeaker',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='eventpagespeakeraward',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+        migrations.AlterField(
+            model_name='fullfeaturedsnippet',
+            name='locale',
+            field=models.ForeignKey(editable=False, on_delete=django.db.models.deletion.PROTECT, related_name='+', to='wagtailcore.locale', verbose_name='locale'),
+        ),
+    ]
diff --git a/wagtail/test/urls_multilang.py b/wagtail/test/urls_multilang.py
index 16125f7318..222bb160f4 100644
--- a/wagtail/test/urls_multilang.py
+++ b/wagtail/test/urls_multilang.py
@@ -2,5 +2,10 @@ from django.conf.urls.i18n import i18n_patterns
 from django.urls import include, path
 
 from wagtail import urls as wagtail_urls
+from wagtail.admin import urls as wagtailadmin_urls
 
-urlpatterns = i18n_patterns(path("", include(wagtail_urls)))
+urlpatterns = [
+    path("admin/", include(wagtailadmin_urls)),
+]
+
+urlpatterns += i18n_patterns(path("", include(wagtail_urls)))
diff --git a/wagtail/test/urls_multilang_non_root.py b/wagtail/test/urls_multilang_non_root.py
new file mode 100644
index 0000000000..7715be7746
--- /dev/null
+++ b/wagtail/test/urls_multilang_non_root.py
@@ -0,0 +1,11 @@
+from django.conf.urls.i18n import i18n_patterns
+from django.urls import include, path
+
+from wagtail import urls as wagtail_urls
+from wagtail.admin import urls as wagtailadmin_urls
+
+urlpatterns = [
+    path("admin/", include(wagtailadmin_urls)),
+]
+
+urlpatterns += i18n_patterns(path("site/", include(wagtail_urls)))
diff --git a/wagtail/tests/test_blocks.py b/wagtail/tests/test_blocks.py
index 524fe19677..adcf0e097c 100644
--- a/wagtail/tests/test_blocks.py
+++ b/wagtail/tests/test_blocks.py
@@ -17,6 +17,7 @@ from django.utils.translation import gettext_lazy as _
 
 from wagtail import blocks
 from wagtail.blocks.base import get_error_json_data
+from wagtail.blocks.definition_lookup import BlockDefinitionLookup
 from wagtail.blocks.field_block import FieldBlockAdapter
 from wagtail.blocks.list_block import ListBlockAdapter, ListBlockValidationError
 from wagtail.blocks.static_block import StaticBlockAdapter
@@ -5882,3 +5883,105 @@ class TestValidationErrorAsJsonData(TestCase):
                 ],
             },
         )
+
+
+class TestBlockDefinitionLookup(TestCase):
+    def test_simple_lookup(self):
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.RichTextBlock", [], {}),
+            }
+        )
+        char_block = lookup.get_block(0)
+        char_block.set_name("title")
+        self.assertIsInstance(char_block, blocks.CharBlock)
+        self.assertTrue(char_block.required)
+
+        rich_text_block = lookup.get_block(1)
+        self.assertIsInstance(rich_text_block, blocks.RichTextBlock)
+
+        # A subsequent call to get_block with the same index should return a new instance;
+        # this ensures that state changes such as set_name are independent of other blocks
+        char_block_2 = lookup.get_block(0)
+        char_block_2.set_name("subtitle")
+        self.assertIsInstance(char_block, blocks.CharBlock)
+        self.assertTrue(char_block.required)
+        self.assertIsNot(char_block, char_block_2)
+        self.assertEqual(char_block.name, "title")
+        self.assertEqual(char_block_2.name, "subtitle")
+
+    def test_structblock_lookup(self):
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.RichTextBlock", [], {}),
+                2: (
+                    "wagtail.blocks.StructBlock",
+                    [
+                        [
+                            ("title", 0),
+                            ("description", 1),
+                        ],
+                    ],
+                    {},
+                ),
+            }
+        )
+        struct_block = lookup.get_block(2)
+        self.assertIsInstance(struct_block, blocks.StructBlock)
+        title_block = struct_block.child_blocks["title"]
+        self.assertIsInstance(title_block, blocks.CharBlock)
+        self.assertTrue(title_block.required)
+        description_block = struct_block.child_blocks["description"]
+        self.assertIsInstance(description_block, blocks.RichTextBlock)
+
+    def test_streamblock_lookup(self):
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.RichTextBlock", [], {}),
+                2: (
+                    "wagtail.blocks.StreamBlock",
+                    [
+                        [
+                            ("heading", 0),
+                            ("paragraph", 1),
+                        ],
+                    ],
+                    {},
+                ),
+            }
+        )
+        stream_block = lookup.get_block(2)
+        self.assertIsInstance(stream_block, blocks.StreamBlock)
+        title_block = stream_block.child_blocks["heading"]
+        self.assertIsInstance(title_block, blocks.CharBlock)
+        self.assertTrue(title_block.required)
+        description_block = stream_block.child_blocks["paragraph"]
+        self.assertIsInstance(description_block, blocks.RichTextBlock)
+
+    def test_listblock_lookup(self):
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.ListBlock", [0], {}),
+            }
+        )
+        list_block = lookup.get_block(1)
+        self.assertIsInstance(list_block, blocks.ListBlock)
+        list_item_block = list_block.child_block
+        self.assertIsInstance(list_item_block, blocks.CharBlock)
+        self.assertTrue(list_item_block.required)
+
+        # Passing a class as the child block is still valid; this is not converted
+        # to a reference
+        lookup = BlockDefinitionLookup(
+            {
+                0: ("wagtail.blocks.ListBlock", [blocks.CharBlock], {}),
+            }
+        )
+        list_block = lookup.get_block(0)
+        self.assertIsInstance(list_block, blocks.ListBlock)
+        list_item_block = list_block.child_block
+        self.assertIsInstance(list_item_block, blocks.CharBlock)
diff --git a/wagtail/tests/test_streamfield.py b/wagtail/tests/test_streamfield.py
index df799fb1f4..6ad72b7e85 100644
--- a/wagtail/tests/test_streamfield.py
+++ b/wagtail/tests/test_streamfield.py
@@ -721,3 +721,156 @@ class TestGetBlockByContentPath(TestCase):
         self.assertEqual(bound_block.value, "Barnaby Rudge")
         bound_block = field.get_block_by_content_path(self.page.body, ["456", "999"])
         self.assertIsNone(bound_block)
+
+
+class TestConstructStreamFieldFromLookup(TestCase):
+    def test_construct_block_list_from_lookup(self):
+        field = StreamField(
+            [
+                ("heading", 0),
+                ("paragraph", 1),
+                ("button", 3),
+            ],
+            block_lookup={
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.RichTextBlock", [], {}),
+                2: ("wagtail.blocks.PageChooserBlock", [], {}),
+                3: (
+                    "wagtail.blocks.StructBlock",
+                    [
+                        [
+                            ("page", 2),
+                            ("link_text", 0),
+                        ]
+                    ],
+                    {},
+                ),
+            },
+        )
+        stream_block = field.stream_block
+        self.assertIsInstance(stream_block, blocks.StreamBlock)
+        self.assertEqual(len(stream_block.child_blocks), 3)
+
+        heading_block = stream_block.child_blocks["heading"]
+        self.assertIsInstance(heading_block, blocks.CharBlock)
+        self.assertTrue(heading_block.required)
+        self.assertEqual(heading_block.name, "heading")
+
+        paragraph_block = stream_block.child_blocks["paragraph"]
+        self.assertIsInstance(paragraph_block, blocks.RichTextBlock)
+        self.assertEqual(paragraph_block.name, "paragraph")
+
+        button_block = stream_block.child_blocks["button"]
+        self.assertIsInstance(button_block, blocks.StructBlock)
+        self.assertEqual(button_block.name, "button")
+        self.assertEqual(len(button_block.child_blocks), 2)
+        page_block = button_block.child_blocks["page"]
+        self.assertIsInstance(page_block, blocks.PageChooserBlock)
+        link_text_block = button_block.child_blocks["link_text"]
+        self.assertIsInstance(link_text_block, blocks.CharBlock)
+        self.assertEqual(link_text_block.name, "link_text")
+
+    def test_construct_top_level_block_from_lookup(self):
+        field = StreamField(
+            4,
+            block_lookup={
+                0: ("wagtail.blocks.CharBlock", [], {"required": True}),
+                1: ("wagtail.blocks.RichTextBlock", [], {}),
+                2: ("wagtail.blocks.PageChooserBlock", [], {}),
+                3: (
+                    "wagtail.blocks.StructBlock",
+                    [
+                        [
+                            ("page", 2),
+                            ("link_text", 0),
+                        ]
+                    ],
+                    {},
+                ),
+                4: (
+                    "wagtail.blocks.StreamBlock",
+                    [
+                        [
+                            ("heading", 0),
+                            ("paragraph", 1),
+                            ("button", 3),
+                        ]
+                    ],
+                    {},
+                ),
+            },
+        )
+        stream_block = field.stream_block
+        self.assertIsInstance(stream_block, blocks.StreamBlock)
+        self.assertEqual(len(stream_block.child_blocks), 3)
+
+        heading_block = stream_block.child_blocks["heading"]
+        self.assertIsInstance(heading_block, blocks.CharBlock)
+        self.assertTrue(heading_block.required)
+        self.assertEqual(heading_block.name, "heading")
+
+        paragraph_block = stream_block.child_blocks["paragraph"]
+        self.assertIsInstance(paragraph_block, blocks.RichTextBlock)
+        self.assertEqual(paragraph_block.name, "paragraph")
+
+        button_block = stream_block.child_blocks["button"]
+        self.assertIsInstance(button_block, blocks.StructBlock)
+        self.assertEqual(button_block.name, "button")
+        self.assertEqual(len(button_block.child_blocks), 2)
+        page_block = button_block.child_blocks["page"]
+        self.assertIsInstance(page_block, blocks.PageChooserBlock)
+        link_text_block = button_block.child_blocks["link_text"]
+        self.assertIsInstance(link_text_block, blocks.CharBlock)
+        self.assertEqual(link_text_block.name, "link_text")
+
+
+class TestDeconstructStreamFieldWithLookup(TestCase):
+    def test_deconstruct(self):
+        class ButtonBlock(blocks.StructBlock):
+            page = blocks.PageChooserBlock()
+            link_text = blocks.CharBlock(required=True)
+
+        field = StreamField(
+            [
+                ("heading", blocks.CharBlock(required=True)),
+                ("paragraph", blocks.RichTextBlock()),
+                ("button", ButtonBlock()),
+            ],
+            blank=True,
+        )
+        field.set_attributes_from_name("body")
+
+        name, path, args, kwargs = field.deconstruct()
+        self.assertEqual(name, "body")
+        self.assertEqual(path, "wagtail.fields.StreamField")
+        self.assertEqual(
+            args,
+            [
+                [
+                    ("heading", 0),
+                    ("paragraph", 1),
+                    ("button", 3),
+                ]
+            ],
+        )
+        self.assertEqual(
+            kwargs,
+            {
+                "blank": True,
+                "block_lookup": {
+                    0: ("wagtail.blocks.CharBlock", (), {"required": True}),
+                    1: ("wagtail.blocks.RichTextBlock", (), {}),
+                    2: ("wagtail.blocks.PageChooserBlock", (), {}),
+                    3: (
+                        "wagtail.blocks.StructBlock",
+                        [
+                            [
+                                ("page", 2),
+                                ("link_text", 0),
+                            ]
+                        ],
+                        {},
+                    ),
+                },
+            },
+        )