Repository: lektor/lektor (python)

Issue:
Deprecated Werkzeug APIs in use
Werkzeug 2.3 deprecated a number of functions in the `werkzeug.urls` module. Lektor uses some of these methods:
* `werkzeug.urls.url_parse` (in favor of `urllib.parse.urlsplit`)
* `werkzeug.urls.url_join` (in favor of `urllib.parse.urljoin`)

It also removes the `werkzeug.urls.URL` wrapper class.

As a result, running Lektor with Werkzeug 2.3+ causes hundreds of deprecation warnings.



Pull request:
Fix #1142: address deprecation of werkzeug.urls.url_parse
<!---
Remember to please check that your code passes tests locally. Test with the Makefile
commands `make test`, or if you only need to test just Python / JS `make test-python`
or `make test-js`.
--->

### Issue(s) Resolved

<!--- List the issue(s) below, in the form "Fixes #1234"; one per line --->

Fixes #1142

### Related Issues / Links

Documentation updates for this PR are in lektor/lektor-website#371

<!---
Are there any similar or related issues or pull requests?
Did you make a pull request to update the docs?
--->

### Description of Changes

This PR disuses `werkzeug.urls.url_parse` in favor of the equivalents from stdlib's `urllib.parse`, thus avoiding deprecation warnings from werkzeug>=2.3.0, and (hopefully) ensuring compatibility with werkzeug 2.4.

#### Breaking Change of Publisher API

Currently, (before this PR) the `target_url` parameter of the [`Publisher.publish`](https://github.com/lektor/lektor/blob/cf6e23af371a10b581cb606d228fbdd831595007/lektor/publisher.py#L275-L281) method was expected to be a `werkzeug.urls.URL` instance.  Since `werkzeug.urls.URL` is going away, that is obviously not sustainable.  In this PR we change to passing a `str` for `target_url`.

Doing this the simple way, however, would _break existing external plugins_ (e.g. [lektor-algolia], [lektor-gae], [lektor-s3], [lektor-netlify], [lektor-git-src-publisher], [lektor-quiniu], [lektor-surge]) that implement custom publishers.
To prevent such breakage, in this PR we pass a custom subclass of `str` that implements (most of) the attributes and methods of `werkzeug.urls.URL` (but issues a deprecation warning when they are accessed.)

[lektor-algolia]: https://github.com/sherbondy/lektor-algolia/
[lektor-gae]: https://github.com/isotherm/lektor-gae/
[lektor-s3]: https://github.com/spenczar/lektor-s3/
[lektor-netlify]: https://github.com/nixjdm/lektor-netlify/
[lektor-git-src-publisher]: https://github.com/CamilionEU/lektor-git-src-publisher/
[lektor-quiniu]: https://github.com/Arstman/lektor-qiniu/
[lektor-surge]: https://github.com/nixjdm/lektor-surge/

#### Breaking Changelet — Url.query

Instances of `lektor.utils.Url` are what is exposed to jinja templates for Lektor record fields of type `"url"`.

This PR changes the semantics of `lektor.utils.Url.query` from returning the URI-encoded (ASCII) *query* to returning the IRI-encoded (internationalize, unicode) version of the _query_.  The other similar attributes of `Url` (e.g. `Url.host`, `Url.path`, `Url.anchor`) all already returned the IRI versions, so I'm not sure why `.query` was being treated differently.

#### Deprecation — Pad.make_absolute_url

This PR deprecates `lektor.db.Pad.make_absolute_url`.  We have not used it since 93de7d988d9bd5882e0c70f7d304b00a7fe78218, and, anyway, `lektor.db.Pad.make_url` provides equivalent functionality.


- [x] Wrote at least one-line docstrings (for any new functions)
- [x] Added unit test(s) covering the changes (if testable)
- [x] Link to corresponding documentation pull request for [getlektor.com](https://www.getlektor.com/docs/api/publisher/publish/)

<!--- Explain what you've done and why --->

<!--- Thanks for your help making Lektor better for everyone! --->


Patch:
diff --git a/lektor/compat.py b/lektor/compat.py
index b73e3c2..7df0859 100644
--- a/lektor/compat.py
+++ b/lektor/compat.py
@@ -1,11 +1,19 @@
+from __future__ import annotations
+
 import os
 import stat
 import sys
 import tempfile
+import urllib.parse
 from functools import partial
 from itertools import chain
+from typing import Any
+from urllib.parse import urlsplit
+
+from werkzeug import urls as werkzeug_urls
+from werkzeug.datastructures import MultiDict
 
-__all__ = ["TemporaryDirectory", "importlib_metadata"]
+__all__ = ["TemporaryDirectory", "importlib_metadata", "werkzeug_urls_URL"]
 
 
 def _ensure_tree_writeable(path: str) -> None:
@@ -55,3 +63,118 @@ if sys.version_info >= (3, 8):
 else:
     TemporaryDirectory = FixedTemporaryDirectory
     import importlib_metadata
+
+
+class _CompatURL(urllib.parse.SplitResult):
+    """This is a replacement for ``werkzeug.urls.URL``.
+
+    Here we implement those attributes and methods of ``URL`` which are
+    likely to be used by existing Lektor publishing plugins.
+
+    Currently unreimplemented here are the ``encode_netloc``, ``decode_netloc``,
+    ``get_file_location``, and ``encode`` methods of ``werkzeug.urls.URL``.
+
+    NB: Use of this class is deprecated. DO NOT USE THIS IN NEW CODE!
+
+    """
+
+    def __str__(self) -> str:
+        return self.geturl()
+
+    def replace(self, **kwargs: Any) -> _CompatURL:
+        return self._replace(**kwargs)
+
+    @property
+    def host(self) -> str | None:
+        return self.hostname
+
+    @property
+    def ascii_host(self) -> str | None:
+        host = self.hostname
+        if host is None:
+            return None
+        try:
+            return host.encode("idna").decode("ascii")
+        except UnicodeError:
+            return host
+
+    @property
+    def auth(self) -> str | None:
+        auth, _, _ = self.netloc.rpartition("@")
+        return auth if auth != "" else None
+
+    @property
+    def username(self) -> str | None:
+        username = super().username
+        if username is None:
+            return None
+        return _unquote_legacy(username)
+
+    @property
+    def raw_username(self) -> str | None:
+        return super().username
+
+    @property
+    def password(self) -> str | None:
+        password = super().password
+        if password is None:
+            return None
+        return _unquote_legacy(password)
+
+    @property
+    def raw_password(self) -> str | None:
+        return super().password
+
+    def decode_query(
+        self,
+        charset: str = "utf-8",
+        include_empty: bool = True,
+        errors: str = "replace",
+        # parse_qsl does not support the separator parameter in python < 3.7.10.
+        # separator: str = "&",
+    ) -> MultiDict:
+        return MultiDict(
+            urllib.parse.parse_qsl(
+                self.query,
+                keep_blank_values=include_empty,
+                encoding=charset,
+                errors=errors,
+                # separator=separator,
+            )
+        )
+
+    def join(
+        self, url: str | tuple[str, str, str, str, str], allow_fragments: bool = True
+    ) -> _CompatURL:
+        if isinstance(url, tuple):
+            url = urllib.parse.urlunsplit(url)
+        joined = urllib.parse.urljoin(self.geturl(), url, allow_fragments)
+        return _CompatURL._make(urlsplit(joined))
+
+    def to_url(self) -> str:
+        return self.geturl()
+
+    def to_uri_tuple(self) -> _CompatURL:
+        return _CompatURL._make(urlsplit(werkzeug_urls.iri_to_uri(self.geturl())))
+
+    def to_iri_tuple(self) -> _CompatURL:
+        return _CompatURL._make(urlsplit(werkzeug_urls.uri_to_iri(self.geturl())))
+
+
+def _unquote_legacy(value: str) -> str:
+    try:
+        return urllib.parse.unquote(value, "utf-8", "strict")
+    except UnicodeError:
+        return urllib.parse.unquote(value, "latin1")
+
+
+# Provide a replacement for the deprecated werkzeug.urls.URL class
+#
+# NB: Do not use this in new code!
+#
+# We only use this in lektor.publishers in order to provide some backward
+# compatibility for custom publishers from existing Lektor plugins.
+# At such point as we decide that backward-compatibility is no longer
+# needed, will be deleted.
+#
+werkzeug_urls_URL = getattr(werkzeug_urls, "URL", _CompatURL)
diff --git a/lektor/db.py b/lektor/db.py
index 7997a7e..8445610 100644
--- a/lektor/db.py
+++ b/lektor/db.py
@@ -11,12 +11,12 @@ from datetime import timedelta
 from functools import total_ordering
 from itertools import islice
 from operator import methodcaller
+from urllib.parse import urljoin
 
 from jinja2 import is_undefined
 from jinja2 import Undefined
 from jinja2.exceptions import UndefinedError
 from jinja2.utils import LRUCache
-from werkzeug.urls import url_join
 from werkzeug.utils import cached_property
 
 from lektor import metaformat
@@ -1580,6 +1580,7 @@ class Pad:
         """The env for this pad."""
         return self.db.env
 
+    @deprecated("use Pad.make_url instead", version="3.4.0")
     def make_absolute_url(self, url):
         """Given a URL this makes it absolute if this is possible."""
         base_url = self.db.config["PROJECT"].get("url")
@@ -1588,7 +1589,7 @@ class Pad:
                 "To use absolute URLs you need to configure "
                 "the URL in the project config."
             )
-        return url_join(base_url.rstrip("/") + "/", url.lstrip("/"))
+        return urljoin(base_url.rstrip("/") + "/", url.lstrip("/"))
 
     def make_url(self, url, base_url=None, absolute=None, external=None):
         """Helper method that creates a finalized URL based on the parameters
@@ -1613,9 +1614,9 @@ class Pad:
                     "To use absolute URLs you need to "
                     "configure the URL in the project config."
                 )
-            return url_join(external_base_url, url.lstrip("/"))
+            return urljoin(external_base_url, url.lstrip("/"))
         if absolute:
-            return url_join(self.db.config.base_path, url.lstrip("/"))
+            return urljoin(self.db.config.base_path, url.lstrip("/"))
         if base_url is None:
             raise RuntimeError(
                 "Cannot calculate a relative URL if no base URL has been provided."
diff --git a/lektor/environment/config.py b/lektor/environment/config.py
index 598ab17..a2fcea6 100644
--- a/lektor/environment/config.py
+++ b/lektor/environment/config.py
@@ -2,9 +2,9 @@ import copy
 import os
 import re
 from collections import OrderedDict
+from urllib.parse import urlsplit
 
 from inifile import IniFile
-from werkzeug.urls import url_parse
 from werkzeug.utils import cached_property
 
 from lektor.constants import PRIMARY_ALT
@@ -257,7 +257,7 @@ class Config:
     def base_url(self):
         """The external base URL."""
         url = self.values["PROJECT"].get("url")
-        if url and url_parse(url).scheme:
+        if url and urlsplit(url).scheme:
             return url.rstrip("/") + "/"
         return None
 
@@ -266,7 +266,7 @@ class Config:
         """The base path of the URL."""
         url = self.values["PROJECT"].get("url")
         if url:
-            return url_parse(url).path.rstrip("/") + "/"
+            return urlsplit(url).path.rstrip("/") + "/"
         return "/"
 
     @cached_property
diff --git a/lektor/publisher.py b/lektor/publisher.py
index 3c7e4ca..6bcd5a1 100644
--- a/lektor/publisher.py
+++ b/lektor/publisher.py
@@ -1,8 +1,11 @@
+from __future__ import annotations
+
 import errno
 import hashlib
 import io
 import os
 import posixpath
+import urllib.parse
 from contextlib import contextmanager
 from contextlib import ExitStack
 from contextlib import suppress
@@ -27,11 +30,13 @@ from typing import Optional
 from typing import Sequence
 from typing import Tuple
 from typing import TYPE_CHECKING
+from urllib.parse import urlsplit
 from warnings import warn
 
-from werkzeug import urls
+from werkzeug.datastructures import MultiDict
 
 from lektor.compat import TemporaryDirectory
+from lektor.compat import werkzeug_urls_URL
 from lektor.exception import LektorException
 from lektor.utils import bool_from_string
 from lektor.utils import locate_executable
@@ -43,6 +48,15 @@ if TYPE_CHECKING:  # pragma: no cover
     from lektor.environment import Environment
 
 
+def _parse_query(query: str, **kwargs: Any) -> MultiDict:
+    return MultiDict(urllib.parse.parse_qsl(query, **kwargs))
+
+
+def _ascii_host(host: str) -> str:
+    """Translate internationalized domain name to IDNA-encoded ASCII."""
+    return host.encode("idna").decode("ascii")
+
+
 @contextmanager
 def _ssh_key_file(
     credentials: Optional[Mapping[str, str]]
@@ -274,7 +288,7 @@ class Publisher:
 
     def publish(
         self,
-        target_url: urls.URL,
+        target_url: str,
         credentials: Optional[Mapping[str, str]] = None,
         **extra: Any,
     ) -> Iterator[str]:
@@ -289,7 +303,8 @@ class RsyncPublisher(Publisher):
         target = []
         env = {}
 
-        options = target_url.decode_query()
+        url = urlsplit(target_url)
+        options = _parse_query(url.query, keep_blank_values=True)
         exclude = options.getlist("exclude")
         for file in exclude:
             argline.extend(("--exclude", file))
@@ -298,18 +313,18 @@ class RsyncPublisher(Publisher):
         if delete:
             argline.append("--delete-after")
 
-        with _ssh_command(credentials, target_url.port) as ssh_command:
+        with _ssh_command(credentials, url.port) as ssh_command:
             if ssh_command:
                 argline.extend(("-e", ssh_command))
 
-            username = credentials.get("username") or target_url.username
+            username = credentials.get("username") or url.username
             if username:
                 target.append(username + "@")
 
-            if target_url.ascii_host is not None:
-                target.append(target_url.ascii_host)
+            if url.hostname is not None:
+                target.append(_ascii_host(url.hostname))
                 target.append(":")
-            target.append(target_url.path.rstrip("/") + "/")
+            target.append(url.path.rstrip("/") + "/")
 
             argline.append(self.output_path.rstrip("/\\") + "/")
             argline.append("".join(target))
@@ -321,8 +336,13 @@ class RsyncPublisher(Publisher):
 
 
 class FtpConnection:
-    def __init__(self, url, credentials=None):
+    def __init__(self, target_url, credentials=None):
         credentials = credentials or {}
+        url = urlsplit(target_url)
+        if url.hostname is None:
+            raise PublishError(
+                "No host name was specified in the target URL ({target_url})"
+            )
         self.con = self.make_connection()
         self.url = url
         self.username = credentials.get("username") or url.username
@@ -347,12 +367,15 @@ class FtpConnection:
                 yield line.rstrip()
 
     def connect(self):
-        options = self.url.decode_query()
+        options = _parse_query(self.url.query, keep_blank_values=True)
+        assert self.url.hostname is not None
+        host = _ascii_host(self.url.hostname)
+        port = self.url.port or 21
 
         log = self.log_buffer
         log.append("000 Connecting to server ...")
         try:
-            log.append(self.con.connect(self.url.ascii_host, self.url.port or 21))
+            log.append(self.con.connect(host, port))
         except Exception as e:
             log.append("000 Could not connect.")
             log.append(str(e))
@@ -789,7 +812,7 @@ class GithubPagesPublisher(Publisher):
 
     def publish(
         self,
-        target_url: urls.URL,
+        target_url: str,
         credentials: Optional[Mapping[str, str]] = None,
         **extra: Any,
     ) -> Iterator[str]:
@@ -811,16 +834,17 @@ class GithubPagesPublisher(Publisher):
             yield from repo.publish_ghpages(push_url, branch, cname, preserve_history)
 
     def _parse_url(
-        self, target_url: urls.URL
+        self, target_url: str
     ) -> Tuple[str, str, Optional[str], bool, Sequence[str]]:
-        if not target_url.host:
+        url = urlsplit(target_url)
+        if not url.hostname:
             self.fail("github owner missing from target URL")
-        gh_owner = target_url.host.lower()
-        gh_project = target_url.path.strip("/").lower()
+        gh_owner = url.hostname.lower()
+        gh_project = url.path.strip("/").lower()
         if not gh_project:
             self.fail("github project missing from target URL")
 
-        params = target_url.decode_query()
+        params = _parse_query(url.query, keep_blank_values=True)
         cname = params.get("cname")
         branch = params.get("branch")
         preserve_history = bool_from_string(params.get("preserve_history"), True)
@@ -843,13 +867,13 @@ class GithubPagesPublisher(Publisher):
             else:
                 branch = "gh-pages"
 
-        if target_url.scheme in ("ghpages", "ghpages+ssh"):
+        if url.scheme in ("ghpages", "ghpages+ssh"):
             push_url = f"ssh://git@github.com/{gh_owner}/{gh_project}.git"
             default_port = 22
         else:
             push_url = f"https://github.com/{gh_owner}/{gh_project}.git"
             default_port = 443
-        if target_url.port and target_url.port != default_port:
+        if url.port and url.port != default_port:
             self.fail("github does not support pushing to non-standard ports")
 
         return push_url, branch, cname, preserve_history, warnings
@@ -878,13 +902,14 @@ class GithubPagesPublisher(Publisher):
 
     @staticmethod
     def _parse_credentials(
-        credentials: Optional[Mapping[str, str]], target_url: urls.URL
+        credentials: Optional[Mapping[str, str]], target_url: str
     ) -> Mapping[str, str]:
+        url = urlsplit(target_url)
         creds = dict(credentials or {})
         # Fill in default username/password from target url
         for key, default in [
-            ("username", target_url.username),
-            ("password", target_url.password),
+            ("username", url.username),
+            ("password", url.password),
         ]:
             if not creds.get(key) and default:
                 creds[key] = default
@@ -902,8 +927,56 @@ builtin_publishers = {
 
 
 def publish(env, target, output_path, credentials=None, **extra):
-    url = urls.url_parse(str(target))
+    target_url = _CompatURLStr(target)
+    url = urlsplit(target_url)
     publisher = env.publishers.get(url.scheme)
     if publisher is None:
         raise PublishError('"%s" is an unknown scheme.' % url.scheme)
-    return publisher(env, output_path).publish(url, credentials, **extra)
+    return publisher(env, output_path).publish(target_url, credentials, **extra)
+
+
+class _CompatURLStr(str):
+    """A string that provides some features of the werkzeug.urls.URL split URL class.
+
+    We used to pass a ``werkzeug.urls.URL`` instance as the ``target_url`` argument to
+    the ``Publisher.publish`` method.  Werkzeug has deprecated the ``URL`` class, so
+    now we've changed our API to just pass a ``str`` for ``target_url``.
+
+    There are however, Lektor plugins out in the wild that provide their own custom
+    Publisher classes, and they expect a ``URL`` instance for ``target_url``.
+
+    Here we provide most of the methods and attributes of ``URL`` that might be of use
+    to a publisher, so as to try not to break all those existing plugins.
+
+    .. tip::
+       New plugins may preserve compatibility with older versions of Lektor by
+       first coercing their ``target_url`` parameter to a ``str`` before use.
+       (This works because ``werkzeug.urls.URL.__str__`` returns the reassembled URL.)
+       E.g. using ``urllib.parse.urlsplit`` to parse the URL:
+
+       .. code:: python
+         from urllib.parse import urlsplit
+
+         class CustomPublisher(Publisher):
+             def publish(self, target_url, credentials=None, **extra):
+                 url = urlsplit(str(target_url))
+                 host = url.hostname
+                 ...
+    """
+
+    def __getattr__(self, name: str):
+        if name.startswith("_"):
+            raise AttributeError(name)
+        url = werkzeug_urls_URL(*urlsplit(self))
+        rv = getattr(url, name)
+        warn(
+            "Since Lektor version 3.4, the 'target_url' parameter to the "
+            "'Publisher.publish' method is now a string rather than a "
+            "werkzeug.urls.URL instance.  To ease the transition, some "
+            "methods and attributes of werkzeugs.urls.URL are being emulated, "
+            "however that will not last forever. The plugin should be updated "
+            "to treat 'target_url' as a string.",
+            category=DeprecationWarning,
+            stacklevel=2,
+        )
+        return rv
diff --git a/lektor/types/special.py b/lektor/types/special.py
index 914ba1c..03ed42e 100644
--- a/lektor/types/special.py
+++ b/lektor/types/special.py
@@ -30,4 +30,4 @@ class UrlType(SingleInputType):
     def value_from_raw(self, raw):
         if raw.value is None:
             return raw.missing_value("Missing URL")
-        return Url(raw.value)
+        return Url.from_string(raw.value)
diff --git a/lektor/utils.py b/lektor/utils.py
index c7ddc92..000060d 100644
--- a/lektor/utils.py
+++ b/lektor/utils.py
@@ -9,6 +9,7 @@ import subprocess
 import sys
 import tempfile
 import unicodedata
+import urllib.parse
 import uuid
 import warnings
 from contextlib import contextmanager
@@ -23,14 +24,13 @@ from typing import Hashable
 from typing import Iterable
 from typing import overload
 from typing import TypeVar
-from urllib.parse import urlsplit
 
 from jinja2 import is_undefined
 from markupsafe import Markup
 from slugify import slugify as _slugify
-from werkzeug import urls
 from werkzeug.http import http_date
-from werkzeug.urls import url_parse
+from werkzeug.urls import iri_to_uri
+from werkzeug.urls import uri_to_iri
 
 
 is_windows = os.name == "nt"
@@ -98,7 +98,7 @@ def cleanup_url_path(url_path):
     Raises ValueError if the path contains a _scheme_
     which is neither ``http`` nor ``https``, or a _netloc_.
     """
-    scheme, netloc, path, _, _ = urlsplit(url_path, scheme="http")
+    scheme, netloc, path, _, _ = urllib.parse.urlsplit(url_path, scheme="http")
     if scheme not in ("http", "https"):
         raise ValueError(f"Invalid scheme: {url_path!r}")
     if netloc:
@@ -375,25 +375,87 @@ def tojson_filter(obj, **kwargs):
     return Markup(htmlsafe_json_dump(obj, **kwargs))
 
 
-class Url:
-    def __init__(self, value):
-        self.url = value
-        u = url_parse(value)
-        i = u.to_iri_tuple()
-        self.ascii_url = str(u)
-        self.host = i.host
-        self.ascii_host = u.ascii_host
-        self.port = u.port
-        self.path = i.path
-        self.query = u.query
-        self.anchor = i.fragment
-        self.scheme = u.scheme
-
-    def __unicode__(self):
+class Url(urllib.parse.SplitResult):
+    """Make various parts of a URL accessible.
+
+    This is the type of the values exposed by Lektor record fields of type "url".
+
+    Since Lektor 3.4.0, this is essentially a `urllib.parse.SplitResult` as obtained by
+    calling `urlsplit` on the URL normalized to an IRI.
+
+    Generally, attributes such as ``netloc``, ``host``, ``path``, ``query``, and
+    ``fragment`` return the IRI (internationalied) versions of those components.
+
+    The URI (ASCII-encoded) version of the URL is available from the `ascii_url`
+    attribute.
+
+    NB: Changed in 3.4.0: The ``query`` attribute used to return the URI
+    (ASCII-encoded) version of the query — I'm not sure why. Now it returns
+    the IRI (internationalized) version of the query.
+
+    """
+
+    def __new__(cls, value: str):
+        # XXX: deprecate use of constructor so that eventually we can make its signature
+        # match that of the SplitResult base class.
+        warnings.warn(
+            DeprecatedWarning(
+                "Url",
+                reason=(
+                    "Direct construction of a Url instance is deprecated. "
+                    "Use the Url.from_string classmethod instead."
+                ),
+                version="3.4.0",
+            )
+        )
+        return cls.from_string(value)
+
+    @classmethod
+    def from_string(cls, value: str) -> Url:
+        """Construct instance from URL string.
+
+        The input URL can be a URI (all ASCII) or an IRI (internationalized).
+        """
+        # The iri_to_uri operation is nominally idempotent — it can be passed either an
+        # IRI or a URI (or something inbetween) and will return a URI.  So to fully
+        # normalize input which can be either an IRI or a URI, first convert to URI,
+        # then to IRI.
+        iri = uri_to_iri(iri_to_uri(value))
+        obj = cls._make(urllib.parse.urlsplit(iri))
+        obj.url = value
+        return obj
+
+    def __str__(self) -> str:
+        """The original un-normalized URL string."""
         return self.url
 
-    def __str__(self):
-        return self.ascii_url
+    @property
+    def ascii_url(self) -> str:
+        """The URL encoded to an all-ASCII URI."""
+        return iri_to_uri(self.geturl())
+
+    @property
+    def ascii_host(self) -> str | None:
+        """The hostname part of the URL IDNA-encoded to ASCII."""
+        return urllib.parse.urlsplit(self.ascii_url).hostname
+
+    @property
+    def host(self) -> str | None:
+        """The IRI (internationalized) version of the hostname.
+
+        This attribute is provided for backwards-compatibility.  New code should use the
+        ``hostname`` attribute instead.
+        """
+        return self.hostname
+
+    @property
+    def anchor(self) -> str:
+        """The IRI (internationalized) version of the "anchor" part of the URL.
+
+        This attribute is provided for backwards-compatibility.  New code should use the
+        ``fragment`` attribute instead.
+        """
+        return self.fragment
 
 
 def is_unsafe_to_delete(path, base):
@@ -499,17 +561,12 @@ def is_valid_id(value):
     )
 
 
-def secure_url(url):
-    url = urls.url_parse(url)
-    if url.password is not None:
-        url = url.replace(
-            netloc="%s@%s"
-            % (
-                url.username,
-                url.netloc.split("@")[-1],
-            )
-        )
-    return url.to_url()
+def secure_url(url: str) -> str:
+    parts = urllib.parse.urlsplit(url)
+    if parts.password is not None:
+        _, _, host_port = parts.netloc.rpartition("@")
+        parts = parts._replace(netloc=f"{parts.username}@{host_port}")
+    return parts.geturl()
 
 
 def bool_from_string(val, default=None):


Test patch:
diff --git a/tests/test_compat.py b/tests/test_compat.py
index 03a3eb4..b6410aa 100644
--- a/tests/test_compat.py
+++ b/tests/test_compat.py
@@ -1,8 +1,11 @@
 import os
 from pathlib import Path
+from urllib.parse import urlsplit
 
 import pytest
+from werkzeug import urls as werkzeug_urls
 
+from lektor.compat import _CompatURL
 from lektor.compat import _ensure_tree_writeable
 from lektor.compat import FixedTemporaryDirectory
 from lektor.compat import TemporaryDirectory
@@ -30,3 +33,122 @@ def test_TemporaryDirectory(tmpdir_class):
         file.touch(mode=0)
         os.chmod(tmpdir, 0)
     assert not os.path.exists(tmpdir)
+
+
+def make_CompatURL(url: str) -> _CompatURL:
+    return _CompatURL._make(urlsplit(url))
+
+
+URL_MAKERS = [
+    pytest.param(make_CompatURL, id="_CompatURL"),
+    pytest.param(
+        getattr(werkzeug_urls, "url_parse", None),
+        id="werkzeug.urls.URL",
+        marks=pytest.mark.skipif(
+            not hasattr(werkzeug_urls, "url_parse"),
+            reason="werkzeug.urls.url_parse is not available",
+        ),
+    ),
+]
+
+
+@pytest.fixture(params=URL_MAKERS)
+def make_url(request):
+    return request.param
+
+
+pytestmark = pytest.mark.filterwarnings("ignore:'werkzeug:DeprecationWarning")
+
+
+def test_compatURL_replace(make_url):
+    url = make_url("http://example.org/foo")
+    assert str(url.replace(scheme="https", path="bar")) == "https://example.org/bar"
+
+
+def test_compatURL_host(make_url):
+    assert make_url("http://bücher.example/foo").host == "bücher.example"
+
+
+@pytest.mark.parametrize(
+    "url, expected",
+    [
+        ("http:/foo", [None]),
+        ("http://bücher.example/foo", ["xn--bcher-kva.example"]),
+        # werkzeug < 2.3 strips non-ascii
+        ("http://bad\uFFFF.example.org", ["bad\uFFFF.example.org", "bad.example.org"]),
+    ],
+)
+def test_compatURL_ascii_host(url, expected, make_url):
+    assert make_url(url).ascii_host in expected
+
+
+def test_compatURL_auth(make_url):
+    assert make_url("http://u:p@example.org/").auth == "u:p"
+
+
+@pytest.mark.parametrize(
+    "url, expected",
+    [
+        ("http://example.org/", None),
+        ("http://a%20user@example.org/", "a user"),
+        ("http://%C3%BCser:pw@example.org/", "üser"),
+        ("http://%FCser:pw@example.org/", "üser"),  # latin1
+    ],
+)
+def test_compatURL_username(url, expected, make_url):
+    assert make_url(url).username == expected
+
+
+def test_compatURL_raw_username(make_url):
+    assert make_url("http://a%20user:p@example.org/").raw_username == "a%20user"
+
+
+@pytest.mark.parametrize(
+    "url, expected",
+    [
+        ("http://user@example.org/", None),
+        ("http://user:pass%20word@example.org/", "pass word"),
+    ],
+)
+def test_compatURL_password(url, expected, make_url):
+    assert make_url(url).password == expected
+
+
+def test_compatURL_raw_password(make_url):
+    assert make_url("http://u:pass%20word@example.org/").raw_password == "pass%20word"
+
+
+def test_compatURL_decode_query(make_url):
+    args = make_url("?a=b&c").decode_query()
+    assert args["a"] == "b"
+    assert args["c"] == ""
+
+
+def test_compatURL_join(make_url):
+    base = make_url("http://example.org/top/index.html")
+    assert str(base.join("sibling.html")) == "http://example.org/top/sibling.html"
+
+
+def test_compatURL_join_tuple(make_url):
+    base = make_url("http://example.org/top/index.html")
+    url = ("", "", "sibling.html", "q", "f")
+    assert str(base.join(url)) == "http://example.org/top/sibling.html?q#f"
+
+
+def test_compatURL_to_url(make_url):
+    strval = "http://example.org/foo?q#a"
+    url = make_url(strval)
+    assert url.to_url() == strval
+    assert str(url) == strval
+
+
+@pytest.mark.filterwarnings("ignore:(?i).*werkzeug:DeprecationWarning")
+def test_compatURL_to_uri_tuple(make_url):
+    uri_tuple = make_url("http://example.org/fü/").to_uri_tuple()
+    assert uri_tuple == ("http", "example.org", "/f%C3%BC/", "", "")
+
+
+@pytest.mark.filterwarnings("ignore:(?i).*werkzeug:DeprecationWarning")
+def test_compatURL_to_iri_tuple(make_url):
+    iri_tuple = make_url("http://example.org/f%C3%BC/").to_iri_tuple()
+    assert iri_tuple == ("http", "example.org", "/fü/", "", "")
diff --git a/tests/test_config.py b/tests/test_config.py
index 86d24d9..4dc843c 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -1,3 +1,50 @@
+import inspect
+
+import pytest
+
+from lektor.environment.config import Config
+
+
 def test_custom_attachment_types(env):
     attachment_types = env.load_config().values["ATTACHMENT_TYPES"]
     assert attachment_types[".foo"] == "text"
+
+
+@pytest.fixture(scope="function")
+def config(tmp_path, project_url):
+    projectfile = tmp_path / "scratch.lektorproject"
+    projectfile.write_text(
+        inspect.cleandoc(
+            f"""
+            [project]
+            url = {project_url}
+            """
+        )
+    )
+    return Config(projectfile)
+
+
+@pytest.mark.parametrize(
+    "project_url, expected",
+    [
+        ("", None),
+        ("/path/", None),
+        ("https://example.org", "https://example.org/"),
+    ],
+)
+def test_base_url(config, expected):
+    assert config.base_url == expected
+
+
+@pytest.mark.parametrize(
+    "project_url, expected",
+    [
+        ("", "/"),
+        ("/path", "/path/"),
+        ("/path/", "/path/"),
+        ("https://example.org", "/"),
+        ("https://example.org/pth", "/pth/"),
+    ],
+)
+def test_base_path(config, expected):
+    assert config.base_path == expected
diff --git a/tests/test_db.py b/tests/test_db.py
index 716cad8..ae8049d 100644
--- a/tests/test_db.py
+++ b/tests/test_db.py
@@ -536,3 +536,30 @@ def test_Record_contents_is_deprecated(pad, path):
     with pytest.deprecated_call(match=r"contents") as warnings:
         assert isinstance(pad.get(path).contents, FileContents)
     assert all(w.filename == __file__ for w in warnings)
+
+
+@pytest.mark.parametrize(
+    "url, base_url, absolute, external, project_url, expected",
+    [
+        ("/a/b.html", "/a/", None, None, None, "b.html"),
+        ("/a/b/", "/a/", None, None, None, "b/"),
+        ("/a/b/", "/a", None, None, None, "a/b/"),
+        ("/a/b/", "/a", True, None, None, "/a/b/"),
+        ("/a/b/", "/a", True, None, "https://example.net/pfx/", "/pfx/a/b/"),
+        ("/a/b/", "/a", None, True, "https://example.org", "https://example.org/a/b/"),
+    ],
+)
+def test_Pad_make_url(url, base_url, absolute, external, project_url, expected, pad):
+    if project_url is not None:
+        pad.db.config.values["PROJECT"]["url"] = project_url
+    assert pad.make_url(url, base_url, absolute, external) == expected
+
+
+def test_Pad_make_url_raises_runtime_error_if_no_project_url(pad):
+    with pytest.raises(RuntimeError, match="(?i)configure the url in the project"):
+        pad.make_url("/a/b", external=True)
+
+
+def test_Pad_make_url_raises_runtime_error_if_no_base_url(pad):
+    with pytest.raises(RuntimeError, match="(?i)no base url"):
+        pad.make_url("/a/b")
diff --git a/tests/test_deploy.py b/tests/test_deploy.py
index 3627abb..3b8d9d1 100644
--- a/tests/test_deploy.py
+++ b/tests/test_deploy.py
@@ -2,7 +2,6 @@ from os.path import dirname
 from os.path import join
 
 import pytest
-from werkzeug.urls import url_parse
 
 from lektor.publisher import RsyncPublisher
 
@@ -18,7 +17,7 @@ def test_get_server(env):
 def test_rsync_command_credentials(tmpdir, mocker, env):
     output_path = tmpdir.mkdir("output")
     publisher = RsyncPublisher(env, str(output_path))
-    target_url = url_parse("http://example.com")
+    target_url = "http://example.com"
     credentials = {
         "username": "fakeuser",
         "password": "fakepass",
@@ -159,7 +158,6 @@ output_path = join(dirname(__file__), "OUTPUT_PATH")
 )
 def test_rsync_publisher(target_url, called_command, tmpdir, mocker, env):
     publisher = RsyncPublisher(env, str(output_path))
-    target_url = url_parse(target_url)
     mock_popen = mocker.patch("lektor.publisher.portable_popen")
     with publisher.get_command(target_url, credentials=None):
         assert mock_popen.called
diff --git a/tests/test_publisher.py b/tests/test_publisher.py
index 2f51068..84ff33d 100644
--- a/tests/test_publisher.py
+++ b/tests/test_publisher.py
@@ -15,14 +15,15 @@ from subprocess import PIPE
 from subprocess import run
 
 import pytest
-from werkzeug.urls import url_parse
 
+from lektor.publisher import _CompatURLStr
 from lektor.publisher import _ssh_command
 from lektor.publisher import _ssh_key_file
 from lektor.publisher import Command
 from lektor.publisher import GithubPagesPublisher
 from lektor.publisher import GitRepo
 from lektor.publisher import publish
+from lektor.publisher import Publisher
 from lektor.publisher import PublishError
 from lektor.utils import locate_executable
 
@@ -470,11 +471,10 @@ def test_GithubPagesPublisher_publish(
     repo.publish_ghpages.return_value = iter(["Published!"])
     GitRepo.return_value.__enter__.return_value = repo
 
-    url = url_parse(target_url)
     with ExitStack() as stack:
         if warns:
             stack.enter_context(pytest.deprecated_call())
-        output = list(ghp_publisher.publish(url))
+        output = list(ghp_publisher.publish(target_url))
 
     GitRepo.assert_called_once_with(output_path)
     if "+https:" in target_url:
@@ -490,9 +490,9 @@ def test_GithubPagesPublisher_publish(
 
 @pytest.mark.usefixtures("no_utils")
 def test_GithubPagesPublisher_publish_fails_if_no_git(ghp_publisher):
-    url = url_parse("ghpages://owner/project")
+    target_url = "ghpages://owner/project"
     with pytest.raises(PublishError) as exc_info:
-        list(ghp_publisher.publish(url))
+        list(ghp_publisher.publish(target_url))
     assert re.search(r"git.*not found", str(exc_info.value))
 
 
@@ -544,15 +544,14 @@ def test_GithubPagesPublisher_publish_fails_if_no_git(ghp_publisher):
     ],
 )
 def test_GithubPagesPublisher_parse_url(ghp_publisher, target_url, expected):
-    url = url_parse(target_url)
-    assert ghp_publisher._parse_url(url) == expected
+    assert ghp_publisher._parse_url(target_url) == expected
 
 
 def test_GithubPagesPublisher_parse_url_warns_on_default_master_branch(ghp_publisher):
-    url = url_parse("ghpages://owner/owner.github.io")
+    target_url = "ghpages://owner/owner.github.io"
     with pytest.deprecated_call():
         push_url, branch, cname, preserve_history, warnings = ghp_publisher._parse_url(
-            url
+            target_url
         )
     assert (push_url, branch, cname, preserve_history) == (
         "ssh://git@github.com/owner/owner.github.io.git",
@@ -574,9 +573,8 @@ def test_GithubPagesPublisher_parse_url_warns_on_default_master_branch(ghp_publi
     ],
 )
 def test_GithubPagesPublisher_parse_url_failures(ghp_publisher, target_url, expected):
-    url = url_parse(target_url)
     with pytest.raises(PublishError) as exc_info:
-        ghp_publisher._parse_url(url)
+        ghp_publisher._parse_url(target_url)
     assert expected in str(exc_info.value)
 
 
@@ -606,8 +604,7 @@ def test_GithubPagesPublisher_parse_url_failures(ghp_publisher, target_url, expe
 def test_GithubPagesPublisher_parse_credentials(
     ghp_publisher, credentials, target_url, expected
 ):
-    url = url_parse(target_url)
-    assert ghp_publisher._parse_credentials(credentials, url) == expected
+    assert ghp_publisher._parse_credentials(credentials, target_url) == expected
 
 
 def test_publish(env, output_path, mocker):
@@ -615,10 +612,10 @@ def test_publish(env, output_path, mocker):
     env.add_publisher("publishtest", Publisher)
     credentials = {"foo": "bar"}
     rv = publish(env, "publishtest://host/path", output_path, credentials)
-    url = url_parse("publishtest://host/path")
+    target_url = "publishtest://host/path"
     assert Publisher.mock_calls == [
         mocker.call(env, output_path),
-        mocker.call().publish(url, credentials),
+        mocker.call().publish(target_url, credentials),
     ]
     assert rv is Publisher.return_value.publish.return_value
 
@@ -627,3 +624,37 @@ def test_publish_unknown_scheme(env, output_path):
     with pytest.raises(PublishError) as exc_info:
         publish(env, "unknownscheme://host/path", output_path)
     assert "unknown scheme" in str(exc_info.value)
+
+
+def test_publish_old_plugin(env, output_path, mocker):
+    target_url = "publishtest://Bücher.example/?foo=bar"
+
+    class OldPublisher(Publisher):
+        # pylint: disable=signature-differs
+        def publish(self, target_url, credentials, **extra):
+            return target_url
+
+    env.add_publisher("publishtest", OldPublisher)
+    compat_url = publish(env, target_url, output_path, credentials={})
+
+    assert isinstance(compat_url, _CompatURLStr)
+
+
+def test_CompatURLStr_is_str():
+    url = _CompatURLStr(" scheme://example.org/ ")
+    assert isinstance(url, str)
+    assert str(url) == " scheme://example.org/ "
+
+
+def test_CompatURLStr_raises_attribute_error():
+    url = _CompatURLStr("scheme://example.org/")
+    with pytest.raises(AttributeError):
+        url._replace()
+
+
+def test_CompatURLStr_works_as_URL():
+    url = _CompatURLStr("scheme:///path?foo=bar")
+    with pytest.deprecated_call() as w:
+        warnings.filterwarnings("ignore", "'werkzeug", DeprecationWarning)
+        assert dict(url.decode_query()) == {"foo": "bar"}
+    assert w[0].filename == __file__
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 96b668e..21eb4de 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,6 +1,8 @@
 import warnings
 from contextlib import contextmanager
+from dataclasses import dataclass
 from itertools import islice
+from urllib.parse import urlsplit
 
 import pytest
 
@@ -11,8 +13,10 @@ from lektor.utils import join_path
 from lektor.utils import magic_split_ext
 from lektor.utils import make_relative_url
 from lektor.utils import parse_path
+from lektor.utils import secure_url
 from lektor.utils import slugify
 from lektor.utils import unique_everseen
+from lektor.utils import Url
 
 
 def test_join_path():
@@ -75,6 +79,102 @@ def test_slugify():
     assert slugify("slashed/slug") == "slashed/slug"
 
 
+def test_Url_constructor_deprecated():
+    with pytest.deprecated_call():
+        Url("https://example.org")
+
+
+@dataclass
+class SampleUrl:
+    uri: str
+    iri: str
+
+    @property
+    def split_uri(self):
+        return urlsplit(self.uri)
+
+    @property
+    def split_iri(self):
+        return urlsplit(self.iri)
+
+
+SAMPLE_URLS = [
+    SampleUrl("https://example.org/foo", "https://example.org/foo"),
+    SampleUrl("https://example.org:8001/f%C3%BC", "https://example.org:8001/fü"),
+    SampleUrl(
+        "https://xn--wgv71a119e.idn.icann.org/%E5%A4%A7",
+        "https://日本語.idn.icann.org/大",
+    ),
+    SampleUrl("/?q=sch%C3%B6n#gru%C3%9F", "/?q=schön#gruß"),
+]
+
+
+@pytest.fixture(params=SAMPLE_URLS, ids=lambda sample: sample.uri)
+def sample_url(request):
+    sample_url = request.param
+    # sanity checks
+    assert sample_url.split_uri.scheme == sample_url.split_iri.scheme
+    assert sample_url.split_uri.port == sample_url.split_iri.port
+    return sample_url
+
+
+def test_Url_str(sample_url):
+    assert str(Url.from_string(sample_url.iri)) == sample_url.iri
+    assert str(Url.from_string(sample_url.uri)) == sample_url.uri
+
+
+def test_Url_ascii_url(sample_url):
+    assert Url.from_string(sample_url.iri).ascii_url == sample_url.uri
+    assert Url.from_string(sample_url.uri).ascii_url == sample_url.uri
+
+
+def test_Url_ascii_host(sample_url):
+    assert Url.from_string(sample_url.iri).ascii_host == sample_url.split_uri.hostname
+    assert Url.from_string(sample_url.uri).ascii_host == sample_url.split_uri.hostname
+
+
+def test_Url_scheme(sample_url):
+    assert Url.from_string(sample_url.iri).scheme == sample_url.split_uri.scheme
+    assert Url.from_string(sample_url.uri).scheme == sample_url.split_uri.scheme
+
+
+def test_Url_host(sample_url):
+    assert Url.from_string(sample_url.iri).host == sample_url.split_iri.hostname
+    assert Url.from_string(sample_url.uri).host == sample_url.split_iri.hostname
+
+
+def test_Url_port(sample_url):
+    assert Url.from_string(sample_url.iri).port == sample_url.split_uri.port
+    assert Url.from_string(sample_url.uri).port == sample_url.split_uri.port
+
+
+def test_Url_path(sample_url):
+    assert Url.from_string(sample_url.iri).path == sample_url.split_iri.path
+    assert Url.from_string(sample_url.uri).path == sample_url.split_iri.path
+
+
+def test_Url_query(sample_url):
+    assert Url.from_string(sample_url.iri).query == sample_url.split_iri.query
+    assert Url.from_string(sample_url.uri).query == sample_url.split_iri.query
+
+
+def test_Url_anchor(sample_url):
+    assert Url.from_string(sample_url.iri).anchor == sample_url.split_iri.fragment
+    assert Url.from_string(sample_url.uri).anchor == sample_url.split_iri.fragment
+
+
+@pytest.mark.parametrize(
+    "url, expected",
+    [
+        ("https://user:pw@example.org/p", "https://user@example.org/p"),
+        ("https://user:pw@example.org:8000", "https://user@example.org:8000"),
+        ("https://user@example.org/b", "https://user@example.org/b"),
+    ],
+)
+def test_secure_url(url, expected):
+    assert secure_url(url) == expected
+
+
 def test_url_builder():
 
     assert build_url([]) == "/"