Repository: lektor/lektor (python)

Issue:
Remove werkzeug pin to versions < 2.4
There is a security issue in Werkzeug < 3.0.1 which triggers warnings for security checks.
Reading about the security issue, it is not relevant for the static pages. I did not check, whether it would be relevant for the dynamic editing frontend: https://github.com/advisories/GHSA-hrfv-mqp8-q5rw

The pull request that introduced the current pin of werkzeug<2.4 gives a good overview of why the Pin was needed and how big the effect on Plugins would be: https://github.com/lektor/lektor/pull/1144

However it also mentions another PR that addresses this and that PR has since been merged: https://github.com/lektor/lektor/pull/1143

I suspect that a werkzeug<2.4 isn't necessary any more but I haven't checked whether werkzeug 3 brought other changes that might require incorporation.

Pull request description:
Remove pin on werkzeug <3.0


<!---
Remember to please check that your code passes tests locally. Test with the Makefile
commands `make test`, or if you only need to test just Python / JS `make test-python`
or `make test-js`.
--->

### Issue(s) Resolved

<!--- List the issue(s) below, in the form "Fixes #1234"; one per line --->

Fixes #1171

### Related Issues / Links

<!---
Are there any similar or related issues or pull requests?
Did you make a pull request to update the docs?
--->

### Description of Changes

The latest version of Werkzeug (3.0.1) includes changes to address a vulnerability which could potential be used for DoS attacks.  Though it is not clear that that particular vulnerability is really an issue in common usage of Lektor, removing the `werkzeug<3` pin thus addresses that vulnerability.

### Issues

Should an upper pin to prevent unexpected breakage by new releases of Werkzeug be reinstated?

Both the Flask and Werkzeug projects do seem to zealously deprecate, rename, and remove certain APIs leading to breakage on some minor version bumps, even.  (E.g. see #911, #1018, #1051, #1142 as well as [this blog post](https://blog.miguelgrinberg.com/post/we-have-to-talk-about-flask) and [followup](https://blog.miguelgrinberg.com/post/some-more-to-talk-about-flask).)

Should we add a pin on `werkzeug<3.1` or `werkzeug<4`?

<!--- Explain what you've done and why --->

<!--- Thanks for your help making Lektor better for everyone! --->


Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 94f90fd..6b0858a 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -89,22 +89,22 @@ jobs:
       fail-fast: false
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
-        python: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python: ["3.8", "3.9", "3.10", "3.11", "3.12"]
         exclude:
-          - os: "macos-latest"
-            python: "3.8"
           - os: "macos-latest"
             python: "3.9"
           - os: "macos-latest"
             python: "3.10"
-          - os: "windows-latest"
-            python: "3.8"
+          - os: "macos-latest"
+            python: "3.11"
           - os: "windows-latest"
             python: "3.9"
           - os: "windows-latest"
             python: "3.10"
+          - os: "windows-latest"
+            python: "3.11"
         include:
-          - python: "3.11"
+          - python: "3.12"
             install-ffmpeg: true
     steps:
       - uses: actions/checkout@v3
diff --git a/build_frontend.py b/build_frontend.py
index 7c70142..1f95fb1 100644
--- a/build_frontend.py
+++ b/build_frontend.py
@@ -65,16 +65,20 @@ class FrontendBuildHook(BuildHookInterface):
             app.display_info(f"{APP_JS} exists, skipping frontend build")
             return
 
-        npm = shutil.which("npm")
-        if npm is None:
-            app.abort("npm is not available. can not build frontend")
+        try:
+            proc = subprocess.run(
+                "npm -v", capture_output=True, text=True, shell=True, check=True
+            )
+        except subprocess.CalledProcessError as exc:
+            app.abort(f"{exc.cmd!r} failed (is node/npm installed?)")
+        app.display_info(f"found npm version {proc.stdout.strip()}")
 
         frontend = Path(root, FRONTEND)
         if not frontend.is_dir():
             app.abort("frontend source is missing. can not build frontend")
 
         app.display_info("npm install")
-        subprocess.run((npm, "install"), cwd=frontend, check=True)
+        subprocess.run("npm install", cwd=frontend, shell=True, check=True)
         app.display_info("npm run build")
-        subprocess.run((npm, "run", "build"), cwd=frontend, check=True)
+        subprocess.run("npm run build", cwd=frontend, shell=True, check=True)
         app.display_success("built frontend static files")
diff --git a/lektor/cli.py b/lektor/cli.py
index 6ffeebc..171ddf0 100644
--- a/lektor/cli.py
+++ b/lektor/cli.py
@@ -2,6 +2,7 @@
 import os
 import sys
 import warnings
+from importlib import metadata
 from itertools import chain
 
 import click
@@ -13,7 +14,6 @@ from lektor.cli_utils import pass_context
 from lektor.cli_utils import pruneflag
 from lektor.cli_utils import ResolvedPath
 from lektor.cli_utils import validate_language
-from lektor.compat import importlib_metadata as metadata
 from lektor.project import Project
 from lektor.utils import secure_url
 
diff --git a/lektor/compat.py b/lektor/compat.py
index 7df0859..5aed5c4 100644
--- a/lektor/compat.py
+++ b/lektor/compat.py
@@ -1,68 +1,46 @@
 from __future__ import annotations
 
-import os
-import stat
-import sys
+import importlib.metadata
 import tempfile
 import urllib.parse
-from functools import partial
-from itertools import chain
 from typing import Any
 from urllib.parse import urlsplit
+from warnings import warn
 
 from werkzeug import urls as werkzeug_urls
 from werkzeug.datastructures import MultiDict
 
-__all__ = ["TemporaryDirectory", "importlib_metadata", "werkzeug_urls_URL"]
+from lektor.utils import DeprecatedWarning
 
+__all__ = ["werkzeug_urls_URL"]
 
-def _ensure_tree_writeable(path: str) -> None:
-    """Attempt to ensure that all files in the tree rooted at path are writeable."""
-    dirscans = []
 
-    def fix_mode(path, statfunc):
-        try:
-            # paranoia regarding symlink attacks
-            current_mode = statfunc(follow_symlinks=False).st_mode
-            if not stat.S_ISLNK(current_mode):
-                isdir = stat.S_ISDIR(current_mode)
-                fixed_mode = current_mode | (0o700 if isdir else 0o200)
-                if current_mode != fixed_mode:
-                    os.chmod(path, fixed_mode)
-                if isdir:
-                    dirscans.append(os.scandir(path))
-        except FileNotFoundError:
-            pass
-
-    fix_mode(path, partial(os.stat, path))
-    for entry in chain.from_iterable(dirscans):
-        fix_mode(entry.path, entry.stat)
-
-
-class FixedTemporaryDirectory(tempfile.TemporaryDirectory):
-    """A version of tempfile.TemporaryDirectory that works if dir contains read-only files.
-
-    On python < 3.8 under Windows, if any read-only files are created
-    in a TemporaryDirectory, TemporaryDirectory will throw an
-    exception when it tries to remove them on cleanup. See
-    https://bugs.python.org/issue26660
-
-    This can create issues, e.g., with temporary git repositories since
-    git creates read-only files in its object store.
-
-    """
+_DEPRECATED_ATTRS = {
+    "TemporaryDirectory": tempfile.TemporaryDirectory,
+    "importlib_metadata": importlib.metadata,
+}
 
-    def cleanup(self) -> None:
-        _ensure_tree_writeable(self.name)
-        super().cleanup()
 
-
-if sys.version_info >= (3, 8):
-    TemporaryDirectory = tempfile.TemporaryDirectory
-    from importlib import metadata as importlib_metadata
-else:
-    TemporaryDirectory = FixedTemporaryDirectory
-    import importlib_metadata
+def __getattr__(name):
+    try:
+        value = _DEPRECATED_ATTRS.get(name)
+    except KeyError:
+        # pylint: disable=raise-missing-from
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+    if hasattr(value, "__module__"):
+        replacement = f"{value.__module__}.{value.__name__}"
+    else:
+        replacement = f"{value.__name__}"
+    warn(
+        DeprecatedWarning(
+            name=f"lektor.compat.{name}",
+            reason=f"use {replacement} instead",
+            version="3.4.0",
+        ),
+        stacklevel=2,
+    )
+    return value
 
 
 class _CompatURL(urllib.parse.SplitResult):
diff --git a/lektor/markdown/__init__.py b/lektor/markdown/__init__.py
index 0f90f01..f742449 100644
--- a/lektor/markdown/__init__.py
+++ b/lektor/markdown/__init__.py
@@ -1,5 +1,6 @@
 import sys
 import warnings
+from importlib import metadata
 from typing import Any
 from typing import Dict
 from typing import Hashable
@@ -10,7 +11,6 @@ from weakref import ref as weakref
 
 from markupsafe import Markup
 
-from lektor.compat import importlib_metadata as metadata
 from lektor.markdown.controller import ControllerCache
 from lektor.markdown.controller import FieldOptions
 from lektor.markdown.controller import MarkdownController
diff --git a/lektor/markdown/mistune2.py b/lektor/markdown/mistune2.py
index b94a58c..dbbbc97 100644
--- a/lektor/markdown/mistune2.py
+++ b/lektor/markdown/mistune2.py
@@ -1,15 +1,13 @@
 """MarkdownController implementation for mistune 2.x"""
 from __future__ import annotations
 
-import sys
 from importlib import import_module
-from typing import Any
 from typing import Callable
 from typing import ClassVar
-from typing import Dict
 from typing import List
 from typing import Optional
 from typing import Sequence
+from typing import TypedDict
 
 import mistune.util
 
@@ -46,16 +44,10 @@ class ImprovedRenderer(mistune.HTMLRenderer):  # type: ignore[misc]
         return super().image(src, alt, title)
 
 
-if sys.version_info < (3, 8):
-    # No typing.TypedDict → punt
-    ParserConfigDict = Dict[str, Any]
-else:
-    from typing import TypedDict
-
-    class ParserConfigDict(TypedDict, total=False):
-        block: mistune.BlockParser
-        inline: mistune.InlineParser
-        plugins: Sequence[Callable[[mistune.Markdown], None]]
+class ParserConfigDict(TypedDict, total=False):
+    block: mistune.BlockParser
+    inline: mistune.InlineParser
+    plugins: Sequence[Callable[[mistune.Markdown], None]]
 
 
 MistunePlugin = Callable[[mistune.Markdown], None]
diff --git a/lektor/pluginsystem.py b/lektor/pluginsystem.py
index 0118c6b..b46a8ca 100644
--- a/lektor/pluginsystem.py
+++ b/lektor/pluginsystem.py
@@ -4,13 +4,13 @@ import inspect
 import os
 import sys
 import warnings
+from importlib import metadata
 from pathlib import Path
 from typing import Type
 from weakref import ref as weakref
 
 from inifile import IniFile
 
-from lektor.compat import importlib_metadata as metadata
 from lektor.context import get_ctx
 from lektor.utils import process_extra_flags
 
diff --git a/lektor/publisher.py b/lektor/publisher.py
index f7acbed..ca0e658 100644
--- a/lektor/publisher.py
+++ b/lektor/publisher.py
@@ -17,6 +17,7 @@ from subprocess import CompletedProcess
 from subprocess import DEVNULL
 from subprocess import PIPE
 from subprocess import STDOUT
+from tempfile import TemporaryDirectory
 from typing import Any
 from typing import Callable
 from typing import ContextManager
@@ -35,7 +36,6 @@ from warnings import warn
 
 from werkzeug.datastructures import MultiDict
 
-from lektor.compat import TemporaryDirectory
 from lektor.compat import werkzeug_urls_URL
 from lektor.exception import LektorException
 from lektor.utils import bool_from_string
diff --git a/pyproject.toml b/pyproject.toml
index 04fc5b3..3a3a00b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -23,23 +23,22 @@ classifiers = [
     "Operating System :: OS Independent",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.7",
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
     "Topic :: Internet :: WWW/HTTP :: Dynamic Content",
     "Topic :: Software Development :: Libraries :: Python Modules",
 ]
 
-requires-python = ">=3.7"
+requires-python = ">=3.8"
 dependencies = [
     "Babel",
     "click>=6.0",
     "EXIFRead",
     "filetype>=1.0.7",
     "Flask",
-    "importlib_metadata; python_version<'3.8'",
     "inifile>=0.4.1",
     "Jinja2>=3.0",
     "MarkupSafe",
@@ -53,7 +52,7 @@ dependencies = [
     "requests",
     "tzdata; python_version>='3.9' and sys_platform == 'win32'",
     "watchfiles>=0.12",
-    "Werkzeug>=2.1.0,<3",
+    "Werkzeug>=2.1.0",
 ]
 optional-dependencies.ipython = [
     "ipython",
diff --git a/tox.ini b/tox.ini
index 8cd18ad..7c80c0f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -2,7 +2,7 @@
 minversion = 4.1
 envlist =
     lint
-    {py37,py38,py39,py310,py311}{,-mistune0}
+    {py38,py39,py310,py311,py312}{,-mistune0}
     py311-noutils
     py311-pytz
     py311-tzdata
@@ -13,11 +13,11 @@ isolated_build = true
 
 [gh-actions]
 python =
-    3.7: py37, cover
     3.8: py38, cover
     3.9: py39, cover
     3.10: py310, cover
     3.11: py311, cover
+    3.12: py312, cover
 
 [testenv]
 commands =
@@ -39,11 +39,11 @@ deps =
     mistune0: mistune<2
     pytz: pytz
     tzdata: tzdata
-    pillow6: pillow<6.1.0
+    pillow6: pillow<7.0
     pillow7: pillow<7.1.0
 depends =
-    py{37,38,39,310,311}: cover-clean
-    cover-report: py{37,38,39,310,311}{,-mistune0,-noutils,-pytz,-tzdata,-pillow6,-pillow7}
+    py{38,39,310,311,312}: cover-clean
+    cover-report: py{38,39,310,311,312}{,-mistune0,-noutils,-pytz,-tzdata,-pillow6,-pillow7}
 # XXX: I've been experiencing sporadic failures when running tox in parallel mode.
 # The crux of the error messages when this happens appears to be something like:
 #
@@ -63,7 +63,7 @@ depends =
 # Hopefully, at some point this can be removed.
 download = true
 
-[testenv:py{37,38,39,310,311}-noutils]
+[testenv:py{38,39,310,311,312}-noutils]
 # To test in environment without external utitilities like ffmpeg and git installed,
 # break PATH in noutils environment(s).
 allowlist_externals = env


Test patch:
diff --git a/tests/test_build_frontend.py b/tests/test_build_frontend.py
index d0a2b1d..a3b5d4f 100644
--- a/tests/test_build_frontend.py
+++ b/tests/test_build_frontend.py
@@ -1,4 +1,5 @@
 """Tests for the hatch build hooks in ../build_frontend.py."""
+import os
 import shutil
 import sys
 from importlib.util import module_from_spec
@@ -81,8 +82,10 @@ def test_initialize_skips_build_if_output_exists(
 
 
 @pytest.mark.usefixtures("frontend_src")
-def test_initialize_aborts_if_no_npm(frontend_build_hook, mocker):
-    mocker.patch("shutil.which").return_value = None
+def test_initialize_aborts_if_no_npm(
+    frontend_build_hook, frontend_build_module, monkeypatch
+):
+    monkeypatch.setitem(os.environ, "PATH", "")
     with pytest.raises(SystemExit) as exc_info:
         frontend_build_hook.initialize("standard", build_data={})
     assert exc_info.value.args[0] == 1
diff --git a/tests/test_compat.py b/tests/test_compat.py
index b6410aa..e3369f9 100644
--- a/tests/test_compat.py
+++ b/tests/test_compat.py
@@ -1,38 +1,36 @@
-import os
-from pathlib import Path
+import importlib
+import tempfile
 from urllib.parse import urlsplit
 
 import pytest
 from werkzeug import urls as werkzeug_urls
 
 from lektor.compat import _CompatURL
-from lektor.compat import _ensure_tree_writeable
-from lektor.compat import FixedTemporaryDirectory
-from lektor.compat import TemporaryDirectory
 
 
-def test_ensure_tree_writeable(tmp_path):
-    topdir = tmp_path / "topdir"
-    subdir = topdir / "subdir"
-    regfile = subdir / "regfile"
-    subdir.mkdir(parents=True)
-    regfile.touch(mode=0)
-    subdir.chmod(0)
-    topdir.chmod(0)
-
-    _ensure_tree_writeable(topdir)
-
-    for p in topdir, subdir, regfile:
-        assert os.access(p, os.W_OK)
-
-
-@pytest.mark.parametrize("tmpdir_class", [FixedTemporaryDirectory, TemporaryDirectory])
-def test_TemporaryDirectory(tmpdir_class):
-    with tmpdir_class() as tmpdir:
-        file = Path(tmpdir, "test-file")
-        file.touch(mode=0)
-        os.chmod(tmpdir, 0)
-    assert not os.path.exists(tmpdir)
+@pytest.mark.parametrize(
+    "name, expected_value, replacement",
+    [
+        (
+            "TemporaryDirectory",
+            tempfile.TemporaryDirectory,
+            "tempfile.TemporaryDirectory",
+        ),
+        ("importlib_metadata", importlib.metadata, "importlib.metadata"),
+    ],
+)
+def test_deprecated_attr(name, expected_value, replacement):
+    lektor_compat = importlib.import_module("lektor.compat")
+    with pytest.deprecated_call(match=f"use {replacement} instead") as warnings:
+        value = getattr(lektor_compat, name)
+    assert value is expected_value
+    assert warnings[0].filename == __file__
+
+
+def test_missing_attr():
+    lektor_compat = importlib.import_module("lektor.compat")
+    with pytest.raises(AttributeError):
+        lektor_compat.MISSING  # pylint: disable=pointless-statement
 
 
 def make_CompatURL(url: str) -> _CompatURL:
diff --git a/tests/test_conftest.py b/tests/test_conftest.py
index e8f6fdf..0f78910 100644
--- a/tests/test_conftest.py
+++ b/tests/test_conftest.py
@@ -1,8 +1,7 @@
 import sys
 from contextlib import suppress
 from importlib import import_module
-
-from lektor.compat import importlib_metadata
+from importlib import metadata
 
 # pylint: disable-next=wrong-import-order
 from conftest import restore_import_state  # noreorder
@@ -25,15 +24,11 @@ def test_restore_import_state_restores_unneutered_PathFinder():
     #
     # This tests that restore_import_state manages to unneuter
     # this find.
-    distributions_pre = [
-        dist.metadata["name"] for dist in importlib_metadata.distributions()
-    ]
+    distributions_pre = [dist.metadata["name"] for dist in metadata.distributions()]
 
     with restore_import_state(), suppress(ModuleNotFoundError):
         import_module("importlib_metadata")
 
-    distributions_post = [
-        dist.metadata["name"] for dist in importlib_metadata.distributions()
-    ]
+    distributions_post = [dist.metadata["name"] for dist in metadata.distributions()]
     assert len(distributions_pre) == len(distributions_post)
     assert set(distributions_pre) == set(distributions_post)
diff --git a/tests/test_pluginsystem.py b/tests/test_pluginsystem.py
index 3375215..1144c31 100644
--- a/tests/test_pluginsystem.py
+++ b/tests/test_pluginsystem.py
@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import inspect
 import sys
+from importlib import metadata
 from importlib.abc import Loader
 from importlib.machinery import ModuleSpec
 from pathlib import Path
@@ -12,7 +13,6 @@ from unittest import mock
 import pytest
 
 from lektor.cli import cli
-from lektor.compat import importlib_metadata as metadata
 from lektor.context import Context
 from lektor.packages import add_package_to_project
 from lektor.pluginsystem import _check_dist_name