Repository: matrix-org/synapse

Issue:
Synapse doesn't detect URL preview dependencies if built with poetry-core 1.3.0, raising runtime errors
This is causing CI failures:

> Synapse 1.69.0rc1 does not provide the feature 'url_preview'

even though the requisite dependencies are installed. The problem is that https://github.com/python-poetry/poetry-core/pull/476 now enforces https://peps.python.org/pep-0685/#specification. Wheels built by poetry-core include metadata describing an extra `url-preview` with a `-` but NOT `url_preview` with an `_`. This means the dependency-checking code in Synapse here 

https://github.com/matrix-org/synapse/blob/7b67e93d499cb45f7217e9dfea046ed8b5c455fd/synapse/config/repository.py#L206-L208

will fail. 

People who've also installed from source after the release of poetry-core 1.3.0 are going to be seeing this.

Assuming we want to be able to use future poetry-core releases, we'll need to update the dependency-checking code to account for this.


Discussion:
#14080 should avoid the problem by pinning the build-system requirements. Leaving this issue open though in case we _do_ want to use newer versions of poetry-core in the future.
This just turns a build failure because of one problem into a build failure because the poetry version is wrong.  It doesn't really fix the problem.

Is 1.2.1 safe?   I don't see a 1.3.0 release on poetry github, so I'm a little confused.   I don't understand requiring 1.2.0 exactly if there is 1.2.1 which should be a micro with bugfixes.   And a change like new rules in a micro doesn't make sense either.
> I don't see a 1.3.0 release on poetry github

I'm talking about poetry-core here (latest: 1.3.1), not poetry (latest: 1.2.1).
Sorry, missed that.   poetry-core in pkgsrc is today at 1.0.8.   Same comment about should probably be updated, but hasn't, and that if you need to avoid 1.3, no need to also require >= 1.2.0 because of that.
Are you certain that pkgsrc's infrastructure will use its version poetry-core (1.0.8) to build Synapse? Whenever I have build synapse locally with `poetry build`, the output suggests that both tools fetch the latest version of poetry-core matching the requirements specified in the build system.

```
$ poetry build
Preparing build environment with build-system requirements poetry-core>=1.0.0, setuptools_rust>=1.3
Building matrix-synapse (1.69.0rc1)

[...]

^C

$ strace poetry build 2>&1 | grep -E 'poetry[-_]core'
openat(AT_FDCWD, "/home/dmr/.local/pipx/venvs/poetry/lib64/python3.10/site-packages/poetry_core-1.2.0.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/dmr/.local/pipx/venvs/poetry/lib64/python3.10/site-packages/poetry_core-1.2.0.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/dmr/.local/pipx/venvs/poetry/lib64/python3.10/site-packages/poetry_core-1.2.0.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(1, "Preparing build environment with"..., 99Preparing build environment with build-system requirements poetry-core>=1.0.0, setuptools_rust>=1.3) = 99
read(3, "Collecting poetry-core>=1.0.0\n", 8192) = 30
read(3, "  Using cached poetry_core-1.3.1"..., 8162) = 59

[...]
```

We see problems related to this when installing older versions of poetry itself in https://github.com/matrix-org/synapse/issues/13849.
I am not certain, but if so tools fetching things over the net at runtime is a bug that needs fixing, from the packaging viewpoint.
Do you know if this is packaged in guix?  They are very hard-core on reproducible builds, and I admire them for that.

We also have a fake homedir and warn if anything is left in that.  work/.home is empty after the build.

Re #13849 My quick reaction is that poetry is too unstable and complicated to depend on, but I realize all build tools have issues and it's not an easy decision....
Cross-posting from private communication:

> I had assumed that the done thing was to fetch the latest version of the build-time dependencies, because that's what `python -m build` and `poetry build` seem to do, and I wasn't aware of any poetry/python/pep517-specific machinery that packagers were using to prevent this.
>
>I don't like any of that fetching at build time either---hence why I wanted a specific pin. I can see that an upper bound also achieves that without screwing others over though; it just didn't occur to me.
>
> I'll make sure we have something more flexible in place for the 1.69 release proper---whether that's a range like poetry-core>=1.0,<1.3 or application fixes to work with poetry-core 1.3.x.

Resolving patch:
diff --git a/synapse/config/repository.py b/synapse/config/repository.py
--- a/synapse/config/repository.py
+++ b/synapse/config/repository.py
@@ -205,7 +205,7 @@ def read_config(self, config: JsonDict, **kwargs: Any) -> None:
         )
         self.url_preview_enabled = config.get("url_preview_enabled", False)
         if self.url_preview_enabled:
-            check_requirements("url_preview")
+            check_requirements("url-preview")
 
             proxy_env = getproxies_environment()
             if "url_preview_ip_range_blacklist" not in config:


Test patch: