Repository: matrix-org/synapse

Issue:
Remote dynamic thumbnails are stored at the wrong location
<!--

**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**:
You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)


This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

The remote thumbnails path and filename are derived from `media_id` when they're generated and stored with `dynamic_thumbnails` set to `true`. The said thumbnails are looked up by `filesystem_id` which fails every time. Synapse uses a lot of CPU regenerating the thumbnails all the time. When `dynamic_thumbnails` option is then turned off Synapse can't find the thumbnails generated when the said option was turned on and just returns 404.

### Steps to reproduce

- set `dynamic_thumbnails` to `true`
- try to load a thumbnail of a remote media
- now set `dynamic_thumbnails` to `false`
- try to load the same thumbnail of the remote media
- it returns 404

Local media thumbnails are generated and retrieved just fine. For example, I'm trying to retrieve a media with id `zOUYnsGQKLQypwBoQkRpnkXh` and corresponding filesystem id `WFzEWtlvXlCaZShcaIthgzrG`. The media itself is stored at `/var/lib/matrix-synapse/media/bc/remote_content/matrix.org/WF/zE/WtlvXlCaZShcaIthgzrG` and I can retrieve it from my server. The thumbnail should be located at `/var/lib/matrix-synapse/media/bc/remote_thumbnail/matrix.org/WF/zE/WtlvXlCaZShcaIthgzrG/30-30-image-png` but it's actually stored at `/var/lib/matrix-synapse/media/bc/remote_thumbnail/matrix.org/zO/UY/nsGQKLQypwBoQkRpnkXh/30-30-image-png`. Why do we even need two identifiers? It confuses things pretty much (it's hard to find an offending file in the storage if you know its id without looking it up in the database) and doesn't add any security.

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: own server

<!--
What version of Synapse is running?
You can find the Synapse version by inspecting the server headers (replace matrix.org with
your own homeserver domain):
$ curl -v https://matrix.org/_matrix/client/versions 2>&1 | grep "Server:"
-->
- **Version**: 1.3.1+buster1

- **Install method**: apt repository
<!-- examples: package manager/git clone/pip  -->

- **Platform**: Debian stable amd64
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
This seems to fix it:
```diff
diff --git a/synapse/rest/media/v1/media_repository.py b/synapse/rest/media/v1/media_repository.py
index cf5759e9a6..28073dec4c 100644
--- a/synapse/rest/media/v1/media_repository.py
+++ b/synapse/rest/media/v1/media_repository.py
@@ -526,7 +526,7 @@ def generate_remote_exact_thumbnail(
             try:
                 file_info = FileInfo(
                     server_name=server_name,
-                    file_id=media_id,
+                    file_id=file_id,
                     thumbnail=True,
                     thumbnail_width=t_width,
                     thumbnail_height=t_height,
thanks @rkfg. Do you know if this is a recent regression, or if it has always always been a problem?

Would you be able to make a PR for your patch? (A regression test would be very helpful too, though I appreciate that may need more time than you have.)
Eh, no idea if it's recent but I noticed that thumbnails didn't load after I cleaned up the cache and database. Didn't have time to dig deeper back then and enabling dynamic thumbnails "fixed" it. But recently I noticed that avatars take too much time to load like 1-1.5 **minutes** (if they were not cached) so I decided to take a look. I suspected S3 being slow at first but then noticed a big CPU usage and in the end solved it.

I'd like to make a PR but it needs too much hassle for a one-liner so I'm not really excited to do that. Feel free to make that patch yourself, I don't even need to be mentioned. The fix is too trivial. Hope it will help you reduce the load on matrix.org.
> I'd like to make a PR but it needs too much hassle for a one-liner so I'm not really excited to do that. 

understood, but that means it has to wait for someone else to have time to do it :(

Resolving patch:
diff --git a/synapse/rest/media/v1/media_repository.py b/synapse/rest/media/v1/media_repository.py
--- a/synapse/rest/media/v1/media_repository.py
+++ b/synapse/rest/media/v1/media_repository.py
@@ -526,7 +526,7 @@ def generate_remote_exact_thumbnail(
             try:
                 file_info = FileInfo(
                     server_name=server_name,
-                    file_id=media_id,
+                    file_id=file_id,
                     thumbnail=True,
                     thumbnail_width=t_width,
                     thumbnail_height=t_height,


Test patch: