Repository: matrix-org/synapse

Issue:
Consecutive remote media thumbnail requests with different methods fails
# Description 

It seems like the method column is missing on the unique constraint on `remote_media_cache_thumbnails`, causing internal server errors when trying to fetch the same size of the same thumbnail with different methods.

### Steps to reproduce

- Fetch a thumbnail of a remote image using some specific size and method (scale or crop).

- Fetch a thumbnail of the same image and same size, but specify a different method.

- Receive M_UNKNOWN, Internal server error

Here's the error from server logs:

```
2020-10-22 16:04:50,858 - synapse.http.server - 85 - ERROR - GET-1786524 - Failed handle request via 'ThumbnailResource': <XForwardedForRequest at 0x7fc446756320 method='GET'
 uri='/_matrix/media/r0/thumbnail/remote-server.tld/XXXXXXX_removed_file_id?width=44&height=44&method=crop' clientproto='HTTP/1.0' site=8008>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/server.py", line 230, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/server.py", line 258, in _async_render
    callback_return = await raw_callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/rest/media/v1/thumbnail_resource.py", line 67, in _async_render_GET
    request, server_name, media_id, width, height, method, m_type
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/rest/media/v1/thumbnail_resource.py", line 232, in _select_or_generate_remote_thumbnail
    desired_type,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/rest/media/v1/media_repository.py", line 597, in generate_remote_exact_thumbnail
    t_len,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/media_repository.py", line 374, in store_remote_media_thumbnail
    desc="store_remote_media_thumbnail",
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 727, in simple_insert
    await self.runInteraction(desc, self.simple_insert_txn, table, values)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 577, in runInteraction
    **kwargs
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 647, in runWithConnection
    self._db_pool.runWithConnection(inner_func, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/compat.py", line 464, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 641, in inner_func
    return func(conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 447, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 748, in simple_insert_txn
    txn.execute(sql, vals)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 212, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 238, in _do_execute
    return func(sql, *args)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"
DETAIL:  Key (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type)=(remote-server.tld, XXXXXXX_removed_file_id, 44, 44, image/png) already e
xists.
```

Added comments from tulir:

"someone forgot the `thumbnail_method` from the unique constraint"

"wait no, there are two unique constraints, one with and one without `thumbnail_method`? :D"

```
synapse=# \d remote_media_cache_thumbnails
        Table "public.remote_media_cache_thumbnails"
      Column      |  Type   | Collation | Nullable | Default 
------------------+---------+-----------+----------+---------
 media_origin     | text    |           |          | 
 media_id         | text    |           |          | 
 thumbnail_width  | integer |           |          | 
 thumbnail_height | integer |           |          | 
 thumbnail_method | text    |           |          | 
 thumbnail_type   | text    |           |          | 
 thumbnail_length | integer |           |          | 
 filesystem_id    | text    |           |          | 
Indexes:
    "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key" UNIQUE CONSTRAINT, btree (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type)
    "remote_media_repository_thumbn_media_origin_id_width_height_met" UNIQUE, btree (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method)
```

### Version information

- **Homeserver**: hacklab.fi

- **Version**: 1.21.2



Discussion:
looks like it didn't get correctly removed in #7124
https://github.com/matrix-org/synapse/pull/7124/files#diff-f05c2d0a285e2b60b93cbffe6283864c9b23c19984fe7e4511604797413acc06R83 tries to remove the old constraint but doesn't take into account the fact that (I think) the constraint might have a different name under postgres and sqlite.
@deepbluev7 any chance you'd be willing to take a look at this?
@clokep I think removing both of the old constraints instead should work in that case? I had no idea, that the constraints could have different names and I still don't understand, why they would. I'm a bit busy currently though, so it may be easier for someone else to add a migration, that deletes both of the old indices, if they exist, since it will take me quite a while to context switch into synapse db migration code again.
I suspect they can end up with different names depending on whether you're using sqlite or postgres?
Sorry to bringing this up, but has anyone found any workaround for this?? I have millions of lines on synapse's log file in the last month.
I think the thing to do here is check what the name of the old constraint is on sqlite and add a delta that removes that (similar to what was done in #7124).
From the Polynomial Supporters channel someone else and me noticed this problem today as well - the thumbnails for the flairs are missing and represented with the alt text instead.
When checking the synapse logs I could find this:
```
2021-04-01 12:36:19,519 - synapse.storage.database - 957 - WARNING - GET-136585 - IntegrityError when upserting into remote_media_cache_thumbnails; retrying: duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"
DETAIL:  Key (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type)=(matrix.org, pKRnQRliCyKRqHecxWpPBohl, 16, 16, image/png) already exists.
2021-04-01 12:36:19,533 - synapse.http.server - 94 - ERROR - GET-136585 - Failed handle request via 'ThumbnailResource': <XForwardedForRequest at 0x7f8502a06670 method='GET' uri='/_matrix/media/r0/thumbnail/matrix.org/pKRnQRliCyKRqHecxWpPBohl?width=16&height=16&method=crop' clientproto='HTTP/1.1' site='8008'>
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/rest/media/v1/thumbnail_resource.py", line 80, in _async_render_GET
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/rest/media/v1/thumbnail_resource.py", line 232, in _select_or_generate_remote_thumbnail
    file_path = await self.media_repo.generate_remote_exact_thumbnail(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/rest/media/v1/media_repository.py", line 632, in generate_remote_exact_thumbnail
    await self.store.store_remote_media_thumbnail(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/databases/main/media_repository.py", line 536, in store_remote_media_thumbnail
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"
DETAIL:  Key (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type)=(matrix.org, pKRnQRliCyKRqHecxWpPBohl, 16, 16, image/png) already exists.
2021-04-01 12:36:19,535 - synapse.access.http.8008 - 308 - INFO - GET-136585 - 2003:e9:271c:ae00:c4ba:ee0a:f3b2:1e33 - 8008 - {None} Processed request: 0.044sec/-0.000sec (0.006sec, 0.002sec) (0.003sec/0.019sec/7) 55B 500 "GET /_matrix/media/r0/thumbnail/matrix.org/pKRnQRliCyKRqHecxWpPBohl?width=16&height=16&method=crop HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" [0 dbevts]
```
This is on Synapse 1.30.1.  
Are there ways for me to manually fix this?
Can I help troubleshoot this by offering logs and reproduce things?
ugh this is one of those bugs that is one line to fix. I'll have a go at it now.

Resolving patch:
diff --git a/synapse/storage/databases/main/media_repository.py b/synapse/storage/databases/main/media_repository.py
--- a/synapse/storage/databases/main/media_repository.py
+++ b/synapse/storage/databases/main/media_repository.py
@@ -22,6 +22,9 @@
 BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD = (
     "media_repository_drop_index_wo_method"
 )
+BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD_2 = (
+    "media_repository_drop_index_wo_method_2"
+)
 
 
 class MediaSortOrder(Enum):
@@ -85,23 +88,35 @@ def __init__(self, database: DatabasePool, db_conn, hs):
             unique=True,
         )
 
+        # the original impl of _drop_media_index_without_method was broken (see
+        # https://github.com/matrix-org/synapse/issues/8649), so we replace the original
+        # impl with a no-op and run the fixed migration as
+        # media_repository_drop_index_wo_method_2.
+        self.db_pool.updates.register_noop_background_update(
+            BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD
+        )
         self.db_pool.updates.register_background_update_handler(
-            BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD,
+            BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD_2,
             self._drop_media_index_without_method,
         )
 
     async def _drop_media_index_without_method(self, progress, batch_size):
+        """background update handler which removes the old constraints.
+
+        Note that this is only run on postgres.
+        """
+
         def f(txn):
             txn.execute(
                 "ALTER TABLE local_media_repository_thumbnails DROP CONSTRAINT IF EXISTS local_media_repository_thumbn_media_id_thumbnail_width_thum_key"
             )
             txn.execute(
-                "ALTER TABLE remote_media_cache_thumbnails DROP CONSTRAINT IF EXISTS remote_media_repository_thumbn_media_id_thumbnail_width_thum_key"
+                "ALTER TABLE remote_media_cache_thumbnails DROP CONSTRAINT IF EXISTS remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"
             )
 
         await self.db_pool.runInteraction("drop_media_indices_without_method", f)
         await self.db_pool.updates._end_background_update(
-            BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD
+            BG_UPDATE_REMOVE_MEDIA_REPO_INDEX_WITHOUT_METHOD_2
         )
         return 1
 


Test patch: