Repository: matrix-org/synapse

Issue:
Error doing update: AttributeError: 'str' object has no attribute 'get'
<!--

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

When starting synapse errors like this are spammed all over the log:
```
2020-07-05 10:59:39,226 - synapse.storage.background_updates - 227 - INFO - background_updates-0 - Starting update batch on background update 'insert_room_retention'
2020-07-05 10:59:39,232 - synapse.storage.background_updates - 114 - ERROR - background_updates-0 - Error doing update
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/background_updates.py", line 111, in run_background_updates
    self.BACKGROUND_UPDATE_DURATION_MS
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/background_updates.py", line 222, in do_next_background_update
    await self._do_background_update(desired_duration_ms)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/background_updates.py", line 255, in _do_background_update
    items_updated = await update_handler(progress, batch_size)
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/room.py", line 938, in _background_insert_retention
    "insert_room_retention", _background_insert_retention_txn,
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/database.py", line 527, in runInteraction
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/database.py", line 575, in runWithConnection
    self._db_pool.runWithConnection(inner_func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/usr/local/lib/python3.7/site-packages/twisted/python/compat.py", line 464, in reraise
    raise exception.with_traceback(traceback)
  File "/usr/local/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/database.py", line 572, in inner_func
    return func(conn, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/database.py", line 418, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/room.py", line 921, in _background_insert_retention_txn
    "min_lifetime": retention_policy.get("min_lifetime"),
AttributeError: 'str' object has no attribute 'get'
```

I found that [here](https://github.com/matrix-org/synapse/blob/5cdca53aa07f921029cb8027693095d150c37e32/synapse/storage/data_stores/main/room.py#L885) `json.dumps(ev["content"])` is called which will return a string and thus `retention_policy.get("min_lifetime")` results in an error.

### Steps to reproduce

- start synapse

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: systemtest.tk

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: Synapse 1.15.2, python 3.7.8

- **Install method**: docker
<!-- examples: package manager/git clone/pip  -->

- **Platform**: docker on Debian Buster
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
It can be fixed by replacing `retention_policy = json.dumps(ev["content"])` on line 885 with `retention_policy = ev["content"]`
Hey and thanks for your report and investigating! Yes it looks like your diagnostic is right, I'm not entirely sure what happened there or why it went unnoticed for such a long time.

Resolving patch:
diff --git a/synapse/storage/data_stores/main/room.py b/synapse/storage/data_stores/main/room.py
--- a/synapse/storage/data_stores/main/room.py
+++ b/synapse/storage/data_stores/main/room.py
@@ -916,7 +916,7 @@ def _background_insert_retention_txn(txn):
                     retention_policy = {}
                 else:
                     ev = json.loads(row["json"])
-                    retention_policy = json.dumps(ev["content"])
+                    retention_policy = ev["content"]
 
                 self.db.simple_insert_txn(
                     txn=txn,


Test patch: