Repository: matrix-org/synapse

Issue:
Workers running under PyPy attempt to load psycopg2 (not psycopg2cffi) and crash
<!--

**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**:
You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)


This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

When attempting to run Synapse 0.99.3 (with PostgreSQL and workers) under PyPy, it somehow still tries to load regular psycopg2 (instead of psycopg2cffi) and crashes.

### Steps to reproduce

- create a PyPy virtualenv
- install synapse, psycopg2cffi via pip
- start any worker

### Logs

```
Apr 13 17:53:14 stratofortress systemd[1]: Started Synapse Matrix homeserver worker federation_reader-0.
Apr 13 17:53:23 stratofortress root[23166]: [None] ***** STARTING SERVER *****
Apr 13 17:53:23 stratofortress root[23166]: [None] Server /home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py version 0.99.3 (b=master,7963d79c3)
Apr 13 17:53:23 stratofortress root[23166]: [None] Server hostname: intelfx.name
Apr 13 17:53:23 stratofortress twisted[23166]: [None] Redirected stdout/stderr to logs
Apr 13 17:53:23 stratofortress synapse.server[23166]: [None] Setting up.
Apr 13 17:53:23 stratofortress synapse.config.appservice[23166]: [None] Loaded application service: ApplicationService: {'token': '<redacted>', 'url': 'http://localhost:8409', 'hs_token': '<redacted>', 'sender': '@appservice-irc:intelfx.name', 'server_name': 'intelfx.name', 'namespaces': {'users': [{'exclusive': True, 'regex': re.compile('@fn_.*:.*')}, {'exclusive': True, 'regex': re.compile('@irc_.*_.*:.*')}], 'aliases': [{'exclusive': True, 'regex': re.compile('#fn_.*:.*')}, {'exclusive': True, 'regex': re.compile('#irc_.*_.*:.*')}], 'rooms': []}, 'id': '134313c1983d17ed2560eeff6ef95db173c5e42861f82d8f07f67054a7fd458d', 'ip_range_whitelist': None, 'protocols': set(), 'rate_limited': True}
Apr 13 17:53:23 stratofortress twisted[23166]: [] Traceback (most recent call last):
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/opt/pypy3/lib-python/3/runpy.py", line 193, in _run_module_as_main
Apr 13 17:53:23 stratofortress twisted[23166]: []     "__main__", mod_spec)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/opt/pypy3/lib-python/3/runpy.py", line 85, in _run_code
Apr 13 17:53:23 stratofortress twisted[23166]: []     exec(code, run_globals)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py", line 186, in <module>
Apr 13 17:53:23 stratofortress twisted[23166]: []     start(sys.argv[1:])
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py", line 178, in start
Apr 13 17:53:23 stratofortress twisted[23166]: []     ss.setup()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 224, in setup
Apr 13 17:53:23 stratofortress twisted[23166]: []     self.datastore = self.DATASTORE_CLASS(conn, self)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/account_data.py", line 30, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedAccountDataStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/account_data.py", line 47, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(AccountDataWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/pushers.py", line 26, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedPusherStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/push_rule.py", line 28, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedPushRuleStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/receipts.py", line 40, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedReceiptsStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/events.py", line 61, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedEventStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/push_rule.py", line 74, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(PushRulesWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/appservice.py", line 59, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(ApplicationServiceWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/receipts.py", line 43, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(ReceiptsWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/event_push_actions.py", line 70, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(EventPushActionsWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/stream.py", line 164, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(StreamWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/state.py", line 375, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(StateGroupWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/registration.py", line 32, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(RegistrationWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/room.py", line 24, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(RoomStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/transactions.py", line 61, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(TransactionStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/_base.py", line 37, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(BaseSlavedStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/_base.py", line 203, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     self._db_pool = hs.get_db_pool()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 540, in _get
Apr 13 17:53:23 stratofortress twisted[23166]: []     dep = builder()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 404, in build_db_pool
Apr 13 17:53:23 stratofortress twisted[23166]: []     **self.db_config.get("args", {})
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/twisted/enterprise/adbapi.py", line 211, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     self.dbapi = reflect.namedModule(dbapiName)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/twisted/python/reflect.py", line 159, in namedModule
Apr 13 17:53:23 stratofortress twisted[23166]: []     topLevel = __import__(name)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/psycopg2/__init__.py", line 50, in <module>
Apr 13 17:53:23 stratofortress twisted[23166]: []     from psycopg2._psycopg import (                     # noqa
Apr 13 17:53:23 stratofortress twisted[23166]: [] ImportError: /home/operator/venv/pypy/site-packages/psycopg2/_psycopg.pypy3-71-x86_64-linux-gnu.so: undefined symbol: timeradd
Apr 13 17:53:23 stratofortress systemd[1]: synapse-worker@federation_reader-0.service: Main process exited, code=exited, status=1/FAILURE
Apr 13 17:53:23 stratofortress systemd[1]: synapse-worker@federation_reader-0.service: Failed with result 'exit-code'.
```

### Version information

- **Homeserver**: intelfx.name
- **Version**: Synapse 0.99.3 / PyPy 7.1.0
- **Install method**: pip
- **Platform**: Arch GNU/Linux



Discussion:
what worked for me is to uninstall psycopg2 and install the psycopg2 compat shim from psycopg2cffi (just add a `psycopg2.py` in `site-packages` with the content below:

From: https://pypi.org/project/psycopg2cffi/#installation

> To use this package with Django or SQLAlchemy invoke a compatibility hook (for example, from settings.py in case of Django, or from a psycopg2.py file in site-packages of your virtual env):

```
from psycopg2cffi import compat
compat.register()
```

> This will map psycopg2cffi to psycopg2, so now any code that does import psycopg2 will use psycopg2cffi.

Additionally I got errors due to a missing function - patching psycopg2cffi with a pull request did help: 

```
$ cd /opt/venvs/matrix-synapse-pypy3/site-packages (or where your pypy3 venv for synapse is) 
$ curl https://patch-diff.githubusercontent.com/raw/chtd/psycopg2cffi/pull/98.patch | patch -p1 
```


we swap in psycopg2cffi in `synapse.storage.engines.create_engine` but not `synapse.storage.database.make_pool` :man_facepalming: 
incidentally, this is trivially fixed by anyone with the inclination...

Resolving patch:
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -86,8 +86,12 @@
 
 CONDITIONAL_REQUIREMENTS = {
     "matrix-synapse-ldap3": ["matrix-synapse-ldap3>=0.1"],
-    # we use execute_values with the fetch param, which arrived in psycopg 2.8.
-    "postgres": ["psycopg2>=2.8"],
+    "postgres": [
+        # we use execute_values with the fetch param, which arrived in psycopg 2.8.
+        "psycopg2>=2.8 ; platform_python_implementation != 'PyPy'",
+        "psycopg2cffi>=2.8 ; platform_python_implementation == 'PyPy'",
+        "psycopg2cffi-compat==1.1 ; platform_python_implementation == 'PyPy'",
+    ],
     # ACME support is required to provision TLS certificates from authorities
     # that use the protocol, such as Let's Encrypt.
     "acme": [
diff --git a/synapse/storage/engines/__init__.py b/synapse/storage/engines/__init__.py
--- a/synapse/storage/engines/__init__.py
+++ b/synapse/storage/engines/__init__.py
@@ -12,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import platform
 
 from ._base import BaseDatabaseEngine, IncorrectDatabaseSetup
 from .postgres import PostgresEngine
@@ -28,11 +27,8 @@ def create_engine(database_config) -> BaseDatabaseEngine:
         return Sqlite3Engine(sqlite3, database_config)
 
     if name == "psycopg2":
-        # pypy requires psycopg2cffi rather than psycopg2
-        if platform.python_implementation() == "PyPy":
-            import psycopg2cffi as psycopg2  # type: ignore
-        else:
-            import psycopg2  # type: ignore
+        # Note that psycopg2cffi-compat provides the psycopg2 module on pypy.
+        import psycopg2  # type: ignore
 
         return PostgresEngine(psycopg2, database_config)
 
diff --git a/synapse/storage/engines/sqlite.py b/synapse/storage/engines/sqlite.py
--- a/synapse/storage/engines/sqlite.py
+++ b/synapse/storage/engines/sqlite.py
@@ -12,6 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import platform
 import struct
 import threading
 import typing
@@ -30,6 +31,11 @@ def __init__(self, database_module, database_config):
         database = database_config.get("args", {}).get("database")
         self._is_in_memory = database in (None, ":memory:",)
 
+        if platform.python_implementation() == "PyPy":
+            # pypy's sqlite3 module doesn't handle bytearrays, convert them
+            # back to bytes.
+            database_module.register_adapter(bytearray, lambda array: bytes(array))
+
         # The current max state_group, or None if we haven't looked
         # in the DB yet.
         self._current_state_group_id = None


Test patch: