Repository: matrix-org/synapse

Issue:
Internal error when posting an event with \u0000
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

Although `"\u0000"` is a perfectly valid JSON string, it seems to fail when storing in the database.

### Steps to reproduce

```
HS_URL=
ROOM_ID=
ACCESS_TOKEN=
curl "https://$HS_URL/_matrix/client/r0/rooms/$ROOM_ID/send/m.room.message" -X POST H 'Content-Type: application/json' -H "Authorization: Bearer $ACCESS_TOKEN" --data-raw '{"msgtype":"m.text","body":"\u0000"}'
```

Error seen in the log file:
```
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py", line 172, in handle_queue_loop
    ret = await per_item_callback(item)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py", line 325, in persisting_queue
    item.events_and_contexts, backfilled=item.backfilled
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py", line 537, in _persist_events
    backfilled=backfilled,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py", line 177, in _persist_events_and_state_updates
    new_forward_extremeties=new_forward_extremeties,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 664, in runInteraction
    **kwargs,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 740, in runWithConnection
    self._db_pool.runWithConnection(inner_func, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/python/compat.py", line 464, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 734, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 534, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/logging/utils.py", line 71, in wrapped
    return f(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py", line 394, in _persist_events_txn
    backfilled=backfilled,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py", line 1362, in _update_metadata_tables_txn
    self._store_room_message_txn(txn, event)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py", line 1653, in _store_room_message_txn
    txn, event, "content.body", event.content["body"]
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py", line 1707, in store_event_search_txn
    origin_server_ts=event.origin_server_ts,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/search.py", line 66, in store_search_entries_txn
    txn.executemany(sql, args)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 291, in executemany
    self._do_execute(self.txn.executemany, sql, *args)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py", line 314, in _do_execute
    return func(sql, *args)
ValueError: A string literal cannot contain NUL (0x00) characters.
```

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: cloud.cervoi.se

If not matrix.org:

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.26.0

- **Install method**: apt
<!-- examples: package manager/git clone/pip  -->

- **Platform**: debian 10
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
For the record I got this when trying to send logs from the docker api to matrix. Docker adds an header on each line to determine if it comes from stdout or stderr, and this header contains NULL bytes, which got encoded by my code to `\u0000`.
We're also seeing this on matrix.org occassionally: https://sentry.matrix.org/sentry/synapse-matrixorg/issues/166204/

Not quite sure what the solution is though.
I guess you could check at the validation step (if there is) if the string contains a null byte, and return an error then.
Shouldn't we store JSON data as bytes, anyways? It's untrusted data.
The problem seems to be related to room message search: this is not related to the JSON encoding. This occurs when Synapse inserts the plaintext of a message into the `event_search` table — well, actually, in Postgres it uses `to_tsvector` to turn it into a text-search vector (see Postgres docs about full-text search for more information). SQLite seems to have an equivalent, but I'm not familiar with it.

We could either omit search index entries for messages with null bytes or perhaps replace them with e.g. a space character (to preserve it being a word boundary character — `"hi\u0000bob"` would then give the search words `hi` and `bob`).

I think replacing the message text with spaces for the `event_search` table insertion would be a more appropriate solution. This doesn't seem like it'd be a large change.
Some relevant bits of postgres' documentation:

[`to_tsvector` accepts any `text` as its input](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-PARSING-DOCUMENTS), where `text` is defined [here](https://www.postgresql.org/docs/current/datatype-character.html). The latter page says:

> The characters that can be stored in any of these data types are determined by the database character set, which is selected when the database is created. Regardless of the specific character set, the character with code zero (sometimes called NUL) cannot be stored.

and refers to [this page](https://www.postgresql.org/docs/current/multibyte.html) as a reference on character sets.

Aside: I think if we wanted to go full pedant on this:
 - The spec [defines](https://spec.matrix.org/unstable/appendices/#grammar) strings to be any sequence of Unicode code points
 - I don't think we prescribe a particular character set when server owners configure postgres? We could recommend that server owners configure postgres so that its text types can store as many codepoints as possible (probably UTF-8?), but maybe this is the sensible default for most server OSes anyway.

But that's going by the letter of the law. Replacing U+0000 with good old [U+0020 space](https://unicode-table.com/en/search/?q=+) is probably the best use of our time!


Another thought: has anyone tried registering a user with a null code point?
> I don't think we prescribe a particular character set when server owners configure postgres? We could recommend that server owners configure postgres so that its text types can store as many codepoints as possible (probably UTF-8?), but maybe this is the sensible default for most server OSes anyway.

The docs for what we ask people to do for postgres are at https://github.com/matrix-org/synapse/blob/251cfc4e09c36c87227771f1a3aa305c4b0a2121/docs/postgres.md#set-up-database

They seem to suggest an encoding of UTF-8 and C-locale. Not using those is known to cause issues. Does postgres have an additional charset configuration beyond that?
The docs recommend: `createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse`.
Synapse makes a fuss about the locale being set to `C` — not sure how that influences things.

I think 0x00 → 0x20 is the sensible choice.

> Another thought: has anyone tried registering a user with a null code point?

I guess you're talking about the user directory? (Equally, the room directory will be potentially affected too)
> But that's going by the letter of the law. Replacing U+0000 with good old [U+0020 space](https://unicode-table.com/en/search/?q=+) is probably the best use of our time!

I think this assumes that the data going in is readable text? I don't think this is valid. It could make more sense to escape it, but than you're dealing with needing to have escape logic which is always a pain. 😢 
> I think this assumes that the data going in is readable text? I don't think this is valid. It could make more sense to escape it, but than you're dealing with needing to have escape logic which is always a pain. cry

How would you escape it, FWIW?

Treating NUL as a delimiter (word boundary) seems like the pragmatic approach (that's all that converting it to a space before feeding it to `to_tsvector` means, right?)

(Though since the message is unlikely to be readable text, I can see the option of just exempting it from message search altogether to be sensible, too.)

Resolving patch:
diff --git a/synapse/storage/databases/main/search.py b/synapse/storage/databases/main/search.py
--- a/synapse/storage/databases/main/search.py
+++ b/synapse/storage/databases/main/search.py
@@ -15,12 +15,12 @@
 import logging
 import re
 from collections import namedtuple
-from typing import Collection, List, Optional, Set
+from typing import Collection, Iterable, List, Optional, Set
 
 from synapse.api.errors import SynapseError
 from synapse.events import EventBase
 from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
-from synapse.storage.database import DatabasePool
+from synapse.storage.database import DatabasePool, LoggingTransaction
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
 from synapse.storage.engines import PostgresEngine, Sqlite3Engine
 
@@ -32,14 +32,24 @@
 )
 
 
+def _clean_value_for_search(value: str) -> str:
+    """
+    Replaces any null code points in the string with spaces as
+    Postgres and SQLite do not like the insertion of strings with
+    null code points into the full-text search tables.
+    """
+    return value.replace("\u0000", " ")
+
+
 class SearchWorkerStore(SQLBaseStore):
-    def store_search_entries_txn(self, txn, entries):
+    def store_search_entries_txn(
+        self, txn: LoggingTransaction, entries: Iterable[SearchEntry]
+    ) -> None:
         """Add entries to the search table
 
         Args:
-            txn (cursor):
-            entries (iterable[SearchEntry]):
-                entries to be added to the table
+            txn:
+            entries: entries to be added to the table
         """
         if not self.hs.config.enable_search:
             return
@@ -55,7 +65,7 @@ def store_search_entries_txn(self, txn, entries):
                     entry.event_id,
                     entry.room_id,
                     entry.key,
-                    entry.value,
+                    _clean_value_for_search(entry.value),
                     entry.stream_ordering,
                     entry.origin_server_ts,
                 )
@@ -70,11 +80,16 @@ def store_search_entries_txn(self, txn, entries):
                 " VALUES (?,?,?,?)"
             )
             args = (
-                (entry.event_id, entry.room_id, entry.key, entry.value)
+                (
+                    entry.event_id,
+                    entry.room_id,
+                    entry.key,
+                    _clean_value_for_search(entry.value),
+                )
                 for entry in entries
             )
-
             txn.execute_batch(sql, args)
+
         else:
             # This should be unreachable.
             raise Exception("Unrecognized database engine")
@@ -646,6 +661,7 @@ def f(txn):
                 for key in ("body", "name", "topic"):
                     v = event.content.get(key, None)
                     if v:
+                        v = _clean_value_for_search(v)
                         values.append(v)
 
                 if not values:


Test patch:
diff --git a/tests/storage/test_room_search.py b/tests/storage/test_room_search.py
new file mode 100644
--- /dev/null
+++ b/tests/storage/test_room_search.py
@@ -0,0 +1,74 @@
+# Copyright 2021 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import synapse.rest.admin
+from synapse.rest.client import login, room
+from synapse.storage.engines import PostgresEngine
+
+from tests.unittest import HomeserverTestCase
+
+
+class NullByteInsertionTest(HomeserverTestCase):
+    servlets = [
+        synapse.rest.admin.register_servlets_for_client_rest_resource,
+        login.register_servlets,
+        room.register_servlets,
+    ]
+
+    def test_null_byte(self):
+        """
+        Postgres/SQLite don't like null bytes going into the search tables. Internally
+        we replace those with a space.
+
+        Ensure this doesn't break anything.
+        """
+
+        # Register a user and create a room, create some messages
+        self.register_user("alice", "password")
+        access_token = self.login("alice", "password")
+        room_id = self.helper.create_room_as("alice", tok=access_token)
+
+        # Send messages and ensure they don't cause an internal server
+        # error
+        for body in ["hi\u0000bob", "another message", "hi alice"]:
+            response = self.helper.send(room_id, body, tok=access_token)
+            self.assertIn("event_id", response)
+
+        # Check that search works for the message where the null byte was replaced
+        store = self.hs.get_datastore()
+        result = self.get_success(
+            store.search_msgs([room_id], "hi bob", ["content.body"])
+        )
+        self.assertEquals(result.get("count"), 1)
+        if isinstance(store.database_engine, PostgresEngine):
+            self.assertIn("hi", result.get("highlights"))
+            self.assertIn("bob", result.get("highlights"))
+
+        # Check that search works for an unrelated message
+        result = self.get_success(
+            store.search_msgs([room_id], "another", ["content.body"])
+        )
+        self.assertEquals(result.get("count"), 1)
+        if isinstance(store.database_engine, PostgresEngine):
+            self.assertIn("another", result.get("highlights"))
+
+        # Check that search works for a search term that overlaps with the message
+        # containing a null byte and an unrelated message.
+        result = self.get_success(store.search_msgs([room_id], "hi", ["content.body"]))
+        self.assertEquals(result.get("count"), 2)
+        result = self.get_success(
+            store.search_msgs([room_id], "hi alice", ["content.body"])
+        )
+        if isinstance(store.database_engine, PostgresEngine):
+            self.assertIn("alice", result.get("highlights"))