Repository: matrix-org/synapse

Issue:
Trial tests against postgres 14 are failing since #11635
Latest deps: https://github.com/matrix-org/synapse/actions/runs/3326877285/jobs/5501039194
Develop: https://github.com/matrix-org/synapse/actions/runs/3323214952/jobs/5493346783

```
[FAIL]
Traceback (most recent call last):
  File "/home/runner/work/synapse/synapse/tests/storage/test_room_search.py", line 372, in test_postgres_web_search_for_phrase
    self._check_test_cases(store, self.COMMON_CASES + self.POSTGRES_CASES, index=0)
  File "/home/runner/work/synapse/synapse/tests/storage/test_room_search.py", line 323, in _check_test_cases
    self.assertEquals(
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.10/lib/python3.10/site-packages/twisted/trial/_synctest.py", line 424, in assertEqual
    super().assertEqual(first, second, msg)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/unittest/case.py", line 845, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/unittest/case.py", line 838, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1 : expected '"fox quick' to match 'the quick brown fox jumps over the lazy dog'

tests.storage.test_room_search.MessageSearchTest.test_postgres_web_search_for_phrase
```

From #11635.


Discussion:
I'm assuming that test is to blame---but tests did pass before this was merged in. So not sure what's going on.
This _didn't_ happen on the twisted trial job. Do we have a new flakey test---some kind of race?
Reproduce with

```shell
docker run --rm -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=synapse -p 5432:5432 postgres:14-alpine &
SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_HOST=127.0.0.1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_POSTGRES_PASSWORD=postgres poetry run trial tests.storage.test_room_search.MessageSearchTest.test_postgres_web_search_for_phrase
```

I tried postgres versions 10--13. On each of these, the test succeeds. I wonder if the FTS behaviour has changed in PG 14?
I'll take a look, thanks for filing! 🎉 
Starting in Postgres 14 the behavior of a hanging double quote was changed; it used to ignore the double quote while now it treats it like a phrase.

In PostgreSQL 11 - 13:

```sql
postgres=# SELECT websearch_to_tsquery('english', '"fox quick');
 websearch_to_tsquery 
----------------------
 'fox' & 'quick'
(1 row)
```

In PostgreSQL 14:

```sql
postgres=# SELECT websearch_to_tsquery('english', '"fox quick');
 websearch_to_tsquery 
----------------------
 'fox' <-> 'quick'
(1 row)
```

----

Not sure what the perfect fix is for this...
See the [release notes for psql 14](https://www.postgresql.org/docs/14/release-14.html) (and search for `tsquery`).

Resolving patch:
diff --git a/synapse/storage/databases/main/search.py b/synapse/storage/databases/main/search.py
--- a/synapse/storage/databases/main/search.py
+++ b/synapse/storage/databases/main/search.py
@@ -824,9 +824,8 @@ def _tokenize_query(query: str) -> TokenList:
     in_phrase = False
     parts = deque(query.split('"'))
     for i, part in enumerate(parts):
-        # The contents inside double quotes is treated as a phrase, a trailing
-        # double quote is not implied.
-        in_phrase = bool(i % 2) and i != (len(parts) - 1)
+        # The contents inside double quotes is treated as a phrase.
+        in_phrase = bool(i % 2)
 
         # Pull out the individual words, discarding any non-word characters.
         words = deque(re.findall(r"([\w\-]+)", part, re.UNICODE))


Test patch:
diff --git a/tests/storage/test_room_search.py b/tests/storage/test_room_search.py
--- a/tests/storage/test_room_search.py
+++ b/tests/storage/test_room_search.py
@@ -239,7 +239,6 @@ class MessageSearchTest(HomeserverTestCase):
         ("fox -nope", (True, False)),
         ("fox -brown", (False, True)),
         ('"fox" quick', True),
-        ('"fox quick', True),
         ('"quick brown', True),
         ('" quick "', True),
         ('" nope"', False),
@@ -269,6 +268,15 @@ def prepare(
         response = self.helper.send(self.room_id, self.PHRASE, tok=self.access_token)
         self.assertIn("event_id", response)
 
+        # The behaviour of a missing trailing double quote changed in PostgreSQL 14
+        # from ignoring the initial double quote to treating it as a phrase.
+        main_store = homeserver.get_datastores().main
+        found = False
+        if isinstance(main_store.database_engine, PostgresEngine):
+            assert main_store.database_engine._version is not None
+            found = main_store.database_engine._version < 140000
+        self.COMMON_CASES.append(('"fox quick', (found, True)))
+
     def test_tokenize_query(self) -> None:
         """Test the custom logic to tokenize a user's query."""
         cases = (
@@ -280,9 +288,9 @@ def test_tokenize_query(self) -> None:
             ("fox -brown", ["fox", SearchToken.Not, "brown"]),
             ("- fox", [SearchToken.Not, "fox"]),
             ('"fox" quick', [Phrase(["fox"]), SearchToken.And, "quick"]),
-            # No trailing double quoe.
-            ('"fox quick', ["fox", SearchToken.And, "quick"]),
-            ('"-fox quick', [SearchToken.Not, "fox", SearchToken.And, "quick"]),
+            # No trailing double quote.
+            ('"fox quick', [Phrase(["fox", "quick"])]),
+            ('"-fox quick', [Phrase(["-fox", "quick"])]),
             ('" quick "', [Phrase(["quick"])]),
             (
                 'q"uick brow"n',