Repository: matrix-org/synapse

Issue:
Spec compliance: require a request body on `POST /rooms/{roomId}/receipt/{receiptType}/{eventId}`
We currently don't require a body on `POST /rooms/{roomId}/receipt/{receiptType}/{eventId}`.

Initially, this seems to have been a mistake, but since #10531 [v1.40.0rc2], it's a workaround because Element Android relied on that behaviour (which we broke by actually using the body for hidden read receipts in #10413 [v1.40.0rc1]).

__Blockers:__

- [x] Yell in Client Developers room now
- [x] https://github.com/vector-im/element-android/issues/3789
- [x] Figure out how long Element Android needs to propagate, set target removal date
    Minimum of four weeks following release. Revisit in October.
- [ ] Wait _x time_ for a fixed Android release to propagate

Eventually, we should return to enforcing the spec.


Discussion:
I'd like us to set a timescale for deprecation of this workaround while it's still fresh in our heads.
Fixed in Element Android this week, will be in EA 1.2.1. Have asked for an estimate of user uptake so we can figure out a timeline and will report back once I know.

I've also mentioned this anticipated change in the Matrix Client Developers room.
General feeling is "the longer the better" for Android uptake, but at least 4-6 weeks. We should watch traffic on matrix.org to confirm.
Thanks Dan. Let's revisit this at the end of October in the hope that we will feel comfortable fixing it then?
> "the longer the better" for Android uptake

yeah, especially considering F-Droid (the builds lag behind a bit and generally require manual updating).
I'm certainly guilty of not updating for weeks (though it's also likely that F-Droid users will tend to know how to fend for themselves a bit, and realise that updating might fix bugs)...
Let's check matrix.org stats for EA < 1.2.1, but otherwise target for the tail end of the Synapse 1.46 window (thus merging around 21st Oct to give us a few days before RC1)
Checking stats this week
Did a little bit of sleuthing and for users on matrix.org last Wednesday (2021-10-13), who had `Android` in their user agent string, roughly:

- 85.5% were on known good versions of clients
- 14.4% were on known bad versions of clients
- 0.1% were on unknown clients

I don't think we can unilaterally break that fraction of our Android users, but the known clients are _known_, so we could constrain the hack to only allow empty bodies from those specific clients. This will at least prevent the problem from getting worse.
Specifically, we could limit the hack to _only_ apply when the user agent starts with:

- Anything containing `Riot` (e.g., `Old Riot.im`, `Riot`, `RiotX`, `Element (Riot.im)`)
- Anything starting with `Element/1.[012].*`
- Anything starting with `SchildiChat/1.[012].*`
Question: Do we constrain the workaround per above, or do we wait even longer? The rate of decay is slow enough that I don't think we'll dip under 10% for many months.
I'd be pretty reluctant to add user agent sniffing in to the workaround; it feels nasty to make the homeserver have to discriminate on user agent (and it's not particularly fair for e.g. forks of Element that we're not aware of, which may not be sending the same user agent).

However, 15% of Android users is a substantial number of users to cut off (were you looking at the active sessions, daily visits or all the devices, ooi?). I would be unhappy doing so because it will just be one more thing that makes Matrix as a whole look buggy.
> (were you looking at the active sessions, daily visits or all the devices, ooi?)

I was specifically looking at the results of the following query:

```sql
SELECT COUNT(*), user_agent
FROM user_daily_visits
WHERE timestamp >= extract(epoch from timestamp '2021-10-13')::bigint * 1000
  AND timestamp < extract(epoch from timestamp '2021-10-14')::bigint * 1000 
  AND user_agent ILIKE '%android%'
  AND user_agent NOT ILIKE 'Mozilla%'
GROUP BY user_agent
ORDER BY user_agent
```
> it feels nasty to make the homeserver have to discriminate on user agent

Agreed that a UA-specific hack is... a hack... but at least it stops new callers from accidentally adopting the incorrect behavior, so it at least ensures that the ecosystem does not regress.

> it's not particularly fair

I think it's fair _enough._ To the best of our knowledge (which includes actually breaking people in production), _only_ Element Android and its forks have ever been subject to this particular misimplementation of the spec.

From looking at _all_ clients that advertised themselves as "android" on October 13th, limiting the hack to _just_ old versions of Riot, Element Android, and SchildiChat would get us to a point where _literally_ more than 99.9% of all Android clients we saw on matrix.org would explicitly be known to work.

Basically, I'm willing to risk that long tail of fewer than 0.1% of android clients which we've not explicitly verified whether they comply with the spec or not.
We'll implement the UA hack as a stopgap.
Tracking the UA hack in #11156, punting this into the "look at it next year" bucket.
Next step is to rip out the hack once we're comfortable with ecosystem adoption...

Dan to pull numbers again.
(Waiting until next week Wednesday just so we get a more representative sample of people coming back from holidays)
FWIW #12168 is an update to the hidden read receipt logic which means we don't actually need a body there anymore. We could remove all the code that cares about the body or still move forward with this?
It's not a big deal but I think we should still fix this; it's only fair to other homeserver implementations that we should enforce the spec a bit when we become aware of a deficiency.
Android team are happy for us to drop the workaround patch now!
> Android team are happy for us to drop the workaround patch now!

In that case, next step is to strip out the user-agent specific logic added in https://github.com/matrix-org/synapse/pull/11157, and write a removal notice in the changelog.

Resolving patch:
diff --git a/synapse/rest/client/receipts.py b/synapse/rest/client/receipts.py
--- a/synapse/rest/client/receipts.py
+++ b/synapse/rest/client/receipts.py
@@ -13,12 +13,10 @@
 # limitations under the License.
 
 import logging
-import re
 from typing import TYPE_CHECKING, Tuple
 
 from synapse.api.constants import ReceiptTypes
 from synapse.api.errors import SynapseError
-from synapse.http import get_request_user_agent
 from synapse.http.server import HttpServer
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
 from synapse.http.site import SynapseRequest
@@ -26,8 +24,6 @@
 
 from ._base import client_patterns
 
-pattern = re.compile(r"(?:Element|SchildiChat)/1\.[012]\.")
-
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
@@ -69,14 +65,7 @@ async def on_POST(
         ):
             raise SynapseError(400, "Receipt type must be 'm.read'")
 
-        # Do not allow older SchildiChat and Element Android clients (prior to Element/1.[012].x) to send an empty body.
-        user_agent = get_request_user_agent(request)
-        allow_empty_body = False
-        if "Android" in user_agent:
-            if pattern.match(user_agent) or "Riot" in user_agent:
-                allow_empty_body = True
-        # This call makes sure possible empty body is handled correctly
-        parse_json_object_from_request(request, allow_empty_body)
+        parse_json_object_from_request(request, allow_empty_body=False)
 
         await self.presence_handler.bump_presence_active_time(requester.user)
 


Test patch:
diff --git a/tests/rest/client/test_sync.py b/tests/rest/client/test_sync.py
--- a/tests/rest/client/test_sync.py
+++ b/tests/rest/client/test_sync.py
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import json
+from http import HTTPStatus
 from typing import List, Optional
 
 from parameterized import parameterized
@@ -485,30 +486,7 @@ def test_private_receipt_cannot_override_public(self) -> None:
         # Test that we didn't override the public read receipt
         self.assertIsNone(self._get_read_receipt())
 
-    @parameterized.expand(
-        [
-            # Old Element version, expected to send an empty body
-            (
-                "agent1",
-                "Element/1.2.2 (Linux; U; Android 9; MatrixAndroidSDK_X 0.0.1)",
-                200,
-            ),
-            # Old SchildiChat version, expected to send an empty body
-            ("agent2", "SchildiChat/1.2.1 (Android 10)", 200),
-            # Expected 400: Denies empty body starting at version 1.3+
-            ("agent3", "Element/1.3.6 (Android 10)", 400),
-            ("agent4", "SchildiChat/1.3.6 (Android 11)", 400),
-            # Contains "Riot": Receipts with empty bodies expected
-            ("agent5", "Element (Riot.im) (Android 9)", 200),
-            # Expected 400: Does not contain "Android"
-            ("agent6", "Element/1.2.1", 400),
-            # Expected 400: Different format, missing "/" after Element; existing build that should allow empty bodies, but minimal ongoing usage
-            ("agent7", "Element dbg/1.1.8-dev (Android)", 400),
-        ]
-    )
-    def test_read_receipt_with_empty_body(
-        self, name: str, user_agent: str, expected_status_code: int
-    ) -> None:
+    def test_read_receipt_with_empty_body_is_rejected(self) -> None:
         # Send a message as the first user
         res = self.helper.send(self.room_id, body="hello", tok=self.tok)
 
@@ -517,9 +495,9 @@ def test_read_receipt_with_empty_body(
             "POST",
             f"/rooms/{self.room_id}/receipt/m.read/{res['event_id']}",
             access_token=self.tok2,
-            custom_headers=[("User-Agent", user_agent)],
         )
-        self.assertEqual(channel.code, expected_status_code)
+        self.assertEqual(channel.code, HTTPStatus.BAD_REQUEST)
+        self.assertEqual(channel.json_body["errcode"], "M_NOT_JSON", channel.json_body)
 
     def _get_read_receipt(self) -> Optional[JsonDict]:
         """Syncs and returns the read receipt."""