Repository: matrix-org/synapse

Issue:
Synapse doesn't limit data/time for http previews
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

<!-- Describe here the problem that you are experiencing -->
I have an Icecast stream running and a while back decided to send a link to it to someone over Matrix. A couple of days later I check the admin panel and noticed the following:
![image](https://user-images.githubusercontent.com/10010899/92992703-209fee00-f4ed-11ea-909d-4c24e8a68aba.png)
It seems that, while attempting to make the preview for the url, synapse started listening to the stream and it never stopped, because the stream never ends. Now this is a very specific case which could be solved by me properly disallowing preview fetching on my home network, but imagine if there were other homeservers in that room.

This points to the fact that synapse doesn't properly limit itself when trying to generate http previews, causing it to keep busy forever. In fact, this seems to happen multiple times as at the time of writing there are 3 synapse clients listening to the stream. 

### Steps to reproduce

- Have something with an infinite stream (such as http radio)
- Post a link to this stream in a room
- Any server having previews enabled will now be listening to the stream indefinitely. 

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->
### Expected Behaviour

It would probably be best if Synapse kills the "thread" when it notices it can't make the preview/it takes too long. Looking at the logs, it seems Synapse does notice that the stream is too large, but it still keeps listening:

```
2020-09-12 10:29:43,512 - synapse.http.client - 339 - INFO - GET-3961617 - Received response to GET https://mkmkm.thebias.nl/: 200
2020-09-12 10:33:34,790 - synapse.http.server - 75 - INFO - GET-3961617 - <XForwardedForRequest at 0x7f5c555f3710 method='GET' uri='/_matrix/media/r0/preview_url?url=https%3A%2F%2Fmkmkm.thebias.nl%2F&ts=1599673680000' clientproto='HTTP/1.0' site=8008> SynapseError: 502 - Requested file is too large > 10485760 bytes
2020-09-12 10:33:34,790 - synapse.http.server - 531 - WARNING - GET-3961617 - Not sending response to request <XForwardedForRequest at 0x7f5c555f3710 method='GET' uri='/_matrix/media/r0/preview_url?url=https%3A%2F%2Fmkmkm.thebias.nl%2F&ts=1599673680000' clientproto='HTTP/1.0' site=8008>, already disconnected.
2020-09-12 10:33:34,791 - synapse.access.http.8008 - 311 - INFO - GET-3961617 - 2001:67c:2564:331:5d2:b8cb:8cf7:a474 - 8008 - {@rafael:thebias.nl} Processed request: 231.856sec/-171.857sec (0.004sec, 0.000sec) (0.000sec/0.002sec/1) 0B 200! "GET /_matrix/media/r0/preview_url?url=https%3A%2F%2Fmkmkm.thebias.nl%2F&ts=1599673680000 HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0" [0 dbevts]
```

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: Personal Homeserver

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.19.1

- **Install method**: debian package
<!-- examples: package manager/git clone/pip  -->

- **Platform**: debian 10
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
Looks like we only time out fetching response headers, not reading the body
I can confirm, this issue still exists on Synapse/1.44.0
A side effect of this issue is that files and folders in the `url_cache` directory don't get cleaned up properly: https://github.com/matrix-org/synapse/issues/10863#issuecomment-927886021
as a workaround, i have a nginx frontend in front of my icecast and adedd the following discussion to block synapse clients from hogging the feeds:

```
# Synapse/1.50.0rc1 (b=matrix-org-hotfixes,8e8a00829)
if ($http_user_agent ~ "^Synapse/.*"){
      return 403;
}
```
I guess another work around is to disable url previews completelly in `homeserver.yaml`:
```yaml
url_preview_enabled: False
```

Resolving patch:
diff --git a/synapse/http/client.py b/synapse/http/client.py
--- a/synapse/http/client.py
+++ b/synapse/http/client.py
@@ -731,15 +731,24 @@ async def get_file(
         # straight back in again
 
         try:
-            length = await make_deferred_yieldable(
-                read_body_with_max_size(response, output_stream, max_size)
-            )
+            d = read_body_with_max_size(response, output_stream, max_size)
+
+            # Ensure that the body is not read forever.
+            d = timeout_deferred(d, 30, self.hs.get_reactor())
+
+            length = await make_deferred_yieldable(d)
         except BodyExceededMaxSize:
             raise SynapseError(
                 HTTPStatus.BAD_GATEWAY,
                 "Requested file is too large > %r bytes" % (max_size,),
                 Codes.TOO_LARGE,
             )
+        except defer.TimeoutError:
+            raise SynapseError(
+                HTTPStatus.BAD_GATEWAY,
+                "Requested file took too long to download",
+                Codes.TOO_LARGE,
+            )
         except Exception as e:
             raise SynapseError(
                 HTTPStatus.BAD_GATEWAY, ("Failed to download remote body: %s" % e)


Test patch: