Repository: scrapy/scrapy

Issue:
response_httprepr in DownloaderStats middleware causing scrapy to make unnecessary memory allocation.

### Description

Usage of [`scrapy.utils.response.response_httprepr` ](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/utils/response.py#L45)inside[ `DownloaderStats`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/downloadermiddlewares/stats.py) middleware causing application to make unnecessary memory allocation.
`response_httprepr` used only one time - to calculate response sizes for `downloader/response_bytes` stats in that middleware.

In current implementation `response_httprepr` return `bytes` (immutable type) - in order to calculate `downloader/response_bytes` application will additionally allocate nearly the same memory amount as for original response (only for calculating `len` inside middleware).
https://github.com/scrapy/scrapy/blob/26836c4e1ae9588ee173c5977fc6611364ca7cc7/scrapy/utils/response.py#L45-L60


### Steps to Reproduce
In order to demonstrate influence of this i made this spider:
<details>
  <summary>spider code</summary>

```python
import sys
from importlib import import_module
import scrapy


class MemoryHttpreprSpider(scrapy.Spider):
    name = 'memory_httprepr'
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES':{
            'scrapy.downloadermiddlewares.stats.DownloaderStats': None
        }
    }
# the same as in MemoryUsage extension:
    def get_virtual_size(self):
        size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
        if sys.platform != 'darwin':
            # on macOS ru_maxrss is in bytes, on Linux it is in KB
            size *= 1024
        return size

    def start_requests(self):
        try:
            self.resource = import_module('resource')
        except ImportError:
            pass
        self.logger.info(f"used memory on start: {str(self.get_virtual_size())}")
        yield scrapy.Request(url='https://speed.hetzner.de/100MB.bin', callback=self.parse)
        #yield scrapy.Request(url='http://quotes.toscrape.com', callback=self.parse)

    def parse(self, response, **kwargs):
        self.logger.info(f"used memory after downloading response: {str(self.get_virtual_size())}")
```


</details>
It include:

- usage of `get_virtual_size` method - directly the same as on [`MemoryUsage`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/extensions/memusage.py) extension
- url with 100Mb binary response https://speed.hetzner.de/100MB.bin
- disabled `DownloaderStats` middleware that uses `response_httprepr` and `request_httprepr`
- usual html url http://quotes.toscrape.com (for comparison)

| log output | 100Mb default settings | 100Mb no downloader stats | quotes default |
| --- | --- | --- | --- |
| `[memory_httprepr] used memory on start:` | `61 587 456` | `61 521 920` | `61 558 784` |
| `[memory_httprepr] used memory after downloading response:` | `375 910 400` | `271 179 776` | `61 558 784` |

### Versions

`[scrapy.utils.log] Scrapy 2.4.0 started (bot: httprepr)`
`[scrapy.utils.log] Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (default, Apr 23 2020, 14:32:57) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.1.1, Platform Linux-4.15.0-76-generic-x86_64-with-glibc2.2.5`



Discussion:

Resolving patch:
diff --git a/scrapy/downloadermiddlewares/stats.py b/scrapy/downloadermiddlewares/stats.py
--- a/scrapy/downloadermiddlewares/stats.py
+++ b/scrapy/downloadermiddlewares/stats.py
@@ -1,7 +1,22 @@
 from scrapy.exceptions import NotConfigured
+from scrapy.utils.python import global_object_name, to_bytes
 from scrapy.utils.request import request_httprepr
-from scrapy.utils.response import response_httprepr
-from scrapy.utils.python import global_object_name
+
+from twisted.web import http
+
+
+def get_header_size(headers):
+    size = 0
+    for key, value in headers.items():
+        if isinstance(value, (list, tuple)):
+            for v in value:
+                size += len(b": ") + len(key) + len(v)
+    return size + len(b'\r\n') * (len(headers.keys()) - 1)
+
+
+def get_status_size(response_status):
+    return len(to_bytes(http.RESPONSES.get(response_status, b''))) + 15
+    # resp.status + b"\r\n" + b"HTTP/1.1 <100-599> "
 
 
 class DownloaderStats:
@@ -24,7 +39,8 @@ def process_request(self, request, spider):
     def process_response(self, request, response, spider):
         self.stats.inc_value('downloader/response_count', spider=spider)
         self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)
-        reslen = len(response_httprepr(response))
+        reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4
+        # response.body + b"\r\n"+ response.header + b"\r\n" + response.status
         self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)
         return response
 
diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py
--- a/scrapy/utils/response.py
+++ b/scrapy/utils/response.py
@@ -13,6 +13,7 @@
 
 from twisted.web import http
 from scrapy.utils.python import to_bytes, to_unicode
+from scrapy.utils.decorators import deprecated
 from w3lib import html
 
 
@@ -50,6 +51,7 @@ def response_status_message(status: Union[bytes, float, int, str]) -> str:
     return f'{status_int} {to_unicode(message)}'
 
 
+@deprecated
 def response_httprepr(response: Response) -> bytes:
     """Return raw HTTP representation (as bytes) of the given response. This
     is provided only for reference, since it's not the exact stream of bytes


Test patch:
diff --git a/tests/test_downloadermiddleware_stats.py b/tests/test_downloadermiddleware_stats.py
--- a/tests/test_downloadermiddleware_stats.py
+++ b/tests/test_downloadermiddleware_stats.py
@@ -1,8 +1,10 @@
+from itertools import product
 from unittest import TestCase
 
 from scrapy.downloadermiddlewares.stats import DownloaderStats
 from scrapy.http import Request, Response
 from scrapy.spiders import Spider
+from scrapy.utils.response import response_httprepr
 from scrapy.utils.test import get_crawler
 
 
@@ -37,6 +39,23 @@ def test_process_response(self):
         self.mw.process_response(self.req, self.res, self.spider)
         self.assertStatsEqual('downloader/response_count', 1)
 
+    def test_response_len(self):
+        body = (b'', b'not_empty')  # empty/notempty body
+        headers = ({}, {'lang': 'en'}, {'lang': 'en', 'User-Agent': 'scrapy'})  # 0 headers, 1h and 2h
+        test_responses = [  # form test responses with all combinations of body/headers
+            Response(
+                url='scrapytest.org',
+                status=200,
+                body=r[0],
+                headers=r[1]
+            )
+            for r in product(body, headers)
+        ]
+        for test_response in test_responses:
+            self.crawler.stats.set_value('downloader/response_bytes', 0)
+            self.mw.process_response(self.req, test_response, self.spider)
+            self.assertStatsEqual('downloader/response_bytes', len(response_httprepr(test_response)))
+
     def test_process_exception(self):
         self.mw.process_exception(self.req, MyException(), self.spider)
         self.assertStatsEqual('downloader/exception_count', 1)