Repository: scrapy/scrapy

Issue:
Content-Length header missing in response headers
### Description

The `Content-Length` header missing in the response headers. I stumbled upon this while working on #4897.

### Steps to Reproduce

```
$ scrapy shell https://example.org
(...)
>>> response.headers["Content-Length"]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/.../scrapy/scrapy/http/headers.py", line 40, in __getitem__
    return super().__getitem__(key)[-1]
  File "/.../scrapy/scrapy/utils/datatypes.py", line 23, in __getitem__
    return dict.__getitem__(self, self.normkey(key))
KeyError: b'Content-Length'
```

or

```python
import scrapy

class ContentLengthSpider(scrapy.Spider):
    name = "foo"
    start_urls = ["https://example.org"]

    def parse(self, response):
        print(response.headers["Content-Length"])
```

### Versions

```
Scrapy       : 2.4.1
lxml         : 4.6.2.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.8.2 (default, Apr 18 2020, 17:39:30) - [GCC 7.5.0]
pyOpenSSL    : 20.0.1 (OpenSSL 1.1.1i  8 Dec 2020)
cryptography : 3.4.5
Platform     : Linux-4.15.0-128-generic-x86_64-with-glibc2.27
```

### Additional context

I can see the server does return the header with `cURL`:
```
$ curl -s --http1.1 -D - https://example.org -o /dev/null | grep Content-Length
Content-Length: 1256
```
and `python-requests`:

```python
>>> import requests
>>> requests.get("https://example.org", headers={"Accept-Encoding": "identity"}).headers["Content-Length"]
'1256'
```

It seems to me like Twisted itself is dropping the header:
```python
from twisted.internet import reactor
from twisted.web.client import Agent

agent = Agent(reactor)
d = agent.request(b"GET", b"http://example.org")

def print_response(response):
    print(response.version)
    print(response.headers)

d.addCallback(print_response)
d.addCallback(lambda _: reactor.stop())

reactor.run()
```

```
(b'HTTP', 1, 1)
Headers({b'age': [b'164125'], b'cache-control': [b'max-age=604800'], b'content-type': [b'text/html; charset=UTF-8'], b'date': [b'Wed, 24 Feb 2021 14:38:05 GMT'], b'etag': [b'"3147526947+ident"'], b'expires': [b'Wed, 03 Mar 2021 14:38:05 GMT'], b'last-modified': [b'Thu, 17 Oct 2019 07:18:26 GMT'], b'server': [b'ECS (mic/9ABB)'], b'vary': [b'Accept-Encoding'], b'x-cache': [b'HIT']})
```

I wanted to ask here before opening an issue in Twisted, because this seems like a rather odd thing to do and I'm wondering if I'm missing something :thinking: 
Response headers documentation
Related to #5009


Discussion:
`twisted.web._newclient.HTTPClientParser` has `headers` and `connHeaders` and it looks like it only copies the former into the response object.

```
CONNECTION_CONTROL_HEADERS = set([
            b'content-length', b'connection', b'keep-alive', b'te',
            b'trailers', b'transfer-encoding', b'upgrade',
            b'proxy-connection'])
```
@elacuesta 
I confirm  @wRAR 's statement - headers from `CONNECTION_CONTROL_HEADERS` (including `Content-Length` header`) is not copied to headers of response objects.

Solved it by this monkeypatch:
<details>
<summary>spider code</summary>

```python

import scrapy
from scrapy.crawler import CrawlerProcess
from twisted.web._newclient import HTTPParser

# HTTPParser.CONNECTION_CONTROL_HEADERS.clear()   # <- works (not as expected)
# initially I tried to use the same approach as for _caseMappings
# mentioned in this comment
# https://github.com/scrapy/scrapy/issues/2711#issuecomment-367342284


class HTTTParser_H(HTTPParser):
    def headerReceived(self, name, value):
        name = name.lower()

        if self.isConnectionControlHeader(name):
            self.connHeaders.addRawHeader(name, value)
            self.headers.addRawHeader(name, value)
        else:
            headers = self.headers
            headers.addRawHeader(name, value)

HTTPParser.headerReceived = HTTTParser_H.headerReceived

class ContentLengthSpider(scrapy.Spider):
    name = "foo"
    start_urls = ["https://example.org"]

    def parse(self, response):
        print(response.headers)

process = CrawlerProcess()
process.crawl(ContentLengthSpider)
process.start()

```

</details>

Log output:
```
...
2021-02-24 20:43:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.org> (referer: None)
2021-02-24 20:43:33 [scrapy.core.engine] INFO: Closing spider (finished)
{b'Accept-Ranges': [b'bytes'], b'Age': [b'466759'], b'Cache-Control': [b'max-age=604800'],
b'Content-Type': [b'text/html; charset=UTF-8'],
b'Date': [b'Wed, 24 Feb 2021 18:43:30 GMT'], b'Etag': [b'"3147526947"'],
b'Expires': [b'Wed, 03 Mar 2021 18:43:30 GMT'], b'Last-Modified': [b'Thu, 17 Oct 2019 07:18:26 GMT'],
b'Server': [b'ECS (bsa/EB15)'],
b'Vary': [b'Accept-Encoding'],
b'X-Cache': [b'HIT'], b'Content-Length': [b'648']}
2021-02-24 20:43:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 211,...
```
`Content-Length` reached to scrapy `Response` object  and to spider `parse` method
Interesting, many thanks for the research @wRAR and @GeorgeA92 :smile: 
Looks like a design decision rather than a bug, do you know what is the reason behind it?
Should we consider doing this in Scrapy itself? I was able to work around this in https://github.com/scrapy/scrapy/pull/4897/commits/84e91b670c809aebf3cbe829b4aa2fc84f3de41d, so I have no further need for this specifically.


Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -358,10 +358,19 @@ def _cb_latency(self, result, request, start_time):
         request.meta['download_latency'] = time() - start_time
         return result
 
+    @staticmethod
+    def _headers_from_twisted_response(response):
+        headers = Headers()
+        if response.length is not None:
+            headers[b'Content-Length'] = str(response.length).encode()
+        for key, value in response.headers.getAllRawHeaders():
+            headers[key] = value
+        return headers
+
     def _cb_bodyready(self, txresponse, request):
         headers_received_result = self._crawler.signals.send_catch_log(
             signal=signals.headers_received,
-            headers=Headers(txresponse.headers.getAllRawHeaders()),
+            headers=self._headers_from_twisted_response(txresponse),
             body_length=txresponse.length,
             request=request,
             spider=self._crawler.spider,
@@ -435,7 +444,7 @@ def _cancel(_):
         return d
 
     def _cb_bodydone(self, result, request, url):
-        headers = Headers(result["txresponse"].headers.getAllRawHeaders())
+        headers = self._headers_from_twisted_response(result["txresponse"])
         respcls = responsetypes.from_args(headers=headers, url=url, body=result["body"])
         try:
             version = result["txresponse"].version


Test patch:
diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py
--- a/tests/test_downloader_handlers.py
+++ b/tests/test_downloader_handlers.py
@@ -364,6 +364,13 @@ def test_payload(self):
         d.addCallback(self.assertEqual, body)
         return d
 
+    def test_response_header_content_length(self):
+        request = Request(self.getURL("file"), method=b"GET")
+        d = self.download_request(request, Spider("foo"))
+        d.addCallback(lambda r: r.headers[b'content-length'])
+        d.addCallback(self.assertEqual, b'159')
+        return d
+
 
 class Http10TestCase(HttpTestCase):
     """HTTP 1.0 test case"""