Repository: scrapy/scrapy

Issue:
Proxy Tunnel Error message length
## Summary
Proxy TunnelError message length shouldn't be truncated to 32 bytes

## Motivation
Often there might be some Proxy TunnelErrors which are currently truncated to 32 characters by these lines of code: https://github.com/scrapy/scrapy/blame/master/scrapy/core/downloader/handlers/http11.py#L170 / https://github.com/scrapy/scrapy/blame/master/scrapy/core/downloader/handlers/http11.py#L125

## Describe alternatives you've considered
Is there a legitimate reason why this should be truncated?

## Additional context
I'm getting this exception raised
`scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy zproxy.lum-superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error No pe'}]`
instead of
`scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy zproxy.lum-superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error No peers available'}]`

Tried out by removing the 32 character truncation in my specific use-case and it seems to work.


Discussion:
I see no discussion in https://github.com/scrapy/scrapy/pull/2069 about the introduction of the 32 limit, so I am not 100% sure of why that value was chosen, it may have been an arbitrary choice. But I think truncating makes sense, allowing an input string of any length to get logged there may not be wise.

Maybe we could simply allow for a longer string to be logged? 64?
From the logging length point of view I agree with you. I think 64 might still be a little bit short but probably much more helpful than the current limitation is.
@akinad3 I don't think the length of 64 will be short for any error message here. Should I make the change?
I agree. We can always make it larger later if a scenario where that would help is brought up.

Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -121,8 +121,9 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
     with this endpoint comes from the pool and a CONNECT has already been issued
     for it.
     """
-
-    _responseMatcher = re.compile(br'HTTP/1\.. (?P<status>\d{3})(?P<reason>.{,32})')
+    _truncatedLength = 1000
+    _responseAnswer = r'HTTP/1\.. (?P<status>\d{3})(?P<reason>.{,' + str(_truncatedLength) + r'})'
+    _responseMatcher = re.compile(_responseAnswer.encode())
 
     def __init__(self, reactor, host, port, proxyConf, contextFactory, timeout=30, bindAddress=None):
         proxyHost, proxyPort, self._proxyAuthHeader = proxyConf
@@ -167,7 +168,7 @@ def processProxyResponse(self, rcvd_bytes):
                 extra = {'status': int(respm.group('status')),
                          'reason': respm.group('reason').strip()}
             else:
-                extra = rcvd_bytes[:32]
+                extra = rcvd_bytes[:self._truncatedLength]
             self._tunnelReadyDeferred.errback(
                 TunnelError('Could not open CONNECT tunnel with proxy '
                             f'{self._host}:{self._port} [{extra!r}]')


Test patch: