Repository: scrapy/scrapy

Issue:
Error 302 redirection with headers location starts with 3 slash

### Description

when the 302 response return a headers's location startswith 3 slash, the scrapy redirect to a url different from what the browser do.

### Steps to Reproduce

1.  scrapy shell https://www.hjenglish.com/new/p1285798/

**Expected behavior:** 
redirect to `https://fr.hujiang.com/new/p1285798/` as browser `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36`  do.


**Actual behavior:** 
redirct to `https://www.hjenglish.com/fr.hujiang.com/new/p1285798`

**Reproduces how often:** 

everytime

### Versions
Scrapy       : 1.7.3
lxml         : 4.3.2.0
libxml2      : 2.9.9
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.20.0
Twisted      : 19.7.0
Python       : 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.6.1
Platform     : Windows-10-10.0.17134-SP0


### Additional context

I check the defination of [Location in rfc](https://tools.ietf.org/html/rfc7231#section-7.1.2) and end with [reference resolution](https://tools.ietf.org/html/rfc3986#section-5.3). But I fail to findout how to resolve the Location startswith `///`. So I don't know why Chrome did so.

The behavior of scrapy is determined by [redirect.py#L73](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/redirect.py#L73), which will truncate `///` to `/`。

I'm wandering the differents betweent scarpy and browser...



Discussion:
We aim to behave as close to browsers as possible, so let’s have a look at it.

I was able to replicate this issue, @Gallaecio may I work on this?

@amardeep24 Please, go ahead! :slightly_smiling_face: 
@Gallaecio The difference between Scrapy and a browser is that, when the `location` header in the response is having a relative URL like `///fr.hujiang.com/new/p1285798/` (which is permitted as per the RFC 7213) the browser is making a request to the absolute URL `https://fr.hujiang.com/new/p1285798/` where as Scrapy is considering it as a relative URL w.r.t to the base URL:  `https://www.hjenglish.com/fr.hujiang.com/new/p1285798`.

This is happening because we have `urljoin(request.url, location)` in `redirect.py`, which resolves to `https://www.hjenglish.com/fr.hujiang.com/new/p1285798` from the location sanitized URL `/fr.hujiang.com/new/p1285798/`.

How can we know whether the URL should be relative w.r.t base URL or a completely new URL? should we make two calls one for the `https://www.hjenglish.com/fr.hujiang.com/new/p1285798` and one for the `https://fr.hujiang.com/new/p1285798/` by adding `https://` before the relative URL and scrape the one which succeed ?

First we should find out if the URL sanitizer should be updated to handle this case differently. Otherwise, maybe we need to replace `///` at the start by `<protocol>://` before passing the value to the sanitizer.
@Gallaecio  This fixes the issue:
 ```
import logging
+import re
 from six.moves.urllib.parse import urljoin
+from six.moves.urllib.parse import urlparse
 
 from w3lib.url import safe_url_string
 
@@ -71,6 +73,9 @@ class RedirectMiddleware(BaseRedirectMiddleware):
             return response
 
         location = safe_url_string(response.headers['location'])
+        if response.headers['location'].decode().startswith('/', 1):
+            request_scheme = urlparse(request.url).scheme
+            location = request_scheme + '://' + re.sub('^/*', '', location)
 
         redirected_url = urljoin(request.url, location)
```

Here I checked whether the location header contains a URL starting with multiple `///` if it does then append `<scheme>://` before it and make it an absolute URL else use the old logic. Should I raise a PR?

Please, open a pull request.

Resolving patch:
diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py
--- a/scrapy/downloadermiddlewares/redirect.py
+++ b/scrapy/downloadermiddlewares/redirect.py
@@ -1,5 +1,5 @@
 import logging
-from six.moves.urllib.parse import urljoin
+from six.moves.urllib.parse import urljoin, urlparse
 
 from w3lib.url import safe_url_string
 
@@ -70,7 +70,10 @@ def process_response(self, request, response, spider):
         if 'Location' not in response.headers or response.status not in allowed_status:
             return response
 
-        location = safe_url_string(response.headers['location'])
+        location = safe_url_string(response.headers['Location'])
+        if response.headers['Location'].startswith(b'//'):
+            request_scheme = urlparse(request.url).scheme
+            location = request_scheme + '://' + location.lstrip('/')
 
         redirected_url = urljoin(request.url, location)
 


Test patch:
diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py
--- a/tests/test_downloadermiddleware_redirect.py
+++ b/tests/test_downloadermiddleware_redirect.py
@@ -106,6 +106,22 @@ def test_redirect_302_head(self):
         del rsp.headers['Location']
         assert self.mw.process_response(req, rsp, self.spider) is rsp
 
+    def test_redirect_302_relative(self):
+        url = 'http://www.example.com/302'
+        url2 = '///i8n.example2.com/302'
+        url3 = 'http://i8n.example2.com/302'
+        req = Request(url, method='HEAD')
+        rsp = Response(url, headers={'Location': url2}, status=302)
+
+        req2 = self.mw.process_response(req, rsp, self.spider)
+        assert isinstance(req2, Request)
+        self.assertEqual(req2.url, url3)
+        self.assertEqual(req2.method, 'HEAD')
+
+        # response without Location header but with status code is 3XX should be ignored
+        del rsp.headers['Location']
+        assert self.mw.process_response(req, rsp, self.spider) is rsp
+
 
     def test_max_redirect_times(self):
         self.mw.max_redirect_times = 1