Repository: scrapy/scrapy

Issue:
Using proxy through http fails (https works)
### Description

When I scrape without proxy, both https and http urls work.
Using proxy through https works just fine. My problem is when I try http urls.
In that moment I get the `twisted.web.error.SchemeNotSupported: Unsupported scheme: b''` error

As I see, most of the people have this issue the other way around.

### Steps to Reproduce

1. Scrape a http link with proxy

**Expected behavior:** Get a 200 with the desired data.

**Actual behavior:**
```
ERROR: Error downloading <GET http://*********>
Traceback (most recent call last):
  File "/srv/scraper/venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/srv/scraper/venv/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 42, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 76, in download_request
    return handler.download_request(request, spider)
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 82, in download_request
    return agent.download_request(request)
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 361, in download_request
    d = agent.request(method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File "/srv/scraper/venv/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 262, in request
    endpoint=self._getEndpoint(self._proxyURI),
  File "/srv/scraper/venv/lib/python3.7/site-packages/twisted/web/client.py", line 1729, in _getEndpoint
    return self._endpointFactory.endpointForURI(uri)
  File "/srv/scraper/venv/lib/python3.7/site-packages/twisted/web/client.py", line 1607, in endpointForURI
    raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
twisted.web.error.SchemeNotSupported: Unsupported scheme: b''
```

**Reproduces how often:** Every time I scrape with proxy

### Versions
```
Scrapy       : 2.0.1
lxml         : 4.4.1.0
libxml2      : 2.9.9
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 20.3.0
Python       : 3.7.3 (default, Apr  3 2019, 05:39:12) - [GCC 8.3.0]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.7
Platform     : Linux-4.19.0-5-amd64-x86_64-with-debian-10.0
```
### Additional context

I tried to add some breakpoints at the end to see where it cracks.
I added the following lines in "twisted/web/client/py", before the cracking point:
```python
        endpoint = HostnameEndpoint(self._reactor, host, uri.port, **kwargs)
        import logging
        logger = logging.getLogger(__name__)
        logger.error("%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%")
        logger.error(uri)
        logger.error(uri.host)
        logger.error(uri.port)
        logger.error(uri.scheme)
        logger.error(dir(uri))
        if uri.scheme == b'http':
            return endpoint
        elif uri.scheme == b'https':
            connectionCreator = self._policyForHTTPS.creatorForNetloc(uri.host,
                                                                      uri.port)
            return wrapClientTLS(connectionCreator, endpoint)
        else:
            raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
```

Apparently in this point there is no schema. If I run the same code with a https url, this code is never reached. It seems that getting up to point there is bad and the proxy is not used

(edited to apply formatting)



Discussion:
Could you share a code snippet to reproduce this issue?
```
    def make_request(self, url, callback, meta={}, method='GET'):
        proxy_address = getattr(settings, "PROXY_ADDRESS")

        meta.update({
            'proxy': proxy_address,
            'dont_redirect': self.dont_redirect,
        })
        yield Request(
            url,
            meta=meta,
            method=method,
            callback=callback,
            errback=self.handle_error,
            dont_filter=self.dont_filter
        )
```
Does your `Request.url` and `PROXY_ADDRESS` both have a URL schema?
Apparently, this was the issue. My `PROXY_ADDRESS` did not have a URL schema.
Thanks for the help!

What I want to mention is that this behavior can be confusing. I will describe 4 cases:

 1. `PROXY_ADDRESS`  without URL schema:  `test-proxy.com:3333`

- if Request.url is https it works just fine
- if Request.url is http it results in `twisted.web.error.SchemeNotSupported: Unsupported scheme: b''`

2. `PROXY_ADDRESS`  with URL schema:  `http://second-test-proxy.com:5555`
- Both http and https work fine

In my case, the solution was to add the 'http' in front of my `PROXY_ADDRESS`. I was confused by the fact that it still was working in case `Request.URL` was 'https' case even without URL schema for the `PROXY_ADDRESS`. I don't know if it is a bug or not.
I guess we can take this as an enhancement to support schema-less HTTP proxy URLs.

I checked, and there is no bug, the logic to handle HTTP and HTTPS proxies is different, and the HTTPS one is implemented in a way that the schema is not needed in the proxy URL.

As a reference for people wishing to work on this, it should be as simple as modifying `ScrapyProxyAgent.request` so that the URL parameter passed to `self._getEndpoint` is ensured to have `http://` as schema. Parsing the URL, setting the schema and then unparsing should do the job (https://docs.python.org/3/library/urllib.parse.html).
@Gallaecio  I would like to contribute , I will start this as my first open source contribution
But I may need some help from you 
@liveprasad are you still working on this? If not I can take it.
Hi, I'm pretty new to open source. I have something that is working, but I'm having trouble implementing a test case as required from the contributing docs. 
@HausCloud Create a pull request with the current state of your changes. Maybe we can help you with the rest.
@Gallaecio Will do! Thanks.

UPDATE: Done! If any adjustments are needed, I can fix it! I'd probably need a hint towards the right direction for testing however.
Noticed the pull request was closed accidentally and the branch @HausCloud was working on seems to have been deleted, Is the issue open to work on or is there someone on it already? @Gallaecio 

Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -7,7 +7,7 @@
 from contextlib import suppress
 from io import BytesIO
 from time import time
-from urllib.parse import urldefrag
+from urllib.parse import urldefrag, urlunparse
 
 from twisted.internet import defer, protocol, ssl
 from twisted.internet.endpoints import TCP4ClientEndpoint
@@ -276,7 +276,7 @@ def _get_agent(self, request, timeout):
         bindaddress = request.meta.get('bindaddress') or self._bindAddress
         proxy = request.meta.get('proxy')
         if proxy:
-            _, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
+            proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
             scheme = _parse(request.url)[0]
             proxyHost = to_unicode(proxyHost)
             omitConnectTunnel = b'noconnect' in proxyParams
@@ -301,9 +301,13 @@ def _get_agent(self, request, timeout):
                     pool=self._pool,
                 )
             else:
+                proxyScheme = proxyScheme or b'http'
+                proxyHost = to_bytes(proxyHost, encoding='ascii')
+                proxyPort = to_bytes(str(proxyPort), encoding='ascii')
+                proxyURI = urlunparse((proxyScheme, proxyNetloc, proxyParams, '', '', ''))
                 return self._ProxyAgent(
                     reactor=reactor,
-                    proxyURI=to_bytes(proxy, encoding='ascii'),
+                    proxyURI=to_bytes(proxyURI, encoding='ascii'),
                     connectTimeout=timeout,
                     bindAddress=bindaddress,
                     pool=self._pool,
diff --git a/scrapy/core/downloader/webclient.py b/scrapy/core/downloader/webclient.py
--- a/scrapy/core/downloader/webclient.py
+++ b/scrapy/core/downloader/webclient.py
@@ -1,3 +1,4 @@
+import re
 from time import time
 from urllib.parse import urlparse, urlunparse, urldefrag
 
@@ -32,6 +33,8 @@ def _parse(url):
     and is ascii-only.
     """
     url = url.strip()
+    if not re.match(r'^\w+://', url):
+        url = '//' + url
     parsed = urlparse(url)
     return _parsed_url_args(parsed)
 


Test patch:
diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py
--- a/tests/test_downloader_handlers.py
+++ b/tests/test_downloader_handlers.py
@@ -764,6 +764,16 @@ def test_download_with_proxy_https_timeout(self):
         timeout = yield self.assertFailure(d, error.TimeoutError)
         self.assertIn(domain, timeout.osError)
 
+    def test_download_with_proxy_without_http_scheme(self):
+        def _test(response):
+            self.assertEqual(response.status, 200)
+            self.assertEqual(response.url, request.url)
+            self.assertEqual(response.body, self.expected_http_proxy_request_body)
+
+        http_proxy = self.getURL('').replace('http://', '')
+        request = Request('http://example.com', meta={'proxy': http_proxy})
+        return self.download_request(request, Spider('foo')).addCallback(_test)
+
 
 class HttpDownloadHandlerMock: