Repository: scrapy/scrapy

Issue:
scrapy.Request no init error on invalid url
I stumbled on some weird issue, spider got some invalid url, but instead of crashing loudly when trying to create scrapy.Request() with invalid url it just silently ignored this error. Sample to reproduce

```python
from scrapy.spiders import Spider
from scrapy import Request


class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
    ]

    def parse(self, response):
        invalid_url = "/container.productlist.productslist.productthumbnail.articledetaillink.layerlink:open-layer/0/CLASSIC/-1/WEB$007cARBO$007c13263065/null$007cDisplay$0020Product$002f111499$002fAil$0020blanc$007c?t:ac=13263065"
        yield Request(invalid_url)
```

this generates following output:

```
2017-02-09 12:21:04 [scrapy.core.engine] INFO: Spider opened
2017-02-09 12:21:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-02-09 12:21:04 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2017-02-09 12:21:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
2017-02-09 12:21:04 [scrapy.core.engine] INFO: Closing spider (finished)
```

there is no information about trying to generate this Request with invalid_url, no stacktrace, no error info from middleware. Why?


Discussion:
Disturning indeed.
If you have an invalid URL in `start_urls`, scrapy will complain:

```
2017-02-09 12:30:40 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/scrapy13/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "/home/paul/.virtualenvs/scrapy13/local/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 70, in start_requests
    yield self.make_requests_from_url(url)
  File "/home/paul/.virtualenvs/scrapy13/local/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 73, in make_requests_from_url
    return Request(url, dont_filter=True)
  File "/home/paul/.virtualenvs/scrapy13/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/paul/.virtualenvs/scrapy13/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 57, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: /test
2017-02-09 12:30:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
2017-02-09 12:30:40 [dmoz] DEBUG: parse(<200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>)
2017-02-09 12:30:40 [scrapy.core.engine] INFO: Closing spider (finished)
```

The `ValueError` exception is gobbled up somewhere around callbacks.
aha but it seems like there's something in this specific invalid url I've got there @redapple , because others are raising exception, e.g. 

```python
invalid_url = "/aaaa/"
yield Request(invalid_url)
```

raises `ValueError: Missing scheme in request url: /aaaa/`

I think this check https://github.com/scrapy/scrapy/blob/master/scrapy/http/request/__init__.py#L56

```python
if ':' not in self._url:
            raise ValueError('Missing scheme in request url: %s' % self._url)
```
is not good. Every request that contains ":" will pass through, e.g.

```python
In [4]: Request("aa:")
Out[4]: <GET aa:>
```
True, parsing/checking the URL for a scheme could be done earlier, before making it safe and escaping ajax fragments.
I think this could be solved by:
```
if not urlsplit(url).scheme:
    raise ValueError('Missing scheme in request url: %s' % self._url)
```
I'm not so sure since `urlsplit` is already called in `safe_url_string`. However, I did a very quick and simple test and the performance penalty of calling `urlsplit` over `':' not in self._url` seems barely noticeable when creating 1M requests:

```
from scrapy import Request
for i in range(10**6):
    r = Request('http://www.example.org')

# with urlsplit
real	0m25.828s
user	0m25.565s
sys	0m0.139s

# without urlsplit
real	0m24.612s
user	0m24.384s
sys	0m0.131s
```

What do you think?
@elacuesta urlsplit uses cache, it is likely that's the reason urlsplit is fast in your example.
Interesting, I didn't know that. I was worried about the performance penalty of calling `urlsplit` multiple times inside the `_set_url` method, maybe if it uses cache that would not be a problem, right?
@elacuesta , `urlsplit` in `Request._set_url` and `urlsplit` in `safe_url_string` may not operate on the same input since `safe_url_string` [converts the URL to Unicode](https://github.com/scrapy/w3lib/blob/f46b4c4140dfd0081d34327d91f496f7a221bed7/w3lib/url.py#L95) beforehand if it's not already.
So you can also try with Unicode input and with bytes input if it makes a difference.
Hi, I would like to solve this bug, can anybody please guide me how can I fix this bug? It would be my first bug to solve,that's why I would like to solve it.
thanks :) 
I tried to solve this bug and used
```
if '://' not in self._url:
            raise ValueError('Missing scheme in request url: %s' % self._url)
```
instead
```
if ':' not in self._url:
            raise ValueError('Missing scheme in request url: %s' % self._url)
```
at https://github.com/scrapy/scrapy/blob/master/scrapy/http/request/__init__.py#L56 and it pass all tests but excepts these 6 which all were in https://github.com/scrapy/scrapy/blob/master/tests/test_downloader_handlers.py in class DataURITestCase(unittest.TestCase), and I saw their Request with these url

        request = Request("data:,A%20brief%20note")
        request = Request("data:,A%20brief%20note")
        request = Request("data:;charset=iso-8859-7,%be%d3%be")
        request = Request("data:text/plain;charset=iso-8859-7,%be%d3%be")
        request = Request('data:text/plain;foo=%22foo;bar%5C%22%22;'
                          'charset=utf-8;bar=%22foo;%5C%22 foo ;/,%22'
                          ',%CE%8E%CE%A3%CE%8E')
        request = Request('data:text/plain;base64,SGVsbG8sIHdvcmxkLg%3D%3D')

can't we change  these URL in test cases? Actually I don't know where these type of urls are necessary for Request, surely you people know better than me.
Thanks in advance community.
@kmike @redapple @pawelmhm 

Resolving patch:
diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py
--- a/scrapy/http/request/__init__.py
+++ b/scrapy/http/request/__init__.py
@@ -66,7 +66,7 @@ def _set_url(self, url):
         s = safe_url_string(url, self.encoding)
         self._url = escape_ajax(s)
 
-        if ':' not in self._url:
+        if ('://' not in self._url) and (not self._url.startswith('data:')):
             raise ValueError('Missing scheme in request url: %s' % self._url)
 
     url = property(_get_url, obsolete_setter(_set_url, 'url'))


Test patch:
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -52,6 +52,8 @@ def test_init(self):
 
     def test_url_no_scheme(self):
         self.assertRaises(ValueError, self.request_class, 'foo')
+        self.assertRaises(ValueError, self.request_class, '/foo/')
+        self.assertRaises(ValueError, self.request_class, '/foo:bar')
 
     def test_headers(self):
         # Different ways of setting headers attribute