Repository: scrapy/scrapy

Issue:
Wikipedia robots.txt raises exceptions
I'm scraping a page which in turn links to wikipedia.

But the wikipedia robots.txt is creating some errors/exceptions as below.

Python 2.7.12
Scrapy 1.2.1

```
2016-11-02 13:13:18 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/robots.txt> (referer: None)
2016-11-02 13:13:18 [py.warnings] WARNING: C:\Python27\lib\urllib.py:1303: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return ''.join(map(quoter, s))

2016-11-02 13:13:18 [scrapy] ERROR: Error downloading <GET http://en.wikipedia.org/robots.txt>: u'\xd8'
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 587, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Python27\lib\site-packages\scrapy\downloadermiddlewares\robotstxt.py", line 97, in _parse_robots
    rp.parse(body.splitlines())
  File "C:\Python27\lib\robotparser.py", line 120, in parse
    entry.rulelines.append(RuleLine(line[1], False))
  File "C:\Python27\lib\robotparser.py", line 174, in __init__
    self.path = urllib.quote(path)
  File "C:\Python27\lib\urllib.py", line 1303, in quote
    return ''.join(map(quoter, s))
KeyError: u'\xd8'
```


Discussion:
I can reproduce this with Python 2.7 on Linux but not with Python 3.

```
$ cat wiki.py
import scrapy


class WikipediaSpider(scrapy.Spider):
    name = "wikipedia"

    start_urls = ['https://en.wikipedia.org/']

    def parse(self, response):
        pass
```

```
$ scrapy version -v
Scrapy    : 1.2.1
lxml      : 3.6.4.0
libxml2   : 2.9.4
Twisted   : 16.5.0
Python    : 2.7.12 (default, Jul  1 2016, 15:12:24) - [GCC 5.4.0 20160609]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.2g  1 Mar 2016)
Platform  : Linux-4.4.0-45-generic-x86_64-with-Ubuntu-16.04-xenial

$ scrapy runspider wiki.py -s ROBOTSTXT_OBEY=1
2016-11-02 14:31:24 [scrapy] INFO: Scrapy 1.2.1 started (bot: scrapybot)
2016-11-02 14:31:24 [scrapy] INFO: Overridden settings: {'ROBOTSTXT_OBEY': '1'}
(...)
2016-11-02 14:31:24 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/robots.txt> (referer: None)
2016-11-02 14:31:24 [py.warnings] WARNING: /usr/lib/python2.7/urllib.py:1299: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return ''.join(map(quoter, s))

2016-11-02 14:31:24 [scrapy] ERROR: Error downloading <GET https://en.wikipedia.org/robots.txt>: u'\xd8'
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 649, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/downloadermiddlewares/robotstxt.py", line 97, in _parse_robots
    rp.parse(body.splitlines())
  File "/usr/lib/python2.7/robotparser.py", line 120, in parse
    entry.rulelines.append(RuleLine(line[1], False))
  File "/usr/lib/python2.7/robotparser.py", line 174, in __init__
    self.path = urllib.quote(path)
  File "/usr/lib/python2.7/urllib.py", line 1299, in quote
    return ''.join(map(quoter, s))
KeyError: u'\xd8'
2016-11-02 14:31:24 [scrapy] DEBUG: Redirecting (301) to <GET https://en.wikipedia.org/wiki/Main_Page> from <GET https://en.wikipedia.org/>
2016-11-02 14:31:24 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/Main_Page> (referer: None)
2016-11-02 14:31:25 [scrapy] INFO: Closing spider (finished)
2016-11-02 14:31:25 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 804,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 25391,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/301': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 11, 2, 13, 31, 25, 113618),
 'log_count/DEBUG': 4,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2016, 11, 2, 13, 31, 24, 503170)}
2016-11-02 14:31:25 [scrapy] INFO: Spider closed (finished)

```

It does look like a bug in Python 2's `robotparser`:

```
>>> from robotparser import RobotFileParser
>>> lines = u'''User-agent: *
... Allow: /w/api.php?action=mobileview&
... Allow: /w/load.php?
... Allow: /api/rest_v1/?doc
... Disallow: /w/
... Disallow: /api/
... Disallow: /trap/
... #
... # ar:
... Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
... Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch'''
>>> rp = RobotFileParser('https://en.wikipedia.org/robots.txt')
>>> rp.parse(lines.splitlines())
/usr/lib/python2.7/urllib.py:1299: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return ''.join(map(quoter, s))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/robotparser.py", line 120, in parse
    entry.rulelines.append(RuleLine(line[1], False))
  File "/usr/lib/python2.7/robotparser.py", line 174, in __init__
    self.path = urllib.quote(path)
  File "/usr/lib/python2.7/urllib.py", line 1299, in quote
    return ''.join(map(quoter, s))
KeyError: u'\xd8'
>>> 
```

[This line](https://github.com/python/cpython/blob/2.7/Lib/robotparser.py#L111) seems to be the issue.
Changing to `line[1] = w3lib.url.safe_url_string(line[1].strip())` seems to fix it.

What do you suggest @redapple? Use a backport of Python 3.x robotparser, switch to reppy, create our own robots.txt parser, tell people to use Python 3, something else?

I think we can move to `reppy` for this, even without the [crawl-delay support](https://github.com/scrapy/scrapy/issues/892) for now.

reppy sounds fine, but I have a few small reservations about it:
- it seems reppy is under heavy refactoring right now;
- they combine robots.txt parsing and fetching in a same package, so they have requests in install_requires; it could be weird to have requests as a Scrapy dependency :)

oh right, reading some recent PRs.
Then it may be easier to use a custom RobotFileParser subclass for now.


Resolving patch:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -13,6 +13,7 @@
 from scrapy.http import Request
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.log import failure_to_exc_info
+from scrapy.utils.python import to_native_str
 
 logger = logging.getLogger(__name__)
 
@@ -40,7 +41,8 @@ def process_request(self, request, spider):
         return d
 
     def process_request_2(self, rp, request, spider):
-        if rp is not None and not rp.can_fetch(self._useragent, request.url):
+        if rp is not None and not rp.can_fetch(
+                 to_native_str(self._useragent), request.url):
             logger.debug("Forbidden by robots.txt: %(request)s",
                          {'request': request}, extra={'spider': spider})
             raise IgnoreRequest()
@@ -94,7 +96,9 @@ def _parse_robots(self, response, netloc):
                 # Running rp.parse() will set rp state from
                 # 'disallow all' to 'allow any'.
                 pass
-        rp.parse(body.splitlines())
+        # stdlib's robotparser expects native 'str' ;
+        # with unicode input, non-ASCII encoded bytes decoding fails in Python2
+        rp.parse(to_native_str(body).splitlines())
 
         rp_dfd = self._parsers[netloc]
         self._parsers[netloc] = rp


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 from __future__ import absolute_import
 import re
 from twisted.internet import reactor, error
@@ -30,11 +31,18 @@ def test_robotstxt_settings(self):
     def _get_successful_crawler(self):
         crawler = self.crawler
         crawler.settings.set('ROBOTSTXT_OBEY', True)
-        ROBOTS = re.sub(b'^\s+(?m)', b'', b'''
+        ROBOTS = re.sub(b'^\s+(?m)', b'', u'''
         User-Agent: *
         Disallow: /admin/
         Disallow: /static/
-        ''')
+
+        # taken from https://en.wikipedia.org/robots.txt
+        Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
+        Disallow: /wiki/Käyttäjä:
+
+        User-Agent: UnicödeBöt
+        Disallow: /some/randome/page.html
+        '''.encode('utf-8'))
         response = TextResponse('http://site.local/robots.txt', body=ROBOTS)
         def return_response(request, spider):
             deferred = Deferred()
@@ -48,7 +56,9 @@ def test_robotstxt(self):
         return DeferredList([
             self.assertNotIgnored(Request('http://site.local/allowed'), middleware),
             self.assertIgnored(Request('http://site.local/admin/main'), middleware),
-            self.assertIgnored(Request('http://site.local/static/'), middleware)
+            self.assertIgnored(Request('http://site.local/static/'), middleware),
+            self.assertIgnored(Request('http://site.local/wiki/K%C3%A4ytt%C3%A4j%C3%A4:'), middleware),
+            self.assertIgnored(Request(u'http://site.local/wiki/Käyttäjä:'), middleware)
         ], fireOnOneErrback=True)
 
     def test_robotstxt_ready_parser(self):