Repository: scrapy/scrapy

Issue:
KeyError in robotstxt middleware
I'm getting these errors in robots.txt middleware:

```
2016-01-27 16:18:21 [scrapy.core.scraper] ERROR: Error downloading <GET http://yellowpages.co.th>
Traceback (most recent call last):
  File "/Users/kmike/envs/scraping/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/kmike/svn/scrapy/scrapy/downloadermiddlewares/robotstxt.py", line 65, in robot_parser
    if isinstance(self._parsers[netloc], Deferred):
KeyError: 'yellowpages.co.th'
```

It looks like https://github.com/scrapy/scrapy/pull/1473 caused it (I can't get this issue in Scrapy 1.0.4, but it present in Scrapy master). It happens when page failed to download and HTTP cache is enabled. I haven't debugged it further.

Discussion:

Patch:
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
deleted file mode 100644
index 95b4a7e3c..000000000
--- a/CODE_OF_CONDUCT.md
+++ /dev/null
@@ -1,50 +0,0 @@
-# Contributor Code of Conduct
-
-As contributors and maintainers of this project, and in the interest of
-fostering an open and welcoming community, we pledge to respect all people who
-contribute through reporting issues, posting feature requests, updating
-documentation, submitting pull requests or patches, and other activities.
-
-We are committed to making participation in this project a harassment-free
-experience for everyone, regardless of level of experience, gender, gender
-identity and expression, sexual orientation, disability, personal appearance,
-body size, race, ethnicity, age, religion, or nationality.
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery
-* Personal attacks
-* Trolling or insulting/derogatory comments
-* Public or private harassment
-* Publishing other's private information, such as physical or electronic
-  addresses, without explicit permission
-* Other unethical or unprofessional conduct
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-By adopting this Code of Conduct, project maintainers commit themselves to
-fairly and consistently applying these principles to every aspect of managing
-this project. Project maintainers who do not follow or enforce the Code of
-Conduct may be permanently removed from the project team.
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community.
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting a project maintainer at opensource@scrapinghub.com. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. Maintainers are
-obligated to maintain confidentiality with regard to the reporter of an
-incident.
-
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 1.3.0, available at
-[http://contributor-covenant.org/version/1/3/0/][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/3/0/
diff --git a/README.rst b/README.rst
index 3e050bb1e..6cbed75ee 100644
--- a/README.rst
+++ b/README.rst
@@ -73,12 +73,6 @@ See http://scrapy.org/community/
 Contributing
 ============
 
-Please note that this project is released with a Contributor Code of Conduct
-(see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).
-
-By participating in this project you agree to abide by its terms.
-Please report unacceptable behavior to opensource@scrapinghub.com.
-
 See http://doc.scrapy.org/en/master/contributing.html
 
 Companies using Scrapy
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 82e674cee..ea64d1599 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -445,10 +445,10 @@ Response objects
 
     .. attribute:: Response.body
 
-        The body of this Response. Keep in mind that Response.body
-        is always a bytes object. If you want the unicode version use
-        :attr:`TextResponse.text` (only available in :class:`TextResponse`
-        and subclasses).
+        A str containing the body of this Response. Keep in mind that Response.body
+        is always a str. If you want the unicode version use
+        :meth:`TextResponse.body_as_unicode` (only available in
+        :class:`TextResponse` and subclasses).
 
         This attribute is read-only. To change the body of a Response use
         :meth:`replace`.
@@ -542,21 +542,6 @@ TextResponse objects
     :class:`TextResponse` objects support the following attributes in addition
     to the standard :class:`Response` ones:
 
-    .. attribute:: TextResponse.text
-
-       Response body, as unicode.
-
-       The same as ``response.body.decode(response.encoding)``, but the
-       result is cached after the first call, so you can access
-       ``response.text`` multiple times without extra overhead.
-
-       .. note::
-
-            ``unicode(response.body)`` is not a correct way to convert response
-            body to unicode: you would be using the system default encoding
-            (typically `ascii`) instead of the response encoding.
-
-
     .. attribute:: TextResponse.encoding
 
        A string with the encoding of this response. The encoding is resolved by
@@ -583,6 +568,20 @@ TextResponse objects
     :class:`TextResponse` objects support the following methods in addition to
     the standard :class:`Response` ones:
 
+    .. method:: TextResponse.body_as_unicode()
+
+        Returns the body of the response as unicode. This is equivalent to::
+
+            response.body.decode(response.encoding)
+
+        But **not** equivalent to::
+
+            unicode(response.body)
+
+        Since, in the latter case, you would be using the system default encoding
+        (typically `ascii`) to convert the body to unicode, instead of the response
+        encoding.
+
     .. method:: TextResponse.xpath(query)
 
         A shortcut to ``TextResponse.selector.xpath(query)``::
@@ -595,11 +594,6 @@ TextResponse objects
 
             response.css('p')
 
-    .. method:: TextResponse.body_as_unicode()
-
-        The same as :attr:`text`, but available as a method. This method is
-        kept for backwards compatibility; please prefer ``response.text``.
-
 
 HtmlResponse objects
 --------------------
diff --git a/scrapy/downloadermiddlewares/ajaxcrawl.py b/scrapy/downloadermiddlewares/ajaxcrawl.py
index da373eca2..6b543b823 100644
--- a/scrapy/downloadermiddlewares/ajaxcrawl.py
+++ b/scrapy/downloadermiddlewares/ajaxcrawl.py
@@ -63,7 +63,7 @@ class AjaxCrawlMiddleware(object):
         Return True if a page without hash fragment could be "AJAX crawlable"
         according to https://developers.google.com/webmasters/ajax-crawling/docs/getting-started.
         """
-        body = response.text[:self.lookup_bytes]
+        body = response.body_as_unicode()[:self.lookup_bytes]
         return _has_ajaxcrawlable_meta(body)
 
 
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index d4a33dc36..6fdba90cc 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -83,8 +83,8 @@ class RobotsTxtMiddleware(object):
     def _parse_robots(self, response, netloc):
         rp = robotparser.RobotFileParser(response.url)
         body = ''
-        if hasattr(response, 'text'):
-            body = response.text
+        if hasattr(response, 'body_as_unicode'):
+            body = response.body_as_unicode()
         else: # last effort try
             try:
                 body = response.body.decode('utf-8')
@@ -101,4 +101,6 @@ class RobotsTxtMiddleware(object):
         rp_dfd.callback(rp)
 
     def _robots_error(self, failure, netloc):
-        self._parsers.pop(netloc).callback(None)
+        rp_dfd = self._parsers[netloc]
+        self._parsers[netloc] = None
+        rp_dfd.callback(None)
diff --git a/scrapy/http/request/form.py b/scrapy/http/request/form.py
index 2862dc096..5501634d3 100644
--- a/scrapy/http/request/form.py
+++ b/scrapy/http/request/form.py
@@ -64,8 +64,8 @@ def _urlencode(seq, enc):
 
 def _get_form(response, formname, formid, formnumber, formxpath):
     """Find the form element """
-    root = create_root_node(response.text, lxml.html.HTMLParser,
-                            base_url=get_base_url(response))
+    text = response.body_as_unicode()
+    root = create_root_node(text, lxml.html.HTMLParser, base_url=get_base_url(response))
     forms = root.xpath('//form')
     if not forms:
         raise ValueError("No <form> element found in %s" % response)
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
index 9c667ab7e..1c416bf82 100644
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -67,11 +67,6 @@ class TextResponse(Response):
             self._cached_ubody = html_to_unicode(charset, self.body)[1]
         return self._cached_ubody
 
-    @property
-    def text(self):
-        """ Body as unicode """
-        return self.body_as_unicode()
-
     def urljoin(self, url):
         """Join this Response's url with a possible relative url to form an
         absolute interpretation of the latter."""
diff --git a/scrapy/selector/unified.py b/scrapy/selector/unified.py
index 15f3d26df..5d77f7624 100644
--- a/scrapy/selector/unified.py
+++ b/scrapy/selector/unified.py
@@ -60,7 +60,7 @@ class Selector(_ParselSelector, object_ref):
             response = _response_from_text(text, st)
 
         if response is not None:
-            text = response.text
+            text = response.body_as_unicode()
             kwargs.setdefault('base_url', response.url)
 
         self.response = response
diff --git a/scrapy/utils/iterators.py b/scrapy/utils/iterators.py
index 73857b410..b0688791e 100644
--- a/scrapy/utils/iterators.py
+++ b/scrapy/utils/iterators.py
@@ -137,7 +137,7 @@ def _body_or_str(obj, unicode=True):
         if not unicode:
             return obj.body
         elif isinstance(obj, TextResponse):
-            return obj.text
+            return obj.body_as_unicode()
         else:
             return obj.body.decode('utf-8')
     elif isinstance(obj, six.text_type):
diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py
index 73db2641e..c4ad52f14 100644
--- a/scrapy/utils/response.py
+++ b/scrapy/utils/response.py
@@ -25,7 +25,7 @@ _baseurl_cache = weakref.WeakKeyDictionary()
 def get_base_url(response):
     """Return the base url of the given response, joined with the response url"""
     if response not in _baseurl_cache:
-        text = response.text[0:4096]
+        text = response.body_as_unicode()[0:4096]
         _baseurl_cache[response] = html.get_base_url(text, response.url,
             response.encoding)
     return _baseurl_cache[response]
@@ -37,7 +37,7 @@ _metaref_cache = weakref.WeakKeyDictionary()
 def get_meta_refresh(response):
     """Parse the http-equiv refrsh parameter from the given response"""
     if response not in _metaref_cache:
-        text = response.text[0:4096]
+        text = response.body_as_unicode()[0:4096]
         text = _noscript_re.sub(u'', text)
         text = _script_re.sub(u'', text)
         _metaref_cache[response] = html.get_meta_refresh(text, response.url,


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
index 5f45dcb82..f2e94e171 100644
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -123,6 +123,18 @@ class RobotsTxtMiddlewareTest(unittest.TestCase):
         deferred.addCallback(lambda _: self.assertTrue(middleware._logerror.called))
         return deferred
 
+    def test_robotstxt_immediate_error(self):
+        self.crawler.settings.set('ROBOTSTXT_OBEY', True)
+        err = error.DNSLookupError('Robotstxt address not found')
+        def immediate_failure(request, spider):
+            deferred = Deferred()
+            deferred.errback(failure.Failure(err))
+            return deferred
+        self.crawler.engine.download.side_effect = immediate_failure
+
+        middleware = RobotsTxtMiddleware(self.crawler)
+        return self.assertNotIgnored(Request('http://site.local'), middleware)
+
     def test_ignore_robotstxt_request(self):
         self.crawler.settings.set('ROBOTSTXT_OBEY', True)
         def ignore_request(request, spider):
diff --git a/tests/test_engine.py b/tests/test_engine.py
index baf6ef1bf..9f2c02bff 100644
--- a/tests/test_engine.py
+++ b/tests/test_engine.py
@@ -55,11 +55,12 @@ class TestSpider(Spider):
 
     def parse_item(self, response):
         item = self.item_cls()
-        m = self.name_re.search(response.text)
+        body = response.body_as_unicode()
+        m = self.name_re.search(body)
         if m:
             item['name'] = m.group(1)
         item['url'] = response.url
-        m = self.price_re.search(response.text)
+        m = self.price_re.search(body)
         if m:
             item['price'] = m.group(1)
         return item
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
index c7f36687a..710a5b29d 100644
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -107,11 +107,9 @@ class BaseResponseTest(unittest.TestCase):
             body_bytes = body
 
         assert isinstance(response.body, bytes)
-        assert isinstance(response.text, six.text_type)
         self._assert_response_encoding(response, encoding)
         self.assertEqual(response.body, body_bytes)
         self.assertEqual(response.body_as_unicode(), body_unicode)
-        self.assertEqual(response.text, body_unicode)
 
     def _assert_response_encoding(self, response, encoding):
         self.assertEqual(response.encoding, resolve_encoding(encoding))
@@ -173,10 +171,6 @@ class TextResponseTest(BaseResponseTest):
         self.assertTrue(isinstance(r1.body_as_unicode(), six.text_type))
         self.assertEqual(r1.body_as_unicode(), unicode_string)
 
-        # check response.text
-        self.assertTrue(isinstance(r1.text, six.text_type))
-        self.assertEqual(r1.text, unicode_string)
-
     def test_encoding(self):
         r1 = self.response_class("http://www.example.com", headers={"Content-type": ["text/html; charset=utf-8"]}, body=b"\xc2\xa3")
         r2 = self.response_class("http://www.example.com", encoding='utf-8', body=u"\xa3")
@@ -225,12 +219,12 @@ class TextResponseTest(BaseResponseTest):
                                  headers={"Content-type": ["text/html; charset=utf-8"]},
                                  body=b"\xef\xbb\xbfWORD\xe3\xab")
         self.assertEqual(r6.encoding, 'utf-8')
-        self.assertEqual(r6.text, u'WORD\ufffd\ufffd')
+        self.assertEqual(r6.body_as_unicode(), u'WORD\ufffd\ufffd')
 
     def test_bom_is_removed_from_body(self):
         # Inferring encoding from body also cache decoded body as sideeffect,
         # this test tries to ensure that calling response.encoding and
-        # response.text in indistint order doesn't affect final
+        # response.body_as_unicode() in indistint order doesn't affect final
         # values for encoding and decoded body.
         url = 'http://example.com'
         body = b"\xef\xbb\xbfWORD"
@@ -239,9 +233,9 @@ class TextResponseTest(BaseResponseTest):
         # Test response without content-type and BOM encoding
         response = self.response_class(url, body=body)
         self.assertEqual(response.encoding, 'utf-8')
-        self.assertEqual(response.text, u'WORD')
+        self.assertEqual(response.body_as_unicode(), u'WORD')
         response = self.response_class(url, body=body)
-        self.assertEqual(response.text, u'WORD')
+        self.assertEqual(response.body_as_unicode(), u'WORD')
         self.assertEqual(response.encoding, 'utf-8')
 
         # Body caching sideeffect isn't triggered when encoding is declared in
@@ -249,9 +243,9 @@ class TextResponseTest(BaseResponseTest):
         # body
         response = self.response_class(url, headers=headers, body=body)
         self.assertEqual(response.encoding, 'utf-8')
-        self.assertEqual(response.text, u'WORD')
+        self.assertEqual(response.body_as_unicode(), u'WORD')
         response = self.response_class(url, headers=headers, body=body)
-        self.assertEqual(response.text, u'WORD')
+        self.assertEqual(response.body_as_unicode(), u'WORD')
         self.assertEqual(response.encoding, 'utf-8')
 
     def test_replace_wrong_encoding(self):
@@ -259,18 +253,18 @@ class TextResponseTest(BaseResponseTest):
         r = self.response_class("http://www.example.com", encoding='utf-8', body=b'PREFIX\xe3\xabSUFFIX')
         # XXX: Policy for replacing invalid chars may suffer minor variations
         # but it should always contain the unicode replacement char (u'\ufffd')
-        assert u'\ufffd' in r.text, repr(r.text)
-        assert u'PREFIX' in r.text, repr(r.text)
-        assert u'SUFFIX' in r.text, repr(r.text)
+        assert u'\ufffd' in r.body_as_unicode(), repr(r.body_as_unicode())
+        assert u'PREFIX' in r.body_as_unicode(), repr(r.body_as_unicode())
+        assert u'SUFFIX' in r.body_as_unicode(), repr(r.body_as_unicode())
 
         # Do not destroy html tags due to encoding bugs
         r = self.response_class("http://example.com", encoding='utf-8', \
                 body=b'\xf0<span>value</span>')
-        assert u'<span>value</span>' in r.text, repr(r.text)
+        assert u'<span>value</span>' in r.body_as_unicode(), repr(r.body_as_unicode())
 
         # FIXME: This test should pass once we stop using BeautifulSoup's UnicodeDammit in TextResponse
-        #r = self.response_class("http://www.example.com", body=b'PREFIX\xe3\xabSUFFIX')
-        #assert u'\ufffd' in r.text, repr(r.text)
+        #r = self.response_class("http://www.example.com", body='PREFIX\xe3\xabSUFFIX')
+        #assert u'\ufffd' in r.body_as_unicode(), repr(r.body_as_unicode())
 
     def test_selector(self):
         body = b"<html><head><title>Some page</title><body></body></html>"