Repository: scrapy/scrapy

Issue:
PY3: SitemapSpider fail to extract sitemap URLs from robots.txt in Scrapy 1.1.0rc1
## Environment
- Mac OS X 10.10.5
- Python 3.4.2
- Scrapy 1.1.0rc1
## Steps to Reproduce
1. Save the following spider as `sitemap_spider.py`.
   
   ``` py
   from scrapy.spiders import SitemapSpider
   
   
   class BlogSitemapSpider(SitemapSpider):
      name = "blog_sitemap"
      allowed_domains = ["blog.scrapinghub.com"]
   
      sitemap_urls = [
          'https://blog.scrapinghub.com/robots.txt',
      ]
      sitemap_rules = [
          (r'/2016/', 'parse'),
      ]
   
      def parse(self, response):
          pass
   ```
2. Run the following command.
   
   ```
   $ scrapy runspider sitemap_spider.py
   ```
## Expected Results

The spider crawl several pages according to the sitemaps without error.
## Actual Results

The spider fail to extract sitemap URLs from robots.txt. No pages are crawled.

```
$ scrapy runspider sitemap_spider.py 2016-02-06 20:55:51 [scrapy] INFO: Scrapy 1.1.0rc1 started (bot: scrapybot)
2016-02-06 20:55:51 [scrapy] INFO: Overridden settings: {}
2016-02-06 20:55:52 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logstats.LogStats']
2016-02-06 20:55:52 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-02-06 20:55:52 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-02-06 20:55:52 [scrapy] INFO: Enabled item pipelines:
[]
2016-02-06 20:55:52 [scrapy] INFO: Spider opened
2016-02-06 20:55:52 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-02-06 20:55:52 [scrapy] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com/robots.txt> (referer: None)
2016-02-06 20:55:52 [scrapy] ERROR: Spider error processing <GET https://blog.scrapinghub.com/robots.txt> (referer: None)
Traceback (most recent call last):
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spiders/sitemap.py", line 35, in _parse_sitemap
    for url in sitemap_urls_from_robots(response.body):
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/utils/sitemap.py", line 42, in sitemap_urls_from_robots
    if line.lstrip().startswith('Sitemap:'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
2016-02-06 20:55:52 [scrapy] INFO: Closing spider (finished)
2016-02-06 20:55:52 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 231,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 1009,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 2, 6, 11, 55, 52, 570098),
 'log_count/DEBUG': 1,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/TypeError': 1,
 'start_time': datetime.datetime(2016, 2, 6, 11, 55, 52, 97618)}
2016-02-06 20:55:52 [scrapy] INFO: Spider closed (finished)
```

PY3: SitemapSpider fail to extract sitemap URLs from robots.txt in Scrapy 1.1.0rc1
## Environment
- Mac OS X 10.10.5
- Python 3.4.2
- Scrapy 1.1.0rc1
## Steps to Reproduce
1. Save the following spider as `sitemap_spider.py`.
   
   ``` py
   from scrapy.spiders import SitemapSpider
   
   
   class BlogSitemapSpider(SitemapSpider):
      name = "blog_sitemap"
      allowed_domains = ["blog.scrapinghub.com"]
   
      sitemap_urls = [
          'https://blog.scrapinghub.com/robots.txt',
      ]
      sitemap_rules = [
          (r'/2016/', 'parse'),
      ]
   
      def parse(self, response):
          pass
   ```
2. Run the following command.
   
   ```
   $ scrapy runspider sitemap_spider.py
   ```
## Expected Results

The spider crawl several pages according to the sitemaps without error.
## Actual Results

The spider fail to extract sitemap URLs from robots.txt. No pages are crawled.

```
$ scrapy runspider sitemap_spider.py 2016-02-06 20:55:51 [scrapy] INFO: Scrapy 1.1.0rc1 started (bot: scrapybot)
2016-02-06 20:55:51 [scrapy] INFO: Overridden settings: {}
2016-02-06 20:55:52 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logstats.LogStats']
2016-02-06 20:55:52 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-02-06 20:55:52 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-02-06 20:55:52 [scrapy] INFO: Enabled item pipelines:
[]
2016-02-06 20:55:52 [scrapy] INFO: Spider opened
2016-02-06 20:55:52 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-02-06 20:55:52 [scrapy] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com/robots.txt> (referer: None)
2016-02-06 20:55:52 [scrapy] ERROR: Spider error processing <GET https://blog.scrapinghub.com/robots.txt> (referer: None)
Traceback (most recent call last):
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/spiders/sitemap.py", line 35, in _parse_sitemap
    for url in sitemap_urls_from_robots(response.body):
  File "/private/tmp/scrapy1.1/venv/lib/python3.4/site-packages/scrapy/utils/sitemap.py", line 42, in sitemap_urls_from_robots
    if line.lstrip().startswith('Sitemap:'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
2016-02-06 20:55:52 [scrapy] INFO: Closing spider (finished)
2016-02-06 20:55:52 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 231,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 1009,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 2, 6, 11, 55, 52, 570098),
 'log_count/DEBUG': 1,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/TypeError': 1,
 'start_time': datetime.datetime(2016, 2, 6, 11, 55, 52, 97618)}
2016-02-06 20:55:52 [scrapy] INFO: Spider closed (finished)
```



Discussion:

Resolving patch:
diff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.py
--- a/scrapy/spiders/sitemap.py
+++ b/scrapy/spiders/sitemap.py
@@ -32,7 +32,7 @@ def start_requests(self):
 
     def _parse_sitemap(self, response):
         if response.url.endswith('/robots.txt'):
-            for url in sitemap_urls_from_robots(response.body):
+            for url in sitemap_urls_from_robots(response.text):
                 yield Request(url, callback=self._parse_sitemap)
         else:
             body = self._get_sitemap_body(response)


Test patch:
diff --git a/tests/test_spider.py b/tests/test_spider.py
--- a/tests/test_spider.py
+++ b/tests/test_spider.py
@@ -328,6 +328,18 @@ def test_get_sitemap_body_xml_url_compressed(self):
         r = Response(url="http://www.example.com/sitemap.xml.gz", body=self.GZBODY)
         self.assertSitemapBody(r, self.BODY)
 
+    def test_get_sitemap_urls_from_robotstxt(self):
+        robots = b"""# Sitemap files
+Sitemap: http://example.com/sitemap.xml
+Sitemap: http://example.com/sitemap-product-index.xml
+"""
+
+        r = TextResponse(url="http://www.example.com/robots.txt", body=robots)
+        spider = self.spider_class("example.com")
+        self.assertEqual([req.url for req in spider._parse_sitemap(r)],
+                         ['http://example.com/sitemap.xml',
+                          'http://example.com/sitemap-product-index.xml'])
+
 
 class BaseSpiderDeprecationTest(unittest.TestCase):