Repository: scrapy/scrapy
Pull request #5807 (2023-01-26 16:18:46)

Don't check robots.txt for local files
Currently robots.txt is attempted to be checked for local files (and data urls). You can reproduce with:

```python
from scrapy import Spider


class TestSpider(Spider):
    name = "test"
    start_urls = [
        "data:text/plain,Hello World data",
        "file:///home/cj/hello world.txt",
    ]

    def parse(self, response, **kwargs):
        print(response.text)
```

`$ pipenv run scrapy runspider test.py -s ROBOTSTXT_OBEY=True`

```
2023-01-26 15:50:47 [scrapy.core.engine] INFO: Spider opened
2023-01-26 15:50:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-01-26 15:50:47 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET data:/robots.txt>: invalid data URI
Traceback (most recent call last):
  File "/home/cj/.local/share/virtualenvs/alltheplaces-E_LDghuV/lib/python3.10/site-packages/w3lib/url.py", line 504, in parse_data_uri
    is_base64, data = uri.split(b",", 1)
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cj/.local/share/virtualenvs/alltheplaces-E_LDghuV/lib/python3.10/site-packages/scrapy/core/downloader/middleware.py", line 49, in process_request
    return (yield download_func(request=request, spider=spider))
  File "/home/cj/.local/share/virtualenvs/alltheplaces-E_LDghuV/lib/python3.10/site-packages/twisted/internet/defer.py", line 206, in maybeDeferred
    result = f(*args, **kwargs)
  File "/home/cj/.local/share/virtualenvs/alltheplaces-E_LDghuV/lib/python3.10/site-packages/scrapy/core/downloader/handlers/datauri.py", line 13, in download_request
    uri = parse_data_uri(request.url)
  File "/home/cj/.local/share/virtualenvs/alltheplaces-E_LDghuV/lib/python3.10/site-packages/w3lib/url.py", line 506, in parse_data_uri
    raise ValueError("invalid data URI")
ValueError: invalid data URI
2023-01-26 15:50:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET data:text/plain,Hello%20World%20data> (referer: None)
Hello World data
2023-01-26 15:50:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET file:///robots.txt> (failed 1 times): [Errno 2] No such file or directory: '/robots.txt'
2023-01-26 15:50:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET file:///robots.txt> (failed 2 times): [Errno 2] No such file or directory: '/robots.txt'
2023-01-26 15:50:50 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET file:///robots.txt> (failed 3 times): [Errno 2] No such file or directory: '/robots.txt'
2023-01-26 15:50:50 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET file:///robots.txt>: [Errno 2] No such file or directory: '/robots.txt'
Traceback (most recent call last):
  File "/home/cj/Documents/Projects/scrapy/scrapy/core/downloader/middleware.py", line 52, in process_request
    return (yield download_func(request=request, spider=spider))
  File "/home/cj/.local/share/virtualenvs/scrapy-kVY4T-qA/lib/python3.10/site-packages/twisted/internet/defer.py", line 206, in maybeDeferred
    result = f(*args, **kwargs)
  File "/home/cj/Documents/Projects/scrapy/scrapy/core/downloader/handlers/file.py", line 15, in download_request
    body = Path(filepath).read_bytes()
  File "/usr/lib/python3.10/pathlib.py", line 1126, in read_bytes
    with self.open(mode='rb') as f:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/robots.txt'
2023-01-26 15:50:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET file:///home/cj/hello%20world.txt> (referer: None)
Hello World file

2023-01-26 15:50:50 [scrapy.core.engine] INFO: Closing spider (finished)
```

`$ pipenv run scrapy runspider test.py -s ROBOTSTXT_OBEY=False`

```
2023-01-26 15:47:56 [scrapy.core.engine] INFO: Spider opened
2023-01-26 15:47:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-01-26 15:47:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET data:text/plain,Hello%20World%20data> (referer: None)
Hello World data
2023-01-26 15:47:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET file:///home/cj/hello%20world.txt> (referer: None)
Hello World file

2023-01-26 15:47:57 [scrapy.core.engine] INFO: Closing spider (finished)
```

This patch silently ignores those urls so the rest of the project can continue to honour robots.txt.


Patch:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index 326c35290..8e9beeeef 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -38,6 +38,8 @@ class RobotsTxtMiddleware:
     def process_request(self, request, spider):
         if request.meta.get("dont_obey_robotstxt"):
             return
+        if request.url.startswith("data:") or request.url.startswith("file:"):
+            return
         d = maybeDeferred(self.robot_parser, request, spider)
         d.addCallback(self.process_request_2, request, spider)
         return d


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
index ac08c6006..fd27e637d 100644
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -214,6 +214,19 @@ Disallow: /some/randome/page.html
         middleware.process_request_2(rp, Request("http://site.local/allowed"), None)
         rp.allowed.assert_called_once_with("http://site.local/allowed", "Examplebot")
 
+    def test_robotstxt_local_file(self):
+        middleware = RobotsTxtMiddleware(self._get_emptybody_crawler())
+        assert not middleware.process_request(
+            Request("data:text/plain,Hello World data"), None
+        )
+        assert not middleware.process_request(
+            Request("file:///tests/sample_data/test_site/nothinghere.html"), None
+        )
+        assert isinstance(
+            middleware.process_request(Request("http://site.local/allowed"), None),
+            Deferred,
+        )
+
     def assertNotIgnored(self, request, middleware):
         spider = None  # not actually used
         dfd = maybeDeferred(middleware.process_request, request, spider)