Repository: scrapy/scrapy
Pull request #6298 (2024-03-26 19:41:41)

Handle robots.txt files not UTF-8 encoded
Fixes https://github.com/scrapy/scrapy/issues/6292.

cc @Gallaecio

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/6298?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#6298](https://app.codecov.io/gh/scrapy/scrapy/pull/6298?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (e2ee984) into [master](https://app.codecov.io/gh/scrapy/scrapy/commit/02b97f98e74a994ad3e4d74e7ed55207e508a576?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (02b97f9) will **decrease** coverage by `45.88%`.
> The diff coverage is `0.00%`.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #6298       +/-   ##
===========================================
- Coverage   88.88%   43.00%   -45.88%     
===========================================
  Files         161      161               
  Lines       11971    11971               
  Branches     1929     1929               
===========================================
- Hits        10640     5148     -5492     
- Misses        980     6401     +5421     
- Partials      351      422       +71     
```

| [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6298?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/robotstxt.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6298?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3JvYm90c3R4dC5weQ==) | `40.44% <0.00%> (-44.95%)` | :arrow_down: |

... and [140 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6298/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)

</details>

Patch:
diff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.py
index 6ea2bfd97..ad06137e2 100644
--- a/scrapy/robotstxt.py
+++ b/scrapy/robotstxt.py
@@ -23,7 +23,7 @@ def decode_robotstxt(robotstxt_body, spider, to_native_str_type=False):
         if to_native_str_type:
             robotstxt_body = to_unicode(robotstxt_body)
         else:
-            robotstxt_body = robotstxt_body.decode("utf-8")
+            robotstxt_body = robotstxt_body.decode("utf-8", errors="ignore")
     except UnicodeDecodeError:
         # If we found garbage or robots.txt in an encoding other than UTF-8, disregard it.
         # Switch to 'allow all' state.


Test patch:
diff --git a/tests/test_robotstxt_interface.py b/tests/test_robotstxt_interface.py
index d7a923085..6ad30deed 100644
--- a/tests/test_robotstxt_interface.py
+++ b/tests/test_robotstxt_interface.py
@@ -1,5 +1,7 @@
 from twisted.trial import unittest
 
+from scrapy.robotstxt import decode_robotstxt
+
 
 def reppy_available():
     # check if reppy parser is installed
@@ -141,6 +143,25 @@ class BaseRobotParserTest:
         )
 
 
+class DecodeRobotsTxtTest(unittest.TestCase):
+    def test_native_string_conversion(self):
+        robotstxt_body = "User-agent: *\nDisallow: /\n".encode("utf-8")
+        decoded_content = decode_robotstxt(
+            robotstxt_body, spider=None, to_native_str_type=True
+        )
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+    def test_decode_utf8(self):
+        robotstxt_body = "User-agent: *\nDisallow: /\n".encode("utf-8")
+        decoded_content = decode_robotstxt(robotstxt_body, spider=None)
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+    def test_decode_non_utf8(self):
+        robotstxt_body = b"User-agent: *\n\xFFDisallow: /\n"
+        decoded_content = decode_robotstxt(robotstxt_body, spider=None)
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+
 class PythonRobotParserTest(BaseRobotParserTest, unittest.TestCase):
     def setUp(self):
         from scrapy.robotstxt import PythonRobotParser
diff --git a/tests/test_selector.py b/tests/test_selector.py
index 1b5f3f018..85527bba9 100644
--- a/tests/test_selector.py
+++ b/tests/test_selector.py
@@ -80,10 +80,10 @@ class SelectorTestCase(unittest.TestCase):
         meta = (
             '<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">'
         )
-        head = f"<head>{meta}</head>"
+        head = "<head>" + meta + "</head>"
         body_content = '<span id="blank">\xa3</span>'
-        body = f"<body>{body_content}</body>"
-        html = f"<html>{head}{body}</html>"
+        body = "<body>" + body_content + "</body>"
+        html = "<html>" + head + body + "</html>"
         encoding = "utf-8"
         html_utf8 = html.encode(encoding)