Repository: scrapy/scrapy (Python)

Issue:
Handle robots.txt files not utf-8 encoded
## Summary

`robots.txt` files which are not `utf-8` encoded make `scrapy` raise an `UnicodeDecodeError` atm.

```
.venv/lib/python3.11/site-packages/scrapy/robotstxt.py", line 15, in decode_robotstxt
    robotstxt_body = robotstxt_body.decode("utf-8")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 46: invalid continuation byte
```

## Motivation

Prevents from being able to manage all scrapers in an automated fashion, like described in "Describe alternatives you've considered". 

## Describe alternatives you've considered

Check `robots.txt` file manually for the scraper in question, disable `robots.txt` file parsing globally using `ROBOTSTXT_OBEY`. Prevents to being able to process all scrapers in an automated fashion however.

## Additional context




Discussion:
We can probably follow [Google’s approach](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt):

> Similarly, if the character encoding of the robots. txt file isn't UTF-8, Google may ignore characters that are not part of the UTF-8 range, potentially rendering robots. txt rules invalid.

So fixing this requires changing the offending line to:

```python
robotstxt_body.decode("utf-8", errors="ignore")
```

And implementing a test for it.
@Gallaecio I would like to work on this.
Please, feel free to open a PR. Thanks!



Patch:
diff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.py
index 6ea2bfd977e..ad06137e28b 100644
--- a/scrapy/robotstxt.py
+++ b/scrapy/robotstxt.py
@@ -23,7 +23,7 @@ def decode_robotstxt(robotstxt_body, spider, to_native_str_type=False):
         if to_native_str_type:
             robotstxt_body = to_unicode(robotstxt_body)
         else:
-            robotstxt_body = robotstxt_body.decode("utf-8")
+            robotstxt_body = robotstxt_body.decode("utf-8", errors="ignore")
     except UnicodeDecodeError:
         # If we found garbage or robots.txt in an encoding other than UTF-8, disregard it.
         # Switch to 'allow all' state.


Test patch:
diff --git a/tests/test_robotstxt_interface.py b/tests/test_robotstxt_interface.py
index d7a923085ac..6ad30deeda0 100644
--- a/tests/test_robotstxt_interface.py
+++ b/tests/test_robotstxt_interface.py
@@ -1,5 +1,7 @@
 from twisted.trial import unittest
 
+from scrapy.robotstxt import decode_robotstxt
+
 
 def reppy_available():
     # check if reppy parser is installed
@@ -141,6 +143,25 @@ def test_unicode_url_and_useragent(self):
         )
 
 
+class DecodeRobotsTxtTest(unittest.TestCase):
+    def test_native_string_conversion(self):
+        robotstxt_body = "User-agent: *\nDisallow: /\n".encode("utf-8")
+        decoded_content = decode_robotstxt(
+            robotstxt_body, spider=None, to_native_str_type=True
+        )
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+    def test_decode_utf8(self):
+        robotstxt_body = "User-agent: *\nDisallow: /\n".encode("utf-8")
+        decoded_content = decode_robotstxt(robotstxt_body, spider=None)
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+    def test_decode_non_utf8(self):
+        robotstxt_body = b"User-agent: *\n\xFFDisallow: /\n"
+        decoded_content = decode_robotstxt(robotstxt_body, spider=None)
+        self.assertEqual(decoded_content, "User-agent: *\nDisallow: /\n")
+
+
 class PythonRobotParserTest(BaseRobotParserTest, unittest.TestCase):
     def setUp(self):
         from scrapy.robotstxt import PythonRobotParser