internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: scrapy/scrapy

Issue:
may be 'accessible'?
in the function [request_fingerprint](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/request.py) ，‘accesible’ may be ‘accessible’ in comments. OCD  XD..



Discussion:
Did you know you can edit the file directly on Github (there is an edit button on the top right side) and easily create a pull request with the change? :)
> Did you know you can edit the file directly on Github (there is an edit button on the top right side) and easily create a pull request with the change? :)

Thanks for your reply , I didn't do this just because it's tiny or there is another word 'accesible'?
This issue will be rendered obsolete after #3975

Resolving patch:
diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py
--- a/scrapy/utils/request.py
+++ b/scrapy/utils/request.py
@@ -30,7 +30,7 @@ def request_fingerprint(request, include_headers=None):
     and are equivalent (ie. they should return the same response).
 
     Another example are cookies used to store session ids. Suppose the
-    following page is only accesible to authenticated users:
+    following page is only accessible to authenticated users:
 
     http://www.example.com/members/offers.html
 


Test patch:
Repository: scrapy/scrapy

Issue:
Handle it gracefully when start_url is used instead of start_urls
Over the last year I’ve seen a few cases ([recent example](https://stackoverflow.com/q/58664004/939364)) of this, people missing the `s` at the end of the `start_urls`.

It may be nice to find a way to gracefully let the developer know where the issue is, why there is no crawling happening.


Discussion:
As in the scrappy many files are autogenerated when you start a new project and there start_urls is already declared by default , so if we use start_url instead of start_urls then there would be no crawling.

Resolving patch:
diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py
--- a/scrapy/spiders/__init__.py
+++ b/scrapy/spiders/__init__.py
@@ -58,6 +58,11 @@ def _set_crawler(self, crawler):
 
     def start_requests(self):
         cls = self.__class__
+        if not self.start_urls and hasattr(self, 'start_url'):
+            raise AttributeError(
+                "Crawling could not start: 'start_urls' not found "
+                "or empty (but found 'start_url' attribute instead, "
+                "did you miss an 's'?)")
         if method_is_overridden(cls, Spider, 'make_requests_from_url'):
             warnings.warn(
                 "Spider.make_requests_from_url method is deprecated; it "


Test patch:
diff --git a/tests/test_spider.py b/tests/test_spider.py
--- a/tests/test_spider.py
+++ b/tests/test_spider.py
@@ -385,6 +385,14 @@ def test_follow_links_attribute_population(self):
         self.assertTrue(hasattr(spider, '_follow_links'))
         self.assertFalse(spider._follow_links)
 
+    def test_start_url(self):
+        spider = self.spider_class("example.com")
+        spider.start_url = 'https://www.example.com'
+
+        with self.assertRaisesRegex(AttributeError,
+                                    r'^Crawling could not start.*$'):
+            list(spider.start_requests())
+
 
 class SitemapSpiderTest(SpiderTest):
Repository: scrapy/scrapy

Issue:
Consider making METAREFRESH_IGNORE_TAGS an empty list by default
As a way to allow users to fix #1422, #3768 introduced the `METAREFRESH_IGNORE_TAGS` setting.

To keep backward compatibility, the setting was introduced with `['script', 'noscript']` as default value. However, to reproduce the behavior of web browsers, it seems the right value would be `[]`.

Should we switch the default value of the `METAREFRESH_IGNORE_TAGS` setting to `[]`, even though the change breaks backward compatibility?


Discussion:

Resolving patch:
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -225,7 +225,7 @@
 MEMUSAGE_WARNING_MB = 0
 
 METAREFRESH_ENABLED = True
-METAREFRESH_IGNORE_TAGS = ['script', 'noscript']
+METAREFRESH_IGNORE_TAGS = []
 METAREFRESH_MAXDELAY = 100
 
 NEWSPIDER_MODULE = ''


Test patch:
diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py
--- a/tests/test_downloadermiddleware_redirect.py
+++ b/tests/test_downloadermiddleware_redirect.py
@@ -300,19 +300,21 @@ def test_ignore_tags_default(self):
         body = ('''<noscript><meta http-equiv="refresh" '''
                 '''content="0;URL='http://example.org/newpage'"></noscript>''')
         rsp = HtmlResponse(req.url, body=body.encode())
-        response = self.mw.process_response(req, rsp, self.spider)
-        assert isinstance(response, Response)
+        req2 = self.mw.process_response(req, rsp, self.spider)
+        assert isinstance(req2, Request)
+        self.assertEqual(req2.url, 'http://example.org/newpage')
 
-    def test_ignore_tags_empty_list(self):
-        crawler = get_crawler(Spider, {'METAREFRESH_IGNORE_TAGS': []})
+    def test_ignore_tags_1_x_list(self):
+        """Test that Scrapy 1.x behavior remains possible"""
+        settings = {'METAREFRESH_IGNORE_TAGS': ['script', 'noscript']}
+        crawler = get_crawler(Spider, settings)
         mw = MetaRefreshMiddleware.from_crawler(crawler)
         req = Request(url='http://example.org')
         body = ('''<noscript><meta http-equiv="refresh" '''
                 '''content="0;URL='http://example.org/newpage'"></noscript>''')
         rsp = HtmlResponse(req.url, body=body.encode())
-        req2 = mw.process_response(req, rsp, self.spider)
-        assert isinstance(req2, Request)
-        self.assertEqual(req2.url, 'http://example.org/newpage')
+        response = mw.process_response(req, rsp, self.spider)
+        assert isinstance(response, Response)
 
 if __name__ == "__main__":
     unittest.main()
Repository: scrapy/scrapy

Issue:
Mac OS X, OS X → macOS
We have a few references in the documentation where we use the old name of that OS. We should update them.


Discussion:
Hi, I'd like to take this issue.

Thanks for the interest @tomaskrcka. No need to ask first, please go ahead 😄 

Resolving patch:
diff --git a/scrapy/extensions/memusage.py b/scrapy/extensions/memusage.py
--- a/scrapy/extensions/memusage.py
+++ b/scrapy/extensions/memusage.py
@@ -47,7 +47,7 @@ def from_crawler(cls, crawler):
     def get_virtual_size(self):
         size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
         if sys.platform != 'darwin':
-            # on Mac OS X ru_maxrss is in bytes, on Linux it is in KB
+            # on macOS ru_maxrss is in bytes, on Linux it is in KB
             size *= 1024
         return size
 


Test patch:
Repository: scrapy/scrapy

Issue:
[DOCS] Use HTTPS
Reference:
 - https://httpsiseasy.com
 - https://letsencrypt.org


Discussion:
Could you run this again with the `--verbose` flag perhaps? ENOCONTEXT
I don’t get what the issue is about, or how your comment, @nyov, is related to it :confused: 

https://docs.scrapy.org/ uses HTTPS, so I guess the issue is not about that.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -281,6 +281,7 @@
 
 intersphinx_mapping = {
     'coverage': ('https://coverage.readthedocs.io/en/stable', None),
+    'cssselect': ('https://cssselect.readthedocs.io/en/latest', None),
     'pytest': ('https://docs.pytest.org/en/latest', None),
     'python': ('https://docs.python.org/3', None),
     'sphinx': ('https://www.sphinx-doc.org/en/master', None),


Test patch:
Repository: scrapy/scrapy

Issue:
Remove SCRAPY_SELECTORS_BACKEND from scrapy.settings.deprecated
There is no trace of `SCRAPY_SELECTORS_BACKEND` in the code anymore, so that line should go.

It would be a good chance to review the rest of the lines in that file, some others may be worth removing as well.

Related to https://github.com/scrapy/scrapy/issues/4356


Discussion:
Hi, may I work on this?  Its my first time contributing to a project.

Resolving patch:
diff --git a/scrapy/settings/deprecated.py b/scrapy/settings/deprecated.py
--- a/scrapy/settings/deprecated.py
+++ b/scrapy/settings/deprecated.py
@@ -9,10 +9,8 @@
     ('ENCODING_ALIASES', 'no longer needed (encoding discovery uses w3lib now)'),
     ('STATS_ENABLED', 'no longer supported (change STATS_CLASS instead)'),
     ('SQLITE_DB', 'no longer supported'),
-    ('SELECTORS_BACKEND', 'use SCRAPY_SELECTORS_BACKEND environment variable instead'),
     ('AUTOTHROTTLE_MIN_DOWNLOAD_DELAY', 'use DOWNLOAD_DELAY instead'),
     ('AUTOTHROTTLE_MAX_CONCURRENCY', 'use CONCURRENT_REQUESTS_PER_DOMAIN instead'),
-    ('AUTOTHROTTLE_MAX_CONCURRENCY', 'use CONCURRENT_REQUESTS_PER_DOMAIN instead'),
     ('REDIRECT_MAX_METAREFRESH_DELAY', 'use METAREFRESH_MAXDELAY instead'),
     ('LOG_UNSERIALIZABLE_REQUESTS', 'use SCHEDULER_DEBUG instead'),
 ]


Test patch:
Repository: scrapy/scrapy

Issue:
follow_all fails with an empty list of URLs
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

`follow_all` with an empty list of urls fails with `ValueError('Please supply exactly one of the following arguments: urls, css, xpath')`

What I would expect instead is just an empty generator of requests. 

### Steps to Reproduce

```py
class Spider(scrapy.Spider):

    def parse(self, response):
        yield from response.follow_all([], self.parse)
```

**Expected behavior:** 

No error is raised

**Actual behavior:**

`ValueError('Please supply exactly one of the following arguments: urls, css, xpath')` exception is raised. 


### Versions

2.0

### Additional context

I think the solution is just a matter of changing this line: https://github.com/scrapy/scrapy/blob/master/scrapy/http/response/text.py#L191



Discussion:

Resolving patch:
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -188,9 +188,11 @@ def follow_all(self, urls=None, callback=None, method='GET', headers=None, body=
         selectors from which links cannot be obtained (for instance, anchor tags without an
         ``href`` attribute)
         """
-        arg_count = len(list(filter(None, (urls, css, xpath))))
-        if arg_count != 1:
-            raise ValueError('Please supply exactly one of the following arguments: urls, css, xpath')
+        arguments = [x for x in (urls, css, xpath) if x is not None]
+        if len(arguments) != 1:
+            raise ValueError(
+                "Please supply exactly one of the following arguments: urls, css, xpath"
+            )
         if not urls:
             if css:
                 urls = self.css(css)


Test patch:
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -215,6 +215,10 @@ def test_follow_all_links(self):
         links = map(Link, absolute)
         self._assert_followed_all_urls(links, absolute)
 
+    def test_follow_all_empty(self):
+        r = self.response_class("http://example.com")
+        self.assertEqual([], list(r.follow_all([])))
+
     def test_follow_all_invalid(self):
         r = self.response_class("http://example.com")
         if self.response_class == Response:
Repository: scrapy/scrapy

Issue:
Update redirect link from Python 2 docs to Python 3 docs
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

The documentation has many redirect links to `https://docs.python.org/2/*`. As Scrapy now supports Python 3.5+ I think they should be changed to `https://docs.python.org/3/*` respectively.

### Steps to Reproduce

1. Go to [JsonItemExported](https://docs.scrapy.org/en/latest/topics/exporters.html#jsonitemexporter) and click on `JSONEncoder`
2. You will be redirected to [Python 2](https://docs.python.org/2/library/json.html#json.JSONEncoder) docs.

Above steps can be repeated for many other pages in Scrapy docs.

**Expected behavior:** Redirected to latest Python 3 docs

**Actual behavior:** Redirected to Python 2 docs

**Reproduces how often:** 100%

### Versions

INFO:scrapy.utils.log:Scrapy 2.0.0 started (bot: scrapybot)
INFO:scrapy.utils.log:Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.6.10 (default, Feb 11 2020, 21:11:31) - [GCC 7.4.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-5.3.0-40-generic-x86_64-with-debian-buster-sid
DEBUG:scrapy.utils.log:Using reactor: twisted.internet.epollreactor.EPollReactor
Scrapy       : 2.0.0
lxml         : 4.4.1.0
libxml2      : 2.9.9
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 19.10.0
Python       : 3.6.10 (default, Feb 11 2020, 21:11:31) - [GCC 7.4.0]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019)
cryptography : 2.8
Platform     : Linux-5.3.0-40-generic-x86_64-with-debian-buster-sid

### Additional context

I am willing to fix this issue, waiting for an approval.


Discussion:

Resolving patch:
diff --git a/scrapy/item.py b/scrapy/item.py
--- a/scrapy/item.py
+++ b/scrapy/item.py
@@ -121,9 +121,7 @@ def copy(self):
         return self.__class__(self)
 
     def deepcopy(self):
-        """Return a `deep copy`_ of this item.
-
-        .. _deep copy: https://docs.python.org/library/copy.html#copy.deepcopy
+        """Return a :func:`~copy.deepcopy` of this item.
         """
         return deepcopy(self)
 


Test patch:
Repository: scrapy/scrapy

Issue:
zope.interface 5.0.0 unsupported
zope.interface 5.0.0 was released yesterday and with it some of our tests fail: https://travis-ci.org/github/scrapy/scrapy/jobs/664648176 (actually it's just one test, `CrawlerRunnerTestCase.test_spider_manager_verify_interface`).

I don't know if it's just a problem with that test or something in Scrapy is now broken too. 

Changelog: https://zopeinterface.readthedocs.io/en/latest/changes.html


Discussion:
@wRaR As of [zope.interface#171](https://github.com/zopefoundation/zope.interface/issues/171) they have made "`verifyObject` and `verifyClass` report all errors, if the candidate object has multiple detectable violations. Previously they reported only the first error." 
They achieve this behaviour using `MultipleInvalid` (refer [zope/interface/exceptions.py](https://github.com/zopefoundation/zope.interface/blob/master/src/zope/interface/exceptions.py#L233-L263)). 

Looking at their implementation, I think we have to catch both  `DoesNotImplement` and `MultipleInvalid` (refer [_verfiy](https://github.com/zopefoundation/zope.interface/blob/master/src/zope/interface/verify.py#L74-L92) implementation) in our tests and [crawler.py](https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L129-L135) i.e everywhere `verifyObject` or `verifyClass` are used.

Upd: Shall I send a PR to fix this?

Resolving patch:
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -4,7 +4,15 @@
 import warnings
 
 from twisted.internet import defer
-from zope.interface.verify import DoesNotImplement, verifyClass
+from zope.interface.exceptions import DoesNotImplement
+
+try:
+    # zope >= 5.0 only supports MultipleInvalid
+    from zope.interface.exceptions import MultipleInvalid
+except ImportError:
+    MultipleInvalid = None
+
+from zope.interface.verify import verifyClass
 
 from scrapy import signals, Spider
 from scrapy.core.engine import ExecutionEngine
@@ -124,9 +132,10 @@ def _get_spider_loader(settings):
         """ Get SpiderLoader instance from settings """
         cls_path = settings.get('SPIDER_LOADER_CLASS')
         loader_cls = load_object(cls_path)
+        excs = (DoesNotImplement, MultipleInvalid) if MultipleInvalid else DoesNotImplement
         try:
             verifyClass(ISpiderLoader, loader_cls)
-        except DoesNotImplement:
+        except excs:
             warnings.warn(
                 'SPIDER_LOADER_CLASS (previously named SPIDER_MANAGER_CLASS) does '
                 'not fully implement scrapy.interfaces.ISpiderLoader interface. '


Test patch:
Repository: scrapy/scrapy

Issue:
The documentation cannot be built with Sphinx 3



Discussion:
can I take this up?
Seems like this is an issue in the `sphinx-hoverxref` extension: https://github.com/readthedocs/sphinx-hoverxref/issues/45.
If we can't wait for the upcoming release as mentioned in https://github.com/readthedocs/sphinx-hoverxref/issues/45#issuecomment-609462935, we might want to try temporarily disabling it in [our configuration](https://github.com/scrapy/scrapy/blob/2.0.1/docs/conf.py#L31).

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -295,3 +295,12 @@
 # ------------------------------------
 
 hoverxref_auto_ref = True
+hoverxref_project = "scrapy"
+hoverxref_version = release
+hoverxref_role_types = {
+    "class": "tooltip",
+    "confval": "tooltip",
+    "hoverxref": "tooltip",
+    "mod": "tooltip",
+    "ref": "tooltip",
+}


Test patch:
Repository: scrapy/scrapy

Issue:
responsetypes.py ResponseTypes.from_headers typo as 'Content-type'
Shall this line:

https://github.com/scrapy/scrapy/blob/8845773d44329194ee73fa4985a94e768bf664e7/scrapy/responsetypes.py#L74

be changed to below?
`
content_type=headers[b'Content-Type'],
`
I guess this typo is with all the versions until today.


Discussion:
It does not need to be modified from a functional perspective (`headers` is a case-insensitive dictionary), but we could change it for consistency.
Can i take this up?

Resolving patch:
diff --git a/scrapy/responsetypes.py b/scrapy/responsetypes.py
--- a/scrapy/responsetypes.py
+++ b/scrapy/responsetypes.py
@@ -71,7 +71,7 @@ def from_headers(self, headers):
         cls = Response
         if b'Content-Type' in headers:
             cls = self.from_content_type(
-                content_type=headers[b'Content-type'],
+                content_type=headers[b'Content-Type'],
                 content_encoding=headers.get(b'Content-Encoding')
             )
         if cls is Response and b'Content-Disposition' in headers:


Test patch:
Repository: scrapy/scrapy

Issue:
Fix the hoverxref configuration
> You shouldn't override hoverxref_version and hoverxref_project since they are taken automatically from Read the Docs.
>
> If you want to avoid your CI failing because of this, you can define the environment variables as Read the Docs does:
> 
> READTHEDOCS_PROJECT=scrapy
> READTHEDOCS_VERSION=''
> 
> With the current configuration, all the versions built on Read the Docs will point to a different version on Read the Docs and this will conflict. For example, current master version in Read the Docs defines hoverxref_version='2.0.0' but that version does not exist on Read the Docs and the tooltip does not known where to get the content from.

@humitos at https://github.com/scrapy/scrapy/pull/4480#discussion_r409026912


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -295,8 +295,6 @@
 # ------------------------------------
 
 hoverxref_auto_ref = True
-hoverxref_project = "scrapy"
-hoverxref_version = release
 hoverxref_role_types = {
     "class": "tooltip",
     "confval": "tooltip",


Test patch:
Repository: scrapy/scrapy

Issue:
Deprecate Response.body_as_unicode



Discussion:

Resolving patch:
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -5,6 +5,7 @@
 See documentation in docs/topics/request-response.rst
 """
 
+import warnings
 from contextlib import suppress
 from typing import Generator
 from urllib.parse import urljoin
@@ -14,6 +15,7 @@
                             http_content_type_encoding, resolve_encoding)
 from w3lib.html import strip_html5_whitespace
 
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Request
 from scrapy.http.response import Response
 from scrapy.utils.python import memoizemethod_noargs, to_unicode
@@ -61,6 +63,9 @@ def _declared_encoding(self):
 
     def body_as_unicode(self):
         """Return body as unicode"""
+        warnings.warn('Response.body_as_unicode() is deprecated, '
+                      'please use Response.text instead.',
+                      ScrapyDeprecationWarning)
         return self.text
 
     @property


Test patch:
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -1,7 +1,9 @@
 import unittest
+from warnings import catch_warnings
 
 from w3lib.encoding import resolve_encoding
 
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import (Request, Response, TextResponse, HtmlResponse,
                          XmlResponse, Headers)
 from scrapy.selector import Selector
@@ -660,6 +662,13 @@ def test_follow_all_too_many_arguments(self):
         with self.assertRaises(ValueError):
             response.follow_all(css='a[href*="example.com"]', xpath='//a[contains(@href, "example.com")]')
 
+    def test_body_as_unicode_deprecation_warning(self):
+        with catch_warnings(record=True) as warnings:
+            r1 = self.response_class("http://www.example.com", body=u'Hello', encoding='utf-8')
+            self.assertEqual(r1.body_as_unicode(), u'Hello')
+            self.assertEqual(len(warnings), 1)
+            self.assertEqual(warnings[0].category, ScrapyDeprecationWarning)
+
 
 class HtmlResponseTest(TextResponseTest):
Repository: scrapy/scrapy

Issue:
Extend hoverxref_roles
@humitos [suggested](https://github.com/scrapy/scrapy/issues/4475#issuecomment-613350667) extending the `hoverxref_roles` setting of the corresponding Sphinx extension so that the display-on-hover behavior of the documentations works for things like signal or setting references.


Discussion:
I'd like to give this a shot.
@redrolor are you still working on this? If not I can take it.
As @redrolor didn't reply I have sent a pull request for this.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -300,3 +300,4 @@
     "mod": "tooltip",
     "ref": "tooltip",
 }
+hoverxref_roles = ['command', 'reqmeta', 'setting', 'signal']


Test patch: