internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: scrapy/scrapy

Issue:
Scrapy warns about itself
I'm using Scrapy 2.6.2. Created a new project, run a spider, and there is 

```
2022-08-02 23:50:26 [py.warnings] WARNING: [redacted]/python3.8/site-packages/scrapy/core/engine.py:279: ScrapyDeprecationWarning: Passing a 'spider' argument to ExecutionEngine.download is deprecated
  return self.download(result, spider) if isinstance(result, Request) else result
```

message in logs. We also have the same warning in our testing suite, but I haven't realized it's not a test-only issue.


Discussion:

Resolving patch:
diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py
--- a/scrapy/core/engine.py
+++ b/scrapy/core/engine.py
@@ -257,9 +257,7 @@ def _schedule_request(self, request: Request, spider: Spider) -> None:
 
     def download(self, request: Request, spider: Optional[Spider] = None) -> Deferred:
         """Return a Deferred which fires with a Response as result, only downloader middlewares are applied"""
-        if spider is None:
-            spider = self.spider
-        else:
+        if spider is not None:
             warnings.warn(
                 "Passing a 'spider' argument to ExecutionEngine.download is deprecated",
                 category=ScrapyDeprecationWarning,
@@ -267,7 +265,7 @@ def download(self, request: Request, spider: Optional[Spider] = None) -> Deferre
             )
             if spider is not self.spider:
                 logger.warning("The spider '%s' does not match the open spider", spider.name)
-        if spider is None:
+        if self.spider is None:
             raise RuntimeError(f"No open spider to crawl: {request}")
         return self._download(request, spider).addBoth(self._downloaded, request, spider)
 
@@ -278,11 +276,14 @@ def _downloaded(
         self.slot.remove_request(request)
         return self.download(result, spider) if isinstance(result, Request) else result
 
-    def _download(self, request: Request, spider: Spider) -> Deferred:
+    def _download(self, request: Request, spider: Optional[Spider]) -> Deferred:
         assert self.slot is not None  # typing
 
         self.slot.add_request(request)
 
+        if spider is None:
+            spider = self.spider
+
         def _on_success(result: Union[Response, Request]) -> Union[Response, Request]:
             if not isinstance(result, (Response, Request)):
                 raise TypeError(f"Incorrect type: expected Response or Request, got {type(result)}: {result!r}")


Test patch:
Repository: scrapy/scrapy

Issue:
Support for Twisted 22.8.0
Twisted 22.8.0 was released recently, and it says:

> Twisted now works with Cryptography versions 37 and above, and as a result, its minimum TLS protocol version has been upgraded to TLSv1.2.

Consequently, tests on some envs, including 3.8, now fail because they install older cryptography.


Discussion:
Also, I haven't yet checked but it may mean that Scrapy with this Twisted version will no longer work with some servers (it also needs to be checked if it's a hard restriction or the TLS version can be lowered  with settings)
We've found an interesting data point regarding the "support older TLS" part: https://chromestatus.com/feature/5759116003770368 says older TLS is no longer supported in Chromium and I can confirm it using https://badssl.com/ (https://tls-v1-1.badssl.com:1011/). While we usually strive to support as many real-world servers as possible, it should be a lower priority to support servers that are already dropped from a major browser.
Same for Firefox. “This website might not support the TLS 1.2 protocol, which is the minimum version supported by Firefox.” Let’s burn TLS 1.1 and earlier with fire :fire: .

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -20,13 +20,13 @@ def has_environment_marker_platform_impl_support():
 
 install_requires = [
     'Twisted>=18.9.0',
-    'cryptography>=2.8',
+    'cryptography>=3.3',
     'cssselect>=0.9.1',
     'itemloaders>=1.0.1',
     'parsel>=1.5.0',
-    'pyOpenSSL>=19.1.0',
+    'pyOpenSSL>=21.0.0',
     'queuelib>=1.4.2',
-    'service_identity>=16.0.0',
+    'service_identity>=18.1.0',
     'w3lib>=1.17.0',
     'zope.interface>=5.1.0',
     'protego>=0.1.15',


Test patch:
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
--- a/tests/test_crawler.py
+++ b/tests/test_crawler.py
@@ -327,7 +327,7 @@ def test_reactor_default(self):
 
     def test_reactor_default_twisted_reactor_select(self):
         log = self.run_script('reactor_default_twisted_reactor_select.py')
-        if platform.system() == 'Windows':
+        if platform.system() in ['Windows', 'Darwin']:
             # The goal of this test function is to test that, when a reactor is
             # installed (the default one here) and a different reactor is
             # configured (select here), an error raises.
Repository: scrapy/scrapy

Issue:
Silence the `filelock` logger
After we started using `tldextract` we sometimes get log messages from `filelock` with the DEBUG level, it makes sense to silence them like we do for some other libraries in https://github.com/scrapy/scrapy/blob/fe60c1224e39aa3d85b20afd54566f135d9de085/scrapy/utils/log.py#L45-L59


Discussion:

Resolving patch:
diff --git a/scrapy/utils/log.py b/scrapy/utils/log.py
--- a/scrapy/utils/log.py
+++ b/scrapy/utils/log.py
@@ -46,6 +46,9 @@ def filter(self, record):
     'version': 1,
     'disable_existing_loggers': False,
     'loggers': {
+        'filelock': {
+            'level': 'ERROR',
+        },
         'hpack': {
             'level': 'ERROR',
         },


Test patch:
Repository: scrapy/scrapy

Issue:
pypy3-pinned OpenSSL error
Yet another CI issue, visible in https://github.com/scrapy/scrapy/actions/runs/3849823417/jobs/6559259481

> /home/runner/work/scrapy/scrapy/.tox/pypy3-pinned/site-packages/cryptography/hazmat/bindings/_openssl.pypy37-pp73-x86_64-linux-gnu.so: undefined symbol: FIPS_mode

This may be a problem with that specific binary distribution (it's `cryptography==3.3`), not sure why it worked before, maybe something was rebuilt recently. 


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -19,7 +19,7 @@ def has_environment_marker_platform_impl_support():
 
 install_requires = [
     'Twisted>=18.9.0',
-    'cryptography>=3.3',
+    'cryptography>=3.4.6',
     'cssselect>=0.9.1',
     'itemloaders>=1.0.1',
     'parsel>=1.5.0',


Test patch:
Repository: scrapy/scrapy

Issue:
TLS logging broken with new cryptography
https://github.com/pyca/cryptography/pull/8391 dropped `SSL_get_server_tmp_key()` so we need to disable the code that uses it if it's not available.


Discussion:

Resolving patch:
diff --git a/scrapy/utils/ssl.py b/scrapy/utils/ssl.py
--- a/scrapy/utils/ssl.py
+++ b/scrapy/utils/ssl.py
@@ -20,6 +20,9 @@ def x509name_to_string(x509name):
 
 def get_temp_key_info(ssl_object):
     # adapted from OpenSSL apps/s_cb.c::ssl_print_tmp_key()
+    if not hasattr(pyOpenSSLutil.lib, "SSL_get_server_tmp_key"):
+        # removed in cryptography 40.0.0
+        return None
     temp_key_p = pyOpenSSLutil.ffi.new("EVP_PKEY **")
     if not pyOpenSSLutil.lib.SSL_get_server_tmp_key(ssl_object, temp_key_p):
         return None


Test patch:
Repository: scrapy/scrapy

Issue:
_sent_failed cut the errback chain in MailSender
`MailSender._sent_failed` return `None`, instead of `failure`.  This cut the errback call chain, making impossible to detect in the code fail in the mails in client code.



Discussion:
Well this is fair bug, and simple to fix (I guess). How should we go forward with it, though?
We return/raise the failure (not sure which, design-wise) but this is not backwards-compatible. What other things must we do around the fix to deal with the lack of compatibility between versions?
@Gallaecio @wRAR 
Hello @aplanas, I am interested in working on this issue, Could you please assign it to me?

Resolving patch:
diff --git a/scrapy/mail.py b/scrapy/mail.py
--- a/scrapy/mail.py
+++ b/scrapy/mail.py
@@ -164,6 +164,7 @@ def _sent_failed(self, failure, to, cc, subject, nattachs):
                 "mailerr": errstr,
             },
         )
+        return failure
 
     def _sendmail(self, to_addrs, msg):
         from twisted.internet import reactor


Test patch:
Repository: scrapy/scrapy

Issue:
Remove pkg_resources usage
Importing pkg_resources raises a deprecation warning now so it makes sense to remove it. We can even use importlib without checking if it's available as we no longer support 3.7. I see only `pkg_resources.parse_version` and `pkg_resources.iter_entry_points` used in Scrapy, I think they can be replaced by things from `packaging` and `importlib.metadata` respectively.


Discussion:

Resolving patch:
diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py
--- a/scrapy/cmdline.py
+++ b/scrapy/cmdline.py
@@ -3,8 +3,7 @@
 import inspect
 import os
 import sys
-
-import pkg_resources
+from importlib.metadata import entry_points
 
 import scrapy
 from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
@@ -49,7 +48,7 @@ def _get_commands_from_module(module, inproject):
 
 def _get_commands_from_entry_points(inproject, group="scrapy.commands"):
     cmds = {}
-    for entry_point in pkg_resources.iter_entry_points(group):
+    for entry_point in entry_points().get(group, {}):
         obj = entry_point.load()
         if inspect.isclass(obj):
             cmds[entry_point.name] = obj()
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,23 +1,10 @@
 from pathlib import Path
 
-from pkg_resources import parse_version
-from setuptools import __version__ as setuptools_version
 from setuptools import find_packages, setup
 
 version = (Path(__file__).parent / "scrapy/VERSION").read_text("ascii").strip()
 
 
-def has_environment_marker_platform_impl_support():
-    """Code extracted from 'pytest/setup.py'
-    https://github.com/pytest-dev/pytest/blob/7538680c/setup.py#L31
-
-    The first known release to support environment marker with range operators
-    it is 18.5, see:
-    https://setuptools.readthedocs.io/en/latest/history.html#id235
-    """
-    return parse_version(setuptools_version) >= parse_version("18.5")
-
-
 install_requires = [
     "Twisted>=18.9.0",
     "cryptography>=36.0.0",
@@ -36,19 +23,10 @@ def has_environment_marker_platform_impl_support():
     "tldextract",
     "lxml>=4.4.1",
 ]
-extras_require = {}
-cpython_dependencies = [
-    "PyDispatcher>=2.0.5",
-]
-if has_environment_marker_platform_impl_support():
-    extras_require[
-        ':platform_python_implementation == "CPython"'
-    ] = cpython_dependencies
-    extras_require[':platform_python_implementation == "PyPy"'] = [
-        "PyPyDispatcher>=2.1.0",
-    ]
-else:
-    install_requires.extend(cpython_dependencies)
+extras_require = {
+    ':platform_python_implementation == "CPython"': ["PyDispatcher>=2.0.5"],
+    ':platform_python_implementation == "PyPy"': ["PyPyDispatcher>=2.1.0"],
+}
 
 
 setup(


Test patch:
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
--- a/tests/test_crawler.py
+++ b/tests/test_crawler.py
@@ -6,7 +6,7 @@
 import warnings
 from pathlib import Path
 
-from pkg_resources import parse_version
+from packaging.version import parse as parse_version
 from pytest import mark, raises
 from twisted import version as twisted_version
 from twisted.internet import defer
Repository: scrapy/scrapy

Issue:
Re-enable deps on 3.12 when ready
- [x] bpython (requires greenlet)
- [x] uvloop
- [x] pyftpdlib

Related: https://github.com/scrapy/scrapy/pull/6083


Discussion:
Could you please explain the issue in detail?
This issue requires time, and may span over a month, so it may not be a good choice as a first contribution to Scrapy.
Neither greenlet nor uvloop support 3.12 yet so no action is needed here yet.

Resolving patch:
diff --git a/scrapy/contracts/__init__.py b/scrapy/contracts/__init__.py
--- a/scrapy/contracts/__init__.py
+++ b/scrapy/contracts/__init__.py
@@ -41,7 +41,9 @@ def wrapper(response, **cb_kwargs):
                     cb_result = cb(response, **cb_kwargs)
                     if isinstance(cb_result, (AsyncGenerator, CoroutineType)):
                         raise TypeError("Contracts don't support async callbacks")
-                    return list(iterate_spider_output(cb_result))
+                    return list(  # pylint: disable=return-in-finally
+                        iterate_spider_output(cb_result)
+                    )
 
             request.callback = wrapper
 
@@ -68,7 +70,7 @@ def wrapper(response, **cb_kwargs):
                 else:
                     results.addSuccess(self.testcase_post)
                 finally:
-                    return output
+                    return output  # pylint: disable=return-in-finally
 
             request.callback = wrapper
 


Test patch:
diff --git a/tests/requirements.txt b/tests/requirements.txt
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -7,8 +7,7 @@ pytest-cov==4.0.0
 pytest-xdist
 sybil >= 1.3.0  # https://github.com/cjw296/sybil/issues/20#issuecomment-605433422
 testfixtures
-# uvloop currently doesn't build on 3.12
-uvloop; platform_system != "Windows" and python_version < "3.12"
+uvloop; platform_system != "Windows"
 
 bpython  # optional for shell wrapper tests
 brotli; implementation_name != 'pypy'  # optional for HTTP compress downloader middleware tests
Repository: scrapy/scrapy

Issue:
Deprecate Reppy support
https://github.com/scrapy/scrapy/issues/5226#issuecomment-897621867


Discussion:
https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#built-in-downloader-middleware-reference @Gallaecio  I am seeing that **Reappy Parser** Do I just need to mention there that "only work with Python 3.8 and earlier" or do I need to make some other changes. Thanks
That’s it.

Please, do it as part of https://github.com/scrapy/scrapy/issues/5226, this ticket is for potential future work deprecating support for Reppy.

[And now the latest `gcc` also cannot build the package either](https://github.com/seomoz/reppy/issues/132) :cry: 
@Gallaecio do we need to update once more the docs to mention gcc's current issue for previous versions of python? Or can we close as-is?
Neither. I would not bother changing the docs, but this needs to remain open until we actually deprecate Reppy support, to be done around Oct 14th 2023. i.e. 1 year before [Python 3.8 end of life](https://endoflife.date/python), as [agreed](https://github.com/scrapy/scrapy/issues/5226#issuecomment-897621867).
Nice, thanks @Gallaecio !

![giphy](https://user-images.githubusercontent.com/51033921/225641825-40192744-db04-43e0-85b4-4b66bad5245f.gif)

This is now ready.

Resolving patch:
diff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.py
--- a/scrapy/robotstxt.py
+++ b/scrapy/robotstxt.py
@@ -1,7 +1,9 @@
 import logging
 import sys
 from abc import ABCMeta, abstractmethod
+from warnings import warn
 
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.utils.python import to_unicode
 
 logger = logging.getLogger(__name__)
@@ -79,6 +81,7 @@ def allowed(self, url, user_agent):
 
 class ReppyRobotParser(RobotParser):
     def __init__(self, robotstxt_body, spider):
+        warn("ReppyRobotParser is deprecated.", ScrapyDeprecationWarning, stacklevel=2)
         from reppy.robots import Robots
 
         self.spider = spider


Test patch:
Repository: scrapy/scrapy

Issue:
Use `defusedxml.xmlrpc`
https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b411-import-xmlrpclib

https://github.com/tiran/defusedxml?tab=readme-ov-file#defusedxmlxmlrpc


Discussion:

Resolving patch:
diff --git a/scrapy/http/request/rpc.py b/scrapy/http/request/rpc.py
--- a/scrapy/http/request/rpc.py
+++ b/scrapy/http/request/rpc.py
@@ -7,9 +7,13 @@
 import xmlrpc.client as xmlrpclib
 from typing import Any, Optional
 
+import defusedxml.xmlrpc
+
 from scrapy.http.request import Request
 from scrapy.utils.python import get_func_args
 
+defusedxml.xmlrpc.monkey_patch()
+
 DUMPS_ARGS = get_func_args(xmlrpclib.dumps)
 
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,6 +22,7 @@
     "packaging",
     "tldextract",
     "lxml>=4.4.1",
+    "defusedxml>=0.7.1",
 ]
 extras_require = {
     ':platform_python_implementation == "CPython"': ["PyDispatcher>=2.0.5"],


Test patch:
Repository: scrapy/scrapy

Issue:
'json' selector type not documented
https://docs.scrapy.org/en/latest/topics/selectors.html

`type defines the selector type, it can be "html", "xml" or None (default).`

But my tests are now failing because it's returning `json` type for actual json docs now.


Discussion:
can I send PR for this?

@mohmad-null 
> But my tests are now failing because it's returning json type for actual json docs now.

Can You provide a bit more detailed info about Your tests that failed?
Does it look like https://github.com/scrapy/scrapy/issues/5923 or https://github.com/scrapy/parsel/issues/275 ?
> Can You provide a bit more detailed info about Your tests that failed?

The failing tests were specific to my code. Basically I expected one selector type and now get another one, so the code went down a different channel and I got a different result. Nothing scrapy related there.



Resolving patch:
diff --git a/scrapy/selector/unified.py b/scrapy/selector/unified.py
--- a/scrapy/selector/unified.py
+++ b/scrapy/selector/unified.py
@@ -47,7 +47,7 @@ class Selector(_ParselSelector, object_ref):
     ``response`` isn't available. Using ``text`` and ``response`` together is
     undefined behavior.
 
-    ``type`` defines the selector type, it can be ``"html"``, ``"xml"``
+    ``type`` defines the selector type, it can be ``"html"``, ``"xml"``, ``"json"``
     or ``None`` (default).
 
     If ``type`` is ``None``, the selector automatically chooses the best type


Test patch:
Repository: scrapy/scrapy

Issue:
Set METAREFRESH_IGNORE_TAGS to ["noscript"] by default
I was wrong in https://github.com/scrapy/scrapy/issues/3844. The default value should be `["noscript"]`, to deal with [antibot behaviors](https://github.com/scrapy/scrapy/commit/ec1ef0235f9deee0c263c9b31652d3e74a754acc).

Found by @mukthy.
Set METAREFRESH_IGNORE_TAGS to ["noscript"] by default
I was wrong in https://github.com/scrapy/scrapy/issues/3844. The default value should be `["noscript"]`, to deal with [antibot behaviors](https://github.com/scrapy/scrapy/commit/ec1ef0235f9deee0c263c9b31652d3e74a754acc).

Found by @mukthy.


Discussion:
on it
on it

Resolving patch:
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -239,7 +239,7 @@
 MEMUSAGE_WARNING_MB = 0
 
 METAREFRESH_ENABLED = True
-METAREFRESH_IGNORE_TAGS = []
+METAREFRESH_IGNORE_TAGS = ["noscript"]
 METAREFRESH_MAXDELAY = 100
 
 NEWSPIDER_MODULE = ""


Test patch:
diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py
--- a/tests/test_downloadermiddleware_redirect.py
+++ b/tests/test_downloadermiddleware_redirect.py
@@ -395,9 +395,8 @@ def test_ignore_tags_default(self):
             """content="0;URL='http://example.org/newpage'"></noscript>"""
         )
         rsp = HtmlResponse(req.url, body=body.encode())
-        req2 = self.mw.process_response(req, rsp, self.spider)
-        assert isinstance(req2, Request)
-        self.assertEqual(req2.url, "http://example.org/newpage")
+        response = self.mw.process_response(req, rsp, self.spider)
+        assert isinstance(response, Response)
 
     def test_ignore_tags_1_x_list(self):
         """Test that Scrapy 1.x behavior remains possible"""
Repository: scrapy/scrapy

Issue:
Make certain args of `ScrapyAgent` and `TunnelingAgent` required
`scrapy.core.downloader.handlers.http11.ScrapyAgent` and `scrapy.core.downloader.handlers.http11.TunnelingAgent` take some args (`contextFactory` in both and `crawler` in `ScrapyAgent`) that are currently declared optional but in fact need to be passed as the code assumes they aren't None. So ideally they should be declared as required (but that requires reordering args and/or making all args of those constructors keyword-only so there is no clear and simple fix).
Make certain args of `ScrapyAgent` and `TunnelingAgent` required
`scrapy.core.downloader.handlers.http11.ScrapyAgent` and `scrapy.core.downloader.handlers.http11.TunnelingAgent` take some args (`contextFactory` in both and `crawler` in `ScrapyAgent`) that are currently declared optional but in fact need to be passed as the code assumes they aren't None. So ideally they should be declared as required (but that requires reordering args and/or making all args of those constructors keyword-only so there is no clear and simple fix).


Discussion:
Creating PR for this
Creating PR for this

Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -239,15 +239,14 @@ class TunnelingAgent(Agent):
 
     def __init__(
         self,
+        *,
         reactor: ReactorBase,
         proxyConf: Tuple[str, int, Optional[bytes]],
-        contextFactory: Optional[IPolicyForHTTPS] = None,
+        contextFactory: IPolicyForHTTPS,
         connectTimeout: Optional[float] = None,
         bindAddress: Optional[bytes] = None,
         pool: Optional[HTTPConnectionPool] = None,
     ):
-        # TODO make this arg required instead
-        assert contextFactory is not None
         super().__init__(reactor, contextFactory, connectTimeout, bindAddress, pool)
         self._proxyConf: Tuple[str, int, Optional[bytes]] = proxyConf
         self._contextFactory: IPolicyForHTTPS = contextFactory
@@ -335,18 +334,16 @@ class ScrapyAgent:
 
     def __init__(
         self,
-        contextFactory: Optional[IPolicyForHTTPS] = None,
+        *,
+        contextFactory: IPolicyForHTTPS,
         connectTimeout: float = 10,
         bindAddress: Optional[bytes] = None,
         pool: Optional[HTTPConnectionPool] = None,
         maxsize: int = 0,
         warnsize: int = 0,
         fail_on_dataloss: bool = True,
-        crawler: Optional[Crawler] = None,
+        crawler: Crawler,
     ):
-        # TODO make these args required instead
-        assert contextFactory is not None
-        assert crawler is not None
         self._contextFactory: IPolicyForHTTPS = contextFactory
         self._connectTimeout: float = connectTimeout
         self._bindAddress: Optional[bytes] = bindAddress


Test patch: