Repository: scrapy/scrapy

Issue:
Scrapy > 2.3 prevents passing callback in cb_kwargs
To reproduce:

```bash
scrapy startproject tutorial
cd tutorial
scrapy genspider example example.com
```

In example.py

```python
import scrapy


class ExampleSpider(scrapy.Spider):
    name = 'example'

    def start_requests(self):
        yield scrapy.Request(
            'http://example.com',
            callback=self.parse_1,
            cb_kwargs={'callback': self.parse_2}
        )

    def parse_1(self, response, **kwargs):
        yield {'url': response.request.url}
        yield scrapy.Request('https://httpbin.org/get', **kwargs)

    def parse_2(self, response):
        yield {'url': response.request.url}
```

This works in Scrapy 2.3. In Scrapy > 2.3, it errors:

```
2021-08-31 10:24:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.com> (referer: None)
2021-08-31 10:24:38 [scrapy.core.scraper] ERROR: Spider error processing <GET http://example.com> (referer: None)
Traceback (most recent call last):
  File "lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: <200 http://example.com>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lib/python3.6/site-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "lib/python3.6/site-packages/scrapy/core/spidermw.py", line 52, in _process_spider_input
    return scrape_func(response, request, spider)
  File "lib/python3.6/site-packages/scrapy/core/scraper.py", line 151, in call_spider
    dfd.addCallback(callback, **result.request.cb_kwargs)
TypeError: addCallback() got multiple values for argument 'callback'
```

I don't know if this scrapy commit is relevant to the change in behavior: https://github.com/scrapy/scrapy/commit/2aa4f3cbf96ad55aa4a1fa064758338daf16b110

This is a minimal example. Of course, in my application, I'm jumping through these hoops (controlling the callback of a subsequent request) to be able to reuse code in a complex project.


Discussion:
I've reproduced this issue
It happened because this:
https://github.com/scrapy/scrapy/blob/1278e76d9093b1c5c9ec810768d1066772a8a134/scrapy/core/scraper.py#L154-L156
replaced by:
https://github.com/scrapy/scrapy/blob/e63188cbf753d560e43d8489c821bd6eb9fe54e9/scrapy/core/scraper.py#L151
as result of mentioned commit https://github.com/scrapy/scrapy/commit/2aa4f3cbf96ad55aa4a1fa064758338daf16b110

This `dfd.addCallback` -> refer to `twisted.internet.defer.addCallback`
```
    def addCallback(self, callback, *args, **kw):
        """
        Convenience method for adding just a callback.

        See L{addCallbacks}.
        """
        return self.addCallbacks(callback, callbackArgs=args,
                                 callbackKeywords=kw)
```
here `**kw` -  `**result.request.cb_kwargs` , in this case `callback` from transferred `**kw` will be "merged" with original `callback` causing `addCallback() got multiple values for argument 'callback'`
Cc: @elacuesta 

@jpmckinney  On https://github.com/scrapy/scrapy/issues/1138 (`cb_kwargs` feature - result of discuss on this issue)
I didn't found Your case (or anything else similar to it) as potential use case of `cb_kwargs`
There are other ways to pass the callback (e.g. via request meta), so this doesn't necessarily need to be fixed. However, it might be worth mentioning that a `'callback'` key in `cb_kwargs` is disallowed.
If it’s trivial to allow a `callback` parameter, it may be best to do that. Otherwise, we should raise an error that is more helpful, in addition to mention it in the documentation.

Resolving patch:
diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py
--- a/scrapy/core/scraper.py
+++ b/scrapy/core/scraper.py
@@ -156,7 +156,7 @@ def call_spider(self, result: Union[Response, Failure], request: Request, spider
             callback = result.request.callback or spider._parse
             warn_on_generator_with_return_value(spider, callback)
             dfd = defer_succeed(result)
-            dfd.addCallback(callback, **result.request.cb_kwargs)
+            dfd.addCallbacks(callback=callback, callbackKeywords=result.request.cb_kwargs)
         else:  # result is a Failure
             result.request = request
             warn_on_generator_with_return_value(spider, request.errback)


Test patch:
diff --git a/tests/test_request_cb_kwargs.py b/tests/test_request_cb_kwargs.py
--- a/tests/test_request_cb_kwargs.py
+++ b/tests/test_request_cb_kwargs.py
@@ -60,7 +60,7 @@ class KeywordArgumentsSpider(MockServerSpider):
     checks = []
 
     def start_requests(self):
-        data = {'key': 'value', 'number': 123}
+        data = {'key': 'value', 'number': 123, 'callback': 'some_callback'}
         yield Request(self.mockserver.url('/first'), self.parse_first, cb_kwargs=data)
         yield Request(self.mockserver.url('/general_with'), self.parse_general, cb_kwargs=data)
         yield Request(self.mockserver.url('/general_without'), self.parse_general)
@@ -88,7 +88,8 @@ def parse_general(self, response, **kwargs):
         if response.url.endswith('/general_with'):
             self.checks.append(kwargs['key'] == 'value')
             self.checks.append(kwargs['number'] == 123)
-            self.crawler.stats.inc_value('boolean_checks', 2)
+            self.checks.append(kwargs['callback'] == 'some_callback')
+            self.crawler.stats.inc_value('boolean_checks', 3)
         elif response.url.endswith('/general_without'):
             self.checks.append(kwargs == {})
             self.crawler.stats.inc_value('boolean_checks')
@@ -110,7 +111,7 @@ def parse_takes_less(self, response, key):
         TypeError: parse_takes_less() got an unexpected keyword argument 'number'
         """
 
-    def parse_takes_more(self, response, key, number, other):
+    def parse_takes_more(self, response, key, number, callback, other):
         """
         Should raise
         TypeError: parse_takes_more() missing 1 required positional argument: 'other'
@@ -161,11 +162,13 @@ def test_callback_kwargs(self):
         self.assertTrue(
             str(exceptions['takes_less'].exc_info[1]).endswith(
                 "parse_takes_less() got an unexpected keyword argument 'number'"
-            )
+            ),
+            msg="Exception message: " + str(exceptions['takes_less'].exc_info[1]),
         )
         self.assertEqual(exceptions['takes_more'].exc_info[0], TypeError)
         self.assertTrue(
             str(exceptions['takes_more'].exc_info[1]).endswith(
                 "parse_takes_more() missing 1 required positional argument: 'other'"
-            )
+            ),
+            msg="Exception message: " + str(exceptions['takes_more'].exc_info[1]),
         )