Repository: scrapy/scrapy

Issue:
FEED_URI_PARAMS: custom params throws KeyError
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

With Scrapy version 2.4.1, any custom feed uri params (FEED_URI_PARAMS) throws KeyError. 

### Steps to Reproduce

1. on myproject/settings.py
FEED_URI_PARAMS = 'myproject.utils.uri_params'

2. on myproject/utils.py
def uri_params(params, spider):
    return {**params, 'spider_name': spider.name}

3. scrapy crawl <spider_name> -o "%(spider_name)s.jl"

**Expected behavior:** Should export the items to a jsonline file having the spider name.

**Actual behavior:**
ERROR: Error caught on signal handler: <bound method FeedExporter.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fc68cb20bb0>>
Traceback (most recent call last):
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/utils/defer.py", line 157, in maybeDeferred_coro
    result = f(*args, **kw)
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/extensions/feedexport.py", line 294, in open_spider
    uri=uri % uri_params,
KeyError: 'spider_name'

**Reproduces how often:** Everytime

### Versions
Scrapy       : 2.4.1
lxml         : 4.6.1.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.8.5 (default, Jul 28 2020, 12:59:40) - [GCC 9.3.0]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1h  22 Sep 2020)
cryptography : 3.2.1
Platform     : Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.29

### Additional context

on line 480 in /scrapy/extensions/feedexport.py,
**returns params** should be changed to **return uripar_function(params, spider)**
FEED_URI_PARAMS: custom params throws KeyError
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

With Scrapy version 2.4.1, any custom feed uri params (FEED_URI_PARAMS) throws KeyError. 

### Steps to Reproduce

1. on myproject/settings.py
FEED_URI_PARAMS = 'myproject.utils.uri_params'

2. on myproject/utils.py
def uri_params(params, spider):
    return {**params, 'spider_name': spider.name}

3. scrapy crawl <spider_name> -o "%(spider_name)s.jl"

**Expected behavior:** Should export the items to a jsonline file having the spider name.

**Actual behavior:**
ERROR: Error caught on signal handler: <bound method FeedExporter.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fc68cb20bb0>>
Traceback (most recent call last):
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/utils/defer.py", line 157, in maybeDeferred_coro
    result = f(*args, **kw)
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/extensions/feedexport.py", line 294, in open_spider
    uri=uri % uri_params,
KeyError: 'spider_name'

**Reproduces how often:** Everytime

### Versions
Scrapy       : 2.4.1
lxml         : 4.6.1.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.8.5 (default, Jul 28 2020, 12:59:40) - [GCC 9.3.0]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1h  22 Sep 2020)
cryptography : 3.2.1
Platform     : Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.29

### Additional context

on line 480 in /scrapy/extensions/feedexport.py,
**returns params** should be changed to **return uripar_function(params, spider)**


Discussion:
> on line 480 in /scrapy/extensions/feedexport.py,
> returns params should be changed to return uripar_function(params, spider)

Indeed:

https://github.com/scrapy/scrapy/blob/85604e1078b927c2a875040e29c58b4594c8d386/scrapy/extensions/feedexport.py#L487-L488

For your specific use case, though, I _think_ you can just use `%(name)s`, based on these lines above: https://github.com/scrapy/scrapy/blob/85604e1078b927c2a875040e29c58b4594c8d386/scrapy/extensions/feedexport.py#L480-L481
Yes, %(name)s was my workaround solution. Thanks for your comment, Gallaecio.
Hi @jas0nkim @Gallaecio,

I'm new here.  Can I take this issue? :) 

@inspurwusixuan Please go ahead! :tada: 
> on line 480 in /scrapy/extensions/feedexport.py,
> returns params should be changed to return uripar_function(params, spider)

Indeed:

https://github.com/scrapy/scrapy/blob/85604e1078b927c2a875040e29c58b4594c8d386/scrapy/extensions/feedexport.py#L487-L488

For your specific use case, though, I _think_ you can just use `%(name)s`, based on these lines above: https://github.com/scrapy/scrapy/blob/85604e1078b927c2a875040e29c58b4594c8d386/scrapy/extensions/feedexport.py#L480-L481
Yes, %(name)s was my workaround solution. Thanks for your comment, Gallaecio.
Hi @jas0nkim @Gallaecio,

I'm new here.  Can I take this issue? :) 

@inspurwusixuan Please go ahead! :tada: 

Resolving patch:
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -11,14 +11,14 @@
 import warnings
 from datetime import datetime
 from tempfile import NamedTemporaryFile
-from typing import Any, Optional, Tuple
+from typing import Any, Callable, Optional, Tuple, Union
 from urllib.parse import unquote, urlparse
 
 from twisted.internet import defer, threads
 from w3lib.url import file_uri_to_path
 from zope.interface import implementer, Interface
 
-from scrapy import signals
+from scrapy import signals, Spider
 from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
 from scrapy.extensions.postprocessing import PostProcessingManager
 from scrapy.utils.boto import is_botocore_available
@@ -524,7 +524,12 @@ def build_instance(builder, *preargs):
             raise TypeError(f"{feedcls.__qualname__}.{method_name} returned None")
         return instance
 
-    def _get_uri_params(self, spider, uri_params, slot=None):
+    def _get_uri_params(
+        self,
+        spider: Spider,
+        uri_params_function: Optional[Union[str, Callable[[dict, Spider], dict]]],
+        slot: Optional[_FeedSlot] = None,
+    ) -> dict:
         params = {}
         for k in dir(spider):
             params[k] = getattr(spider, k)
@@ -532,9 +537,18 @@ def _get_uri_params(self, spider, uri_params, slot=None):
         params['time'] = utc_now.replace(microsecond=0).isoformat().replace(':', '-')
         params['batch_time'] = utc_now.isoformat().replace(':', '-')
         params['batch_id'] = slot.batch_id + 1 if slot is not None else 1
-        uripar_function = load_object(uri_params) if uri_params else lambda x, y: None
-        uripar_function(params, spider)
-        return params
+        original_params = params.copy()
+        uripar_function = load_object(uri_params_function) if uri_params_function else lambda params, _: params
+        new_params = uripar_function(params, spider)
+        if new_params is None or original_params != params:
+            warnings.warn(
+                'Modifying the params dictionary in-place in the function defined in '
+                'the FEED_URI_PARAMS setting or in the uri_params key of the FEEDS '
+                'setting is deprecated. The function must return a new dictionary '
+                'instead.',
+                category=ScrapyDeprecationWarning
+            )
+        return new_params if new_params is not None else params
 
     def _load_filter(self, feed_options):
         # load the item filter if declared else load the default filter class


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -2608,3 +2608,166 @@ def test_from_crawler(self):
                     ),
                 )
             )
+
+
+class URIParamsTest:
+
+    spider_name = "uri_params_spider"
+
+    def build_settings(self, uri='file:///tmp/foobar', uri_params=None):
+        raise NotImplementedError
+
+    def test_default(self):
+        settings = self.build_settings(
+            uri='file:///tmp/%(name)s',
+        )
+        crawler = get_crawler(settings_dict=settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider(self.spider_name)
+        spider.crawler = crawler
+        with warnings.catch_warnings(record=True) as w:
+            feed_exporter.open_spider(spider)
+            messages = tuple(
+                str(item.message) for item in w
+                if item.category is ScrapyDeprecationWarning
+            )
+            self.assertEqual(messages, tuple())
+
+        self.assertEqual(
+            feed_exporter.slots[0].uri,
+            f'file:///tmp/{self.spider_name}'
+        )
+
+    def test_none(self):
+        def uri_params(params, spider):
+            pass
+
+        settings = self.build_settings(
+            uri='file:///tmp/%(name)s',
+            uri_params=uri_params,
+        )
+        crawler = get_crawler(settings_dict=settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider(self.spider_name)
+        spider.crawler = crawler
+        with warnings.catch_warnings(record=True) as w:
+            feed_exporter.open_spider(spider)
+            messages = tuple(
+                str(item.message) for item in w
+                if item.category is ScrapyDeprecationWarning
+            )
+            self.assertEqual(
+                messages,
+                (
+                    (
+                        'Modifying the params dictionary in-place in the '
+                        'function defined in the FEED_URI_PARAMS setting or '
+                        'in the uri_params key of the FEEDS setting is '
+                        'deprecated. The function must return a new '
+                        'dictionary instead.'
+                    ),
+                )
+            )
+
+        self.assertEqual(
+            feed_exporter.slots[0].uri,
+            f'file:///tmp/{self.spider_name}'
+        )
+
+    def test_empty_dict(self):
+        def uri_params(params, spider):
+            return {}
+
+        settings = self.build_settings(
+            uri='file:///tmp/%(name)s',
+            uri_params=uri_params,
+        )
+        crawler = get_crawler(settings_dict=settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider(self.spider_name)
+        spider.crawler = crawler
+        with warnings.catch_warnings(record=True) as w:
+            with self.assertRaises(KeyError):
+                feed_exporter.open_spider(spider)
+            messages = tuple(
+                str(item.message) for item in w
+                if item.category is ScrapyDeprecationWarning
+            )
+            self.assertEqual(messages, tuple())
+
+    def test_params_as_is(self):
+        def uri_params(params, spider):
+            return params
+
+        settings = self.build_settings(
+            uri='file:///tmp/%(name)s',
+            uri_params=uri_params,
+        )
+        crawler = get_crawler(settings_dict=settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider(self.spider_name)
+        spider.crawler = crawler
+        with warnings.catch_warnings(record=True) as w:
+            feed_exporter.open_spider(spider)
+            messages = tuple(
+                str(item.message) for item in w
+                if item.category is ScrapyDeprecationWarning
+            )
+            self.assertEqual(messages, tuple())
+
+        self.assertEqual(
+            feed_exporter.slots[0].uri,
+            f'file:///tmp/{self.spider_name}'
+        )
+
+    def test_custom_param(self):
+        def uri_params(params, spider):
+            return {**params, 'foo': self.spider_name}
+
+        settings = self.build_settings(
+            uri='file:///tmp/%(foo)s',
+            uri_params=uri_params,
+        )
+        crawler = get_crawler(settings_dict=settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider(self.spider_name)
+        spider.crawler = crawler
+        with warnings.catch_warnings(record=True) as w:
+            feed_exporter.open_spider(spider)
+            messages = tuple(
+                str(item.message) for item in w
+                if item.category is ScrapyDeprecationWarning
+            )
+            self.assertEqual(messages, tuple())
+
+        self.assertEqual(
+            feed_exporter.slots[0].uri,
+            f'file:///tmp/{self.spider_name}'
+        )
+
+
+class URIParamsSettingTest(URIParamsTest, unittest.TestCase):
+
+    def build_settings(self, uri='file:///tmp/foobar', uri_params=None):
+        extra_settings = {}
+        if uri_params:
+            extra_settings['FEED_URI_PARAMS'] = uri_params
+        return {
+            'FEED_URI': uri,
+            **extra_settings,
+        }
+
+
+class URIParamsFeedOptionTest(URIParamsTest, unittest.TestCase):
+
+    def build_settings(self, uri='file:///tmp/foobar', uri_params=None):
+        options = {
+            'format': 'jl',
+        }
+        if uri_params:
+            options['uri_params'] = uri_params
+        return {
+            'FEEDS': {
+                uri: options,
+            },
+        }