Repository: scrapy/scrapy

Issue:
S3 FeedStorage does not support custom S3 like storages
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description
Storage options for S3 feed storage is not exporting to custom S3 like storage as supported by image pipeline. The endpoint `botocore` trying to hit is AWS's servers and it is not respecting the endpoint set using `AWS_ENDPOINT_URL` and `AWS_REGION_NAME`

### Steps to Reproduce

1. set all the necessary parameters in `settings.py` file such as  `AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL, AWS_REGION_NAME, FEED_STORAGE_S3_ACL`.

```
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXX'
AWS_USE_SSL = True  # or True (None by default)
AWS_VERIFY = True  # or True (None by default)
AWS_ENDPOINT_URL = 'https://example.abc.com'  <-- custom s3 like cloud
AWS_REGION_NAME = 'us-east' 
FEED_STORAGE_S3_ACL = 'private'
```
2.  Then setup feeds using
```
FEEDS = {
    's3://bucket/path/to/file-%(name)s-%(time)s.json': {
        'format': 'json',
        'encoding': 'utf8',
        'store_empty': False,
        'fields': None,
        'indent': 4,
        'item_export_kwargs': {
            'export_empty_fields': True,
        }
    }
}
```
3. ``` scrapy crawl spider```

**Expected behavior:** 
This should be able to store files to other S3 like cloud, as it does with `ImagePipeline`

**Actual behavior:**
```
DEBUG: Starting new HTTPS connection (1): bucket.s3.amazonaws.com:443 <---- I think this is the problem

Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>XXXXXXXXXXXXXXXXXX</AWSAccessKeyId>

[scrapy.extensions.feedexport] ERROR: Error storing json feed (36 items) in: s3://bucket/path/to/file-spider-2020-11-04T11-19-05.json

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.
```

**Reproduces how often:** 
Always

### Versions
```
Scrapy       : 2.4.0
lxml         : 4.6.1.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) - [Clang 6.0 (clang-600.0.57)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1h  22 Sep 2020)
cryptography : 3.2.1
Platform     : Darwin-19.6.0-x86_64-i386-64bit
```



Discussion:
Indeed this seems to be implemented only for [media pipelines](https://docs.scrapy.org/en/latest/topics/media-pipeline.html) at the moment.
Thank you :) Would it work if I created a custom pipeline? Sorry, I am new to scrapy
You _could_ write a pipeline that manually sends items to your S3.

A cleaner (and possibly easier) approach would be to:

1. Create a subclass of [scrapy.extensions.feedexport.S3FeedStorage](https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/feedexport.py#L119) that uses `crawler.settings['AWS_ENDPOINT_URL']` where it needs to be used.
2. Point the [FEED_STORAGES](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEED_STORAGES) setting to your subclass for S3: `FEED_STORAGES = {'s3': <your subclass>}`.

The best part of this approach is not that it’s cleaner or probably easier, but that once you get it working it would be trivial for you to send a pull request here to implement this in Scrapy itself :slightly_smiling_face: 
That is great!! Thank you very much :) I will implement it and send a pull request. 

Resolving patch:
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -118,7 +118,7 @@ def store(self, file):
 
 class S3FeedStorage(BlockingFeedStorage):
 
-    def __init__(self, uri, access_key=None, secret_key=None, acl=None, *,
+    def __init__(self, uri, access_key=None, secret_key=None, acl=None, endpoint_url=None, *,
                  feed_options=None):
         if not is_botocore_available():
             raise NotConfigured('missing botocore library')
@@ -128,11 +128,13 @@ def __init__(self, uri, access_key=None, secret_key=None, acl=None, *,
         self.secret_key = u.password or secret_key
         self.keyname = u.path[1:]  # remove first "/"
         self.acl = acl
+        self.endpoint_url = endpoint_url
         import botocore.session
         session = botocore.session.get_session()
         self.s3_client = session.create_client(
             's3', aws_access_key_id=self.access_key,
-            aws_secret_access_key=self.secret_key)
+            aws_secret_access_key=self.secret_key,
+            endpoint_url=self.endpoint_url)
         if feed_options and feed_options.get('overwrite', True) is False:
             logger.warning('S3 does not support appending to files. To '
                            'suppress this warning, remove the overwrite '
@@ -146,6 +148,7 @@ def from_crawler(cls, crawler, uri, *, feed_options=None):
             access_key=crawler.settings['AWS_ACCESS_KEY_ID'],
             secret_key=crawler.settings['AWS_SECRET_ACCESS_KEY'],
             acl=crawler.settings['FEED_STORAGE_S3_ACL'] or None,
+            endpoint_url=crawler.settings['AWS_ENDPOINT_URL'] or None,
             feed_options=feed_options,
         )
 


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -326,6 +326,17 @@ def test_init_with_acl(self):
         self.assertEqual(storage.secret_key, 'secret_key')
         self.assertEqual(storage.acl, 'custom-acl')
 
+    def test_init_with_endpoint_url(self):
+        storage = S3FeedStorage(
+            's3://mybucket/export.csv',
+            'access_key',
+            'secret_key',
+            endpoint_url='https://example.com'
+        )
+        self.assertEqual(storage.access_key, 'access_key')
+        self.assertEqual(storage.secret_key, 'secret_key')
+        self.assertEqual(storage.endpoint_url, 'https://example.com')
+
     def test_from_crawler_without_acl(self):
         settings = {
             'AWS_ACCESS_KEY_ID': 'access_key',
@@ -340,6 +351,20 @@ def test_from_crawler_without_acl(self):
         self.assertEqual(storage.secret_key, 'secret_key')
         self.assertEqual(storage.acl, None)
 
+    def test_without_endpoint_url(self):
+        settings = {
+            'AWS_ACCESS_KEY_ID': 'access_key',
+            'AWS_SECRET_ACCESS_KEY': 'secret_key',
+        }
+        crawler = get_crawler(settings_dict=settings)
+        storage = S3FeedStorage.from_crawler(
+            crawler,
+            's3://mybucket/export.csv',
+        )
+        self.assertEqual(storage.access_key, 'access_key')
+        self.assertEqual(storage.secret_key, 'secret_key')
+        self.assertEqual(storage.endpoint_url, None)
+
     def test_from_crawler_with_acl(self):
         settings = {
             'AWS_ACCESS_KEY_ID': 'access_key',
@@ -355,6 +380,21 @@ def test_from_crawler_with_acl(self):
         self.assertEqual(storage.secret_key, 'secret_key')
         self.assertEqual(storage.acl, 'custom-acl')
 
+    def test_from_crawler_with_endpoint_url(self):
+        settings = {
+            'AWS_ACCESS_KEY_ID': 'access_key',
+            'AWS_SECRET_ACCESS_KEY': 'secret_key',
+            'AWS_ENDPOINT_URL': 'https://example.com',
+        }
+        crawler = get_crawler(settings_dict=settings)
+        storage = S3FeedStorage.from_crawler(
+            crawler,
+            's3://mybucket/export.csv'
+        )
+        self.assertEqual(storage.access_key, 'access_key')
+        self.assertEqual(storage.secret_key, 'secret_key')
+        self.assertEqual(storage.endpoint_url, 'https://example.com')
+
     @defer.inlineCallbacks
     def test_store_botocore_without_acl(self):
         skip_if_no_boto()
@@ -1809,8 +1849,8 @@ def test_init(self):
 
 class S3FeedStorageWithoutFeedOptions(S3FeedStorage):
 
-    def __init__(self, uri, access_key, secret_key, acl):
-        super().__init__(uri, access_key, secret_key, acl)
+    def __init__(self, uri, access_key, secret_key, acl, endpoint_url):
+        super().__init__(uri, access_key, secret_key, acl, endpoint_url)
 
 
 class S3FeedStorageWithoutFeedOptionsWithFromCrawler(S3FeedStorage):