Repository: scrapy/scrapy

Issue:
S3 Feed Export throws boto error: "Connection Reset By Peer"
Posted details on SO: http://stackoverflow.com/questions/27131693/scrapyd-s3-feed-export-connection-reset-by-peer

S3 Feed Export throws boto error: "Connection Reset By Peer"
Posted details on SO: http://stackoverflow.com/questions/27131693/scrapyd-s3-feed-export-connection-reset-by-peer

S3 backend can't handle uploads larger than 5GB 
### Description

When feeds larger than 5GB are sent using AWS S3 backend, I'm receiving the follow error:

```bash
2022-11-24 18:45:31 [scrapy.extensions.feedexport] ERROR: Error storing csv feed (55 items) in: s3://scrapy-test/large_export.csv
Traceback (most recent call last):
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/scrapy/extensions/feedexport.py", line 196, in _store_in_thread
    self.s3_client.put_object(
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/ogabrielsantos/crawler-scrapy/venv/lib/python3.10/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (EntityTooLarge) when calling the PutObject operation: Your proposed upload exceeds the maximum allowed size
```

### Steps to Reproduce

I've write a minimum exemple code to simulate this issue:

```python
from scrapy.spiders import Spider


class LargeExportSpider(Spider):
    name = "large_export"
    start_urls = ["http://news.ycombinator.com/"]
    custom_settings = {
        "FEEDS": {
            "s3://scrapy-test/large_export.csv": {
                "format": "csv",
            },
        },
    }

    def parse(self, response, **kwargs):
        text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque iaculis odio efficitur, ultricies"

        for _ in range(0, 55):  # creates a 5.3GB csv file
            yield {"name": "John Doe", "text": text * 1000000}

```

### Versions
`scrapy version --verbose`:
```bash
Scrapy       : 2.7.1
lxml         : 4.9.1.0
libxml2      : 2.9.13
cssselect    : 1.2.0
parsel       : 1.7.0
w3lib        : 2.0.1
Twisted      : 22.10.0
Python       : 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
pyOpenSSL    : 22.1.0 (OpenSSL 3.0.5 5 Jul 2022)
cryptography : 38.0.1
Platform     : macOS-13.0.1-arm64-arm-64bit
```

`requirements.txt`:
```
botocore==1.29.16
Scrapy==2.7.1
```

### Additional context

Doing some investigation, I've seen that `S3FeedStorage` uses `put_object` which, as [per documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html), has a limit of 5GB per uploaded object:

https://github.com/scrapy/scrapy/blob/6ded3cf4cd134b615239babe28bb28c3ff524b05/scrapy/extensions/feedexport.py#L196

Looks like `boto3` [already have an upload method](https://boto3.amazonaws.com/v1/documentation/api/1.16.53/guide/s3-uploading-files.html) which handles multipart files, but scrapy relies on `botocore`.


Discussion:
Scrapy uses boto for feed exports, so it is likely a boto issue (the one you've linked at SO, https://github.com/boto/boto/issues/2207). Do you know a workaround?

To send big feed files to S3 and avoid this bug, the fix is to use multipart upload. PR #1559 used to implement this for boto2.

It would be great if someone can resurrect this WIP #1559

Looking at the upstream API, it seems like implementing this change is a matter of:
- Documenting the need to install boto3 for S3 support, rather than botocore.
- Implement code that uploads using boto3’s method instead of botocore to upload files. It seems the interfaces are similar.
- If boto3 is not installed, but botocore is, fall back to the current implementation, and log a deprecation warning.
https://github.com/scrapy/scrapy/pull/4077 is an interesting approach, which starts uploading to S3 right away, rather than storing the whole output on disk first, and then uploading it.

But I would rather have [a simpler approach](https://github.com/scrapy/scrapy/issues/960#issuecomment-1326872050) than none. We can always implement the https://github.com/scrapy/scrapy/pull/4077 approach afterwards.
See also the [workaround](https://github.com/scrapy/scrapy/issues/5735#issuecomment-1327465852) by @ogabrielsantos.
I'd like to have a go at [the simpler approach](https://github.com/scrapy/scrapy/issues/960#issuecomment-1326872050), if that's alright.
Scrapy uses boto for feed exports, so it is likely a boto issue (the one you've linked at SO, https://github.com/boto/boto/issues/2207). Do you know a workaround?

To send big feed files to S3 and avoid this bug, the fix is to use multipart upload. PR #1559 used to implement this for boto2.

It would be great if someone can resurrect this WIP #1559

Looking at the upstream API, it seems like implementing this change is a matter of:
- Documenting the need to install boto3 for S3 support, rather than botocore.
- Implement code that uploads using boto3’s method instead of botocore to upload files. It seems the interfaces are similar.
- If boto3 is not installed, but botocore is, fall back to the current implementation, and log a deprecation warning.
https://github.com/scrapy/scrapy/pull/4077 is an interesting approach, which starts uploading to S3 right away, rather than storing the whole output on disk first, and then uploading it.

But I would rather have [a simpler approach](https://github.com/scrapy/scrapy/issues/960#issuecomment-1326872050) than none. We can always implement the https://github.com/scrapy/scrapy/pull/4077 approach afterwards.
See also the [workaround](https://github.com/scrapy/scrapy/issues/5735#issuecomment-1327465852) by @ogabrielsantos.
I'd like to have a go at [the simpler approach](https://github.com/scrapy/scrapy/issues/960#issuecomment-1326872050), if that's alright.
Sounds related to https://github.com/scrapy/scrapy/issues/960 (at the very least, the solution seems to be the same)
I'm workarounding it until an official fix, this way:

`requirements.txt`:
```
boto3==1.26.16
```

`crawlers/feed.py`:
```python
class LargeS3FeedStorage(S3FeedStorage):
    def __init__(
        self,
        uri,
        access_key=None,
        secret_key=None,
        acl=None,
        endpoint_url=None,
        *,
        feed_options=None,
        session_token=None,
    ):
        super().__init__(
            uri, access_key, secret_key, acl, endpoint_url, feed_options=feed_options, session_token=session_token
        )

        self.s3_client = boto3.client(
            "s3",
            aws_access_key_id=self.access_key,
            aws_secret_access_key=self.secret_key,
            aws_session_token=self.session_token,
            endpoint_url=self.endpoint_url,
        )

    def _store_in_thread(self, file):
        file.seek(0)
        self.s3_client.upload_file(Filename=file.file.name, Bucket=self.bucketname, Key=self.keyname)
        file.close()
```

`settings.py`:
```python
FEED_STORAGES = {
    "s3": "crawlers.feed.LargeS3FeedStorage",
}
```

OOPS. Accidentally closed by private repository merge. Reopening.

Resolving patch:
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -30,6 +30,13 @@
 
 logger = logging.getLogger(__name__)
 
+try:
+    import boto3  # noqa: F401
+
+    IS_BOTO3_AVAILABLE = True
+except ImportError:
+    IS_BOTO3_AVAILABLE = False
+
 
 def build_storage(builder, uri, *args, feed_options=None, preargs=(), **kwargs):
     argument_names = get_func_args(builder)
@@ -173,16 +180,38 @@ def __init__(
         self.keyname = u.path[1:]  # remove first "/"
         self.acl = acl
         self.endpoint_url = endpoint_url
-        import botocore.session
-
-        session = botocore.session.get_session()
-        self.s3_client = session.create_client(
-            "s3",
-            aws_access_key_id=self.access_key,
-            aws_secret_access_key=self.secret_key,
-            aws_session_token=self.session_token,
-            endpoint_url=self.endpoint_url,
-        )
+
+        if IS_BOTO3_AVAILABLE:
+            import boto3.session
+
+            session = boto3.session.Session()
+
+            self.s3_client = session.client(
+                "s3",
+                aws_access_key_id=self.access_key,
+                aws_secret_access_key=self.secret_key,
+                aws_session_token=self.session_token,
+                endpoint_url=self.endpoint_url,
+            )
+        else:
+            warnings.warn(
+                "`botocore` usage has been deprecated for S3 feed "
+                "export, please use `boto3` to avoid problems",
+                category=ScrapyDeprecationWarning,
+            )
+
+            import botocore.session
+
+            session = botocore.session.get_session()
+
+            self.s3_client = session.create_client(
+                "s3",
+                aws_access_key_id=self.access_key,
+                aws_secret_access_key=self.secret_key,
+                aws_session_token=self.session_token,
+                endpoint_url=self.endpoint_url,
+            )
+
         if feed_options and feed_options.get("overwrite", True) is False:
             logger.warning(
                 "S3 does not support appending to files. To "
@@ -205,10 +234,16 @@ def from_crawler(cls, crawler, uri, *, feed_options=None):
 
     def _store_in_thread(self, file):
         file.seek(0)
-        kwargs = {"ACL": self.acl} if self.acl else {}
-        self.s3_client.put_object(
-            Bucket=self.bucketname, Key=self.keyname, Body=file, **kwargs
-        )
+        if IS_BOTO3_AVAILABLE:
+            kwargs = {"ExtraArgs": {"ACL": self.acl}} if self.acl else {}
+            self.s3_client.upload_fileobj(
+                Bucket=self.bucketname, Key=self.keyname, Fileobj=file, **kwargs
+            )
+        else:
+            kwargs = {"ACL": self.acl} if self.acl else {}
+            self.s3_client.put_object(
+                Bucket=self.bucketname, Key=self.keyname, Body=file, **kwargs
+            )
         file.close()
 
 


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -35,6 +35,7 @@
 from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
 from scrapy.exporters import CsvItemExporter, JsonItemExporter
 from scrapy.extensions.feedexport import (
+    IS_BOTO3_AVAILABLE,
     BlockingFeedStorage,
     FeedExporter,
     FileFeedStorage,
@@ -235,8 +236,10 @@ def test_invalid_folder(self):
 
 
 class S3FeedStorageTest(unittest.TestCase):
-    def test_parse_credentials(self):
+    def setUp(self):
         skip_if_no_boto()
+
+    def test_parse_credentials(self):
         aws_credentials = {
             "AWS_ACCESS_KEY_ID": "settings_key",
             "AWS_SECRET_ACCESS_KEY": "settings_secret",
@@ -272,8 +275,6 @@ def test_parse_credentials(self):
 
     @defer.inlineCallbacks
     def test_store(self):
-        skip_if_no_boto()
-
         settings = {
             "AWS_ACCESS_KEY_ID": "access_key",
             "AWS_SECRET_ACCESS_KEY": "secret_key",
@@ -285,30 +286,39 @@ def test_store(self):
         verifyObject(IFeedStorage, storage)
 
         file = mock.MagicMock()
-        from botocore.stub import Stubber
-
-        with Stubber(storage.s3_client) as stub:
-            stub.add_response(
-                "put_object",
-                expected_params={
-                    "Body": file,
-                    "Bucket": bucket,
-                    "Key": key,
-                },
-                service_response={},
-            )
 
+        if IS_BOTO3_AVAILABLE:
+            storage.s3_client = mock.MagicMock()
             yield storage.store(file)
-
-            stub.assert_no_pending_responses()
             self.assertEqual(
-                file.method_calls,
-                [
-                    mock.call.seek(0),
-                    # The call to read does not happen with Stubber
-                    mock.call.close(),
-                ],
+                storage.s3_client.upload_fileobj.call_args,
+                mock.call(Bucket=bucket, Key=key, Fileobj=file),
             )
+        else:
+            from botocore.stub import Stubber
+
+            with Stubber(storage.s3_client) as stub:
+                stub.add_response(
+                    "put_object",
+                    expected_params={
+                        "Body": file,
+                        "Bucket": bucket,
+                        "Key": key,
+                    },
+                    service_response={},
+                )
+
+                yield storage.store(file)
+
+                stub.assert_no_pending_responses()
+                self.assertEqual(
+                    file.method_calls,
+                    [
+                        mock.call.seek(0),
+                        # The call to read does not happen with Stubber
+                        mock.call.close(),
+                    ],
+                )
 
     def test_init_without_acl(self):
         storage = S3FeedStorage("s3://mybucket/export.csv", "access_key", "secret_key")
@@ -391,8 +401,7 @@ def test_from_crawler_with_endpoint_url(self):
         self.assertEqual(storage.endpoint_url, "https://example.com")
 
     @defer.inlineCallbacks
-    def test_store_botocore_without_acl(self):
-        skip_if_no_boto()
+    def test_store_without_acl(self):
         storage = S3FeedStorage(
             "s3://mybucket/export.csv",
             "access_key",
@@ -404,11 +413,18 @@ def test_store_botocore_without_acl(self):
 
         storage.s3_client = mock.MagicMock()
         yield storage.store(BytesIO(b"test file"))
-        self.assertNotIn("ACL", storage.s3_client.put_object.call_args[1])
+        if IS_BOTO3_AVAILABLE:
+            acl = (
+                storage.s3_client.upload_fileobj.call_args[1]
+                .get("ExtraArgs", {})
+                .get("ACL")
+            )
+        else:
+            acl = storage.s3_client.put_object.call_args[1].get("ACL")
+        self.assertIsNone(acl)
 
     @defer.inlineCallbacks
-    def test_store_botocore_with_acl(self):
-        skip_if_no_boto()
+    def test_store_with_acl(self):
         storage = S3FeedStorage(
             "s3://mybucket/export.csv", "access_key", "secret_key", "custom-acl"
         )
@@ -418,9 +434,11 @@ def test_store_botocore_with_acl(self):
 
         storage.s3_client = mock.MagicMock()
         yield storage.store(BytesIO(b"test file"))
-        self.assertEqual(
-            storage.s3_client.put_object.call_args[1].get("ACL"), "custom-acl"
-        )
+        if IS_BOTO3_AVAILABLE:
+            acl = storage.s3_client.upload_fileobj.call_args[1]["ExtraArgs"]["ACL"]
+        else:
+            acl = storage.s3_client.put_object.call_args[1]["ACL"]
+        self.assertEqual(acl, "custom-acl")
 
     def test_overwrite_default(self):
         with LogCapture() as log:
@@ -888,15 +906,10 @@ def test_stats_file_failed(self):
     @defer.inlineCallbacks
     def test_stats_multiple_file(self):
         settings = {
-            "AWS_ACCESS_KEY_ID": "access_key",
-            "AWS_SECRET_ACCESS_KEY": "secret_key",
             "FEEDS": {
                 printf_escape(path_to_url(str(self._random_temp_filename()))): {
                     "format": "json",
                 },
-                "s3://bucket/key/foo.csv": {
-                    "format": "csv",
-                },
                 "stdout:": {
                     "format": "xml",
                 },
@@ -908,18 +921,12 @@ def test_stats_multiple_file(self):
         self.assertIn(
             "feedexport/success_count/FileFeedStorage", crawler.stats.get_stats()
         )
-        self.assertIn(
-            "feedexport/success_count/S3FeedStorage", crawler.stats.get_stats()
-        )
         self.assertIn(
             "feedexport/success_count/StdoutFeedStorage", crawler.stats.get_stats()
         )
         self.assertEqual(
             crawler.stats.get_value("feedexport/success_count/FileFeedStorage"), 1
         )
-        self.assertEqual(
-            crawler.stats.get_value("feedexport/success_count/S3FeedStorage"), 1
-        )
         self.assertEqual(
             crawler.stats.get_value("feedexport/success_count/StdoutFeedStorage"), 1
         )
@@ -2535,7 +2542,6 @@ def test_stats_batch_file_success(self):
     @defer.inlineCallbacks
     def test_s3_export(self):
         skip_if_no_boto()
-
         bucket = "mybucket"
         items = [
             self.MyItem({"foo": "bar1", "egg": "spam1"}),
@@ -2707,6 +2713,9 @@ class S3FeedStoragePreFeedOptionsTest(unittest.TestCase):
 
     maxDiff = None
 
+    def setUp(self):
+        skip_if_no_boto()
+
     def test_init(self):
         settings_dict = {
             "FEED_URI": "file:///tmp/foobar",
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -225,12 +225,16 @@ def file_path(self, request, response=None, info=None, item=None):
 
 
 class FilesPipelineTestCaseFieldsMixin:
+    def setUp(self):
+        self.tempdir = mkdtemp()
+
+    def tearDown(self):
+        rmtree(self.tempdir)
+
     def test_item_fields_default(self):
         url = "http://www.example.com/files/1.txt"
         item = self.item_class(name="item1", file_urls=[url])
-        pipeline = FilesPipeline.from_settings(
-            Settings({"FILES_STORE": "s3://example/files/"})
-        )
+        pipeline = FilesPipeline.from_settings(Settings({"FILES_STORE": self.tempdir}))
         requests = list(pipeline.get_media_requests(item, None))
         self.assertEqual(requests[0].url, url)
         results = [(True, {"url": url})]
@@ -245,7 +249,7 @@ def test_item_fields_override_settings(self):
         pipeline = FilesPipeline.from_settings(
             Settings(
                 {
-                    "FILES_STORE": "s3://example/files/",
+                    "FILES_STORE": self.tempdir,
                     "FILES_URLS_FIELD": "custom_file_urls",
                     "FILES_RESULT_FIELD": "custom_files",
                 }