Repository: PrefectHQ/prefect

Issue:
Transient S3Upload failure: KeyError('endpoint_resolver')
## Description
Occasionally, one single mapped instance of an `S3Upload` task will fail with an unexpected `KeyError('endpoint_resolver')` which causes the flow to fail. Every time I have encountered this error, the flow was subsequently successful when it was restarted, without making any changes to the code or to any aspect of the Prefect setup.

Full stacktrace:
```
Unexpected error: KeyError('endpoint_resolver')
Traceback (most recent call last):
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\task_runner.py", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\tasks\aws\s3.py", line 147, in run
    s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\aws.py", line 55, in get_boto_client
    return boto3.client(
  File "c:\my-project\.venv\lib\site-packages\boto3\__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\boto3\session.py", line 258, in client
    return self._session.create_client(
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 824, in create_client
    endpoint_resolver = self._get_internal_component('endpoint_resolver')
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 697, in _get_internal_component
    return self._internal_components.get_component(name)
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 923, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'
```

My personal experience with this error is with a flow which creates around 25 mapped instances of the `S3Upload` task, each of which uploads one file to the same S3 bucket using the same credentials. This flow runs every day and seems to encounter an error every four or five runs, meaning the `S3Upload` task fails around 1% of the time. I think I can exclude transient connection issues as the root cause, because the `S3Upload` tasks all run within a minute or so of each other and only one task ever fails. Curiously, _every_ time I have seen this failure, it has occurred on the mapped task with index `1`.

## Expected Behavior
That all of the `S3Upload` tasks execute reliably without encountering this error. Or, failing that, I would expect that all of the `S3Upload` tasks would fail for this reason if there really is some logical or structural problem with the flow.


## Reproduction
Unfortunately I am unable to reliably replicate this error. Included below are details of my flow and setup for reference. My best suggestion to try to replicate this reliably would be to create a flow which maps over ~25 items and causes that many instances of the `S3Upload` task to execute, then running that flow repeatedly.

Flow logic:
```
from prefect.tasks.aws.s3 import S3Upload

upload_to_s3 = S3Upload(bucket=Config.S3_BUCKET)

with Flow() as flow:
    # ETL logic here which maps over 25 CSV files

    s3_objects = upload_to_s3.map(
        data=csv_bytes,
        key=s3_keys,
        compression=unmapped("gzip"))
```

Following lines are in `config.toml`:

```
[cloud]
use_local_secrets = true

[AWS_CREDENTIALS]
ACCESS_KEY = "foo"
SECRET_ACCESS_KEY = "bar"
```

## Environment
Output from running `prefect diagnostics`:

```
{
  "config_overrides": {
    "AWS_CREDENTIALS": {
      "ACCESS_KEY": true,
      "SECRET_ACCESS_KEY": true
    },
    "backend": true,
    "context": {
      "secrets": false
    },
    "debug": true,
    "server": {
      "database": {
        "host": true
      },
      "graphql": {
        "host": true
      },
      "hasura": {
        "host": true
      },
      "host": true,
      "ui": {
        "graphql_url": true,
        "host": true
      }
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "Windows-10-10.0.14393-SP0",
    "prefect_backend": "server",
    "prefect_version": "0.13.15",
    "python_version": "3.8.1"
  }
}
```
(in the next couple of weeks our team plans to upgrade these flows to use prefect version >=0.14, if the update changes anything I'll update this issue)


Discussion:
This appears to be a boto3 issue -- client creation is not thread safe. See https://github.com/boto/boto3/issues/801

We could consider using `resource` instead

Resolving patch:
diff --git a/src/prefect/tasks/aws/s3.py b/src/prefect/tasks/aws/s3.py
--- a/src/prefect/tasks/aws/s3.py
+++ b/src/prefect/tasks/aws/s3.py
@@ -64,7 +64,9 @@ def run(
         if bucket is None:
             raise ValueError("A bucket name must be provided.")
 
-        s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
+        s3_client = get_boto_client(
+            "s3", credentials=credentials, use_session=True, **self.boto_kwargs
+        )
 
         stream = io.BytesIO()
 
@@ -144,7 +146,9 @@ def run(
         if bucket is None:
             raise ValueError("A bucket name must be provided.")
 
-        s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
+        s3_client = get_boto_client(
+            "s3", credentials=credentials, use_session=True, **self.boto_kwargs
+        )
 
         # compress data if compression is specified
         if compression:


Test patch:
diff --git a/tests/tasks/aws/test_s3.py b/tests/tasks/aws/test_s3.py
--- a/tests/tasks/aws/test_s3.py
+++ b/tests/tasks/aws/test_s3.py
@@ -8,6 +8,15 @@
 from prefect.utilities.configuration import set_temporary_config
 
 
+@pytest.fixture
+def mocked_boto_client(monkeypatch):
+    boto3 = MagicMock()
+    client = boto3.session.Session().client()
+    boto3.client = MagicMock(return_value=client)
+    monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
+    return client
+
+
 class TestS3Download:
     def test_initialization(self):
         task = S3Download()
@@ -22,7 +31,7 @@ def test_raises_if_bucket_not_eventually_provided(self):
         with pytest.raises(ValueError, match="bucket"):
             task.run(key="")
 
-    def test_gzip_compression(self, monkeypatch):
+    def test_gzip_compression(self, mocked_boto_client):
         task = S3Download("bucket")
         byte_string = b"col1,col2,col3\nfake,data,1\nfalse,data,2\n"
         gzip_data = gzip.compress(byte_string)
@@ -30,23 +39,23 @@ def test_gzip_compression(self, monkeypatch):
         def modify_stream(Bucket=None, Key=None, Fileobj=None):
             Fileobj.write(gzip_data)
 
-        client = MagicMock()
-        boto3 = MagicMock(client=MagicMock(return_value=client))
-        monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
-        client.download_fileobj.side_effect = modify_stream
-
+        mocked_boto_client.download_fileobj.side_effect = modify_stream
         returned_data = task.run("key", compression="gzip")
         assert returned_data == str(byte_string, "utf-8")
 
-    def test_raises_on_invalid_compression_method(self, monkeypatch):
+    def test_raises_on_invalid_compression_method(self, mocked_boto_client):
         task = S3Download("test")
-        client = MagicMock()
-        boto3 = MagicMock(client=MagicMock(return_value=client))
-        monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
-
         with pytest.raises(ValueError, match="gz_fake"):
             task.run("key", compression="gz_fake")
 
+    def test_boto3_client_is_created_with_session(self, mocked_boto_client):
+        """ Tests the fix for #3925 """
+        task = S3Download("test")
+        result = task.run("key")
+        assert (
+            "session.Session()" in mocked_boto_client.return_value._extract_mock_name()
+        )
+
 
 class TestS3Upload:
     def test_initialization(self):
@@ -62,11 +71,8 @@ def test_raises_if_bucket_not_eventually_provided(self):
         with pytest.raises(ValueError, match="bucket"):
             task.run(data="")
 
-    def test_generated_key_is_str(self, monkeypatch):
+    def test_generated_key_is_str(self, mocked_boto_client):
         task = S3Upload(bucket="test")
-        client = MagicMock()
-        boto3 = MagicMock(client=MagicMock(return_value=client))
-        monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
         with set_temporary_config({"cloud.use_local_secrets": True}):
             with prefect.context(
                 secrets=dict(
@@ -74,30 +80,30 @@ def test_generated_key_is_str(self, monkeypatch):
                 )
             ):
                 task.run(data="")
-        assert type(client.upload_fileobj.call_args[1]["Key"]) == str
+        assert type(mocked_boto_client.upload_fileobj.call_args[1]["Key"]) == str
 
-    def test_gzip_compression(self, monkeypatch):
+    def test_gzip_compression(self, mocked_boto_client):
         task = S3Upload("bucket")
         byte_string = b"col1,col2,col3\nfake,data,1\nfalse,info,2\n"
 
-        client = MagicMock()
-        boto3 = MagicMock(client=MagicMock(return_value=client))
-        monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
-
         task.run(byte_string, key="key", compression="gzip")
-        args, kwargs = client.upload_fileobj.call_args_list[0]
+        args, kwargs = mocked_boto_client.upload_fileobj.call_args_list[0]
         gzip_data_stream = args[0]
         assert gzip.decompress(gzip_data_stream.read()) == byte_string
 
-    def test_raises_on_invalid_compression_method(self, monkeypatch):
+    def test_raises_on_invalid_compression_method(self, mocked_boto_client):
         task = S3Upload("test")
-        client = MagicMock()
-        boto3 = MagicMock(client=MagicMock(return_value=client))
-        monkeypatch.setattr("prefect.utilities.aws.boto3", boto3)
-
         with pytest.raises(ValueError, match="gz_fake"):
             task.run(b"data", compression="gz_fake")
 
+    def test_boto3_client_is_created_with_session(self, mocked_boto_client):
+        """ Tests the fix for #3925 """
+        task = S3Upload("test")
+        result = task.run("key")
+        assert (
+            "session.Session()" in mocked_boto_client.return_value._extract_mock_name()
+        )
+
 
 class TestS3List:
     def test_initialization(self):