Repository: PrefectHQ/prefect

Issue:
S3ResultHandler fails during Cloud Flow Run
## Description
Running on `0.9.8` we've observed the S3ResultHandler failing with `AttributeError: 'S3ResultHandler' object has no attribute '_client'`. This was also reported by another user in the Prefect Community Slack. (See thread: https://prefect-community.slack.com/archives/CL09KU1K7/p1584980231422300)

For now we are working around this by disabling checkpointing in all of our tasks, but this will obviously prevent tasks from being able to retry, etc.

## Expected Behavior
S3ResultHandler should correctly checkpoint tasks to store outputs in an S3 bucket.

## Reproduction
The following is a stub Flow that recreates the issue:

```
import prefect
from prefect import Flow
from prefect.tasks.shell import ShellTask
from prefect.environments.storage import S3
from prefect.engine.result_handlers import S3ResultHandler

env = <an environment, we use our own secure Dask environment for TLS by extending RemoteEnvironment>

BUCKET = "<redacted>"
s3_storage = S3(bucket=BUCKET)
rh = S3ResultHandler(BUCKET)

shell_task = ShellTask(name="shell", return_all=True)

with Flow("RecreateS3ResultHandlerIssue", environment=env, storage=s3_storage, result_handler=rh) as flow:
    final_cmd = ["ls","cd;ls"]
    embulk_results = shelltask.map(command=final_cmd)

s3_storage.build()
flow.register(project_name="Test Project 1")

# Now run Flow from Cloud either via UI or API
```

The Flow run will fail with this full stack trace:

```
Unexpected error: AttributeError("'S3ResultHandler' object has no attribute '_client'")
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 925, in get_task_run_state
    state._result.store_safe_value()
  File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result.py", line 121, in store_safe_value
    value = self.result_handler.write(self.value)
  File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result_handlers/s3_result_handler.py", line 112, in write
    self.client.upload_fileobj(stream, Bucket=self.bucket, Key=uri)
  File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result_handlers/s3_result_handler.py", line 77, in client
    return self._client
AttributeError: 'S3ResultHandler' object has no attribute '_client'
```

## Environment
We use a long-running Dask cluster on k8s via AWS EKS.

(Hat tip to @JLouSRM for identifying this issue.)


Discussion:
Hi! I will take a look to reproduce this evening, I changed the S3ResultHandler in 0.9.8 re: trying to thread safe the boto3 client, so I'm the one who probably broke it. 
Looking at it I think there's a chance the `client` property is retrieved without `self._client` being set prior. https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/result_handlers/s3_result_handler.py#L77

Possibly defining `self._client = None` in the `__init__` would do the trick

Resolving patch:
diff --git a/src/prefect/engine/result_handlers/s3_result_handler.py b/src/prefect/engine/result_handlers/s3_result_handler.py
--- a/src/prefect/engine/result_handlers/s3_result_handler.py
+++ b/src/prefect/engine/result_handlers/s3_result_handler.py
@@ -36,6 +36,7 @@ class S3ResultHandler(ResultHandler):
     def __init__(self, bucket: str, aws_credentials_secret: str = None) -> None:
         self.bucket = bucket
         self.aws_credentials_secret = aws_credentials_secret
+        self._client = None
         super().__init__()
 
     def initialize_client(self) -> None:
@@ -71,9 +72,10 @@ def client(self) -> "boto3.client":
         Initializes a client if we believe we are in a new thread.
         We consider ourselves in a new thread if we haven't stored a client yet in the current context.
         """
-        if not prefect.context.get("boto3client"):
+        if not prefect.context.get("boto3client") or not self._client:
             self.initialize_client()
             prefect.context["boto3client"] = self._client
+
         return self._client
 
     @client.setter


Test patch:
diff --git a/tests/engine/result_handlers/test_result_handlers.py b/tests/engine/result_handlers/test_result_handlers.py
--- a/tests/engine/result_handlers/test_result_handlers.py
+++ b/tests/engine/result_handlers/test_result_handlers.py
@@ -243,6 +243,19 @@ def __getstate__(self):
             res = cloudpickle.loads(cloudpickle.dumps(handler))
             assert isinstance(res, S3ResultHandler)
 
+    def test_s3_uninitialized_client(self, session):
+        handler = S3ResultHandler(
+            bucket="bob", aws_credentials_secret="AWS_CREDENTIALS"
+        )
+        assert handler.bucket == "bob"
+
+        with prefect.context(
+            secrets=dict(AWS_CREDENTIALS=dict(ACCESS_KEY=1, SECRET_ACCESS_KEY=42)),
+            boto3client="test",
+        ):
+            with set_temporary_config({"cloud.use_local_secrets": True}):
+                assert handler.client is not None
+
 
 @pytest.mark.xfail(raises=ImportError, reason="azure extras not installed.")
 class TestAzureResultHandler: