Repository: PrefectHQ/prefect

Issue:
S3Result with target raises error
## Archived from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**livni.itay**: Hi - I am working with `S3Result`  and receiving a

```
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist
```

Which upon further research - it  can be anything including a permission error. (I tried different buckets with settings) The credentials are stored as AWS_CREDENTIALS in prefect cloud.  With the config.toml set to use cloud secrets

```
[cloud]
use_local_secrets = false
```

Switching back to `result_handler` argument with `S3Result` subclass *did work,* . And combining `result handler` with `target` does not.  Is there something different in the way that credentials are handled between `result` and `result_handler`?

The new prefect is really nice :slightly_smiling_face:

**chris**: Hi itay - could you share the code you used to initialize the `result_handler` and the `result`?

**livni.itay**: `tsx_imb_res = S3Result(bucket="tsx-moc-bcp")`

**livni.itay**: 
```
@task(
    max_retries=3,
    retry_delay=timedelta(seconds=1), # In production this will be change to 20 minutes
    result_handler=tsx_imb_res, 
    target="{task_name}-{today}",
    state_handlers=[imb_handler, error_handler]
)
```



**livni.itay**: Works with `target` commented out

**chris**: Ah! The `result_handler` kwarg is now deprecated, so you should instead try:

```
...
result=tsx_imb_res,
...
```


**livni.itay**: Right that does not work

**livni.itay**: That is the problem

**chris**: ahhh interesting! OK so this might actually be a bug with our `exists` logic on the `S3Result` type.  Would you mind sharing this example code + the traceback you’re seeing?  Sorry about that!

**livni.itay**: Actually it looks like I am not using `target` right?

```
[2020-05-16 21:31:37] INFO - prefect.FlowRunner | Beginning Flow run for 'Our first flow'
[2020-05-16 21:31:37] INFO - prefect.FlowRunner | Starting flow run.
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'tsx_url': Starting task run...
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'tsx_url': finished task run for task with final state: 'Success'
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'get_tsx_moc_imb': Starting task run...
[2020-05-16 21:31:38] ERROR - prefect.TaskRunner | Unexpected error: NoSuchKey('An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.')
Traceback (most recent call last):
  File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 651, in check_target
    if result.exists(target, **prefect.context):
  File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 167, in exists
    Bucket=self.bucket, Key=location.format(**kwargs)
  File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
[2020-05-16 21:31:38] INFO - prefect.TaskRunner | Task 'get_tsx_moc_imb': finished task run for task with final state: 'Skipped'
[2020-05-16 21:31:38] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
```



**livni.itay**: `target="{task_name}-{today}",`

**chris**: your code looks alright to me actually, including your `target` specification; I think this exception catching logic here is flawed: <https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/results/s3_result.py#L169>

**chris**: it’s possible this was tested on a different version of `boto3` or something, we’ll need to investigate a little deeper

**livni.itay**: Cool.  Let me know if you need anything more.

**chris**: I’ll use our bot to open the issue and we can track progress there

**chris**: <@ULVA73B9P> archive “S3Result with target raises error”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1589663959085200?thread_ts=1589663959.085200&cid=CL09KU1K7).




Discussion:
Opening because this is still an active issue
From the thread, relevant package versions are:
```
botocore: 1.15.32
boto3: 1.12.32
```
Oh interesting, looks like we'll want to check something like:
```python
if ex.response['Error']['Code'] == 'NoSuchKey':
    return False
```

or

```python
except client.exceptions.NoSuchKey
```

boto3 has minimal documentation on this

Resolving patch:
diff --git a/src/prefect/engine/results/s3_result.py b/src/prefect/engine/results/s3_result.py
--- a/src/prefect/engine/results/s3_result.py
+++ b/src/prefect/engine/results/s3_result.py
@@ -167,7 +167,7 @@ def exists(self, location: str, **kwargs: Any) -> bool:
                 Bucket=self.bucket, Key=location.format(**kwargs)
             ).load()
         except botocore.exceptions.ClientError as exc:
-            if exc.response["Error"]["Code"] == "404":
+            if exc.response["Error"]["Code"] == "NoSuchKey":
                 return False
             raise
         except Exception as exc:


Test patch:
diff --git a/tests/engine/results/test_s3_result.py b/tests/engine/results/test_s3_result.py
--- a/tests/engine/results/test_s3_result.py
+++ b/tests/engine/results/test_s3_result.py
@@ -79,7 +79,7 @@ def test_s3_result_does_not_exist(self, session):
         import botocore
 
         exc = botocore.exceptions.ClientError(
-            {"Error": {"Code": "404"}}, "list_objects"
+            {"Error": {"Code": "NoSuchKey"}}, "list_objects"
         )
 
         class _client:
@@ -98,7 +98,7 @@ def test_s3_result_exists(self, session):
         import botocore
 
         exc = botocore.exceptions.ClientError(
-            {"Error": {"Code": "404"}}, "list_objects"
+            {"Error": {"Code": "NoSuchKey"}}, "list_objects"
         )
 
         class _client: