Repository: bentoml/BentoML

Issue:
Cannot retrieve BentoService from S3 Minio
**Describe the bug**
I have a Yatai-Service configured with S3 Minio and local Database (Note - I would normally use PostgresSQL - but cannot do that at the moment due to some other unrelated issue).

I run the following in my docker build process:
```
bentoml config set yatai_service.url=http://yatai-url
bentoml retrieve Service:20201014125939_6C7163 --target_dir /bentoservice 

```
However - I run into the following error:`botocore .exceptions.NoCredentialsError: Unable to locate credentials`
Why is it expecting the S3 credentials ? This was working earlier.

Here is the full stack trace. 
```
Step 3/5 : RUN bentoml retrieve Service:20201014125939_6C7163 --target_dir /bentoservice
 ---> Running in e46b6f76e28a
Traceback (most recent call last):
  File "/usr/local/bin/bentoml", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/bentoml/cli/click_utils.py", line 138, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/bentoml/cli/click_utils.py", line 115, in wrapper
    return_value = func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/bentoml/cli/click_utils.py", line 99, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/bentoml/cli/bento_management.py", line 267, in retrieve
    safe_retrieve(bento_service_bundle_path, target_dir)
  File "/usr/local/lib/python3.8/site-packages/bentoml/saved_bundle/loader.py", line 227, in safe_retrieve
    with _resolve_remote_bundle_path(bundle_path) as local_bundle_path:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/bentoml/saved_bundle/loader.py", line 60, in _resolve_remote_bundle_path
    s3.download_fileobj(bucket_name, object_name, fileobj)
  File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
    response = client.head_object(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.8/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.8/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
The command '/bin/sh -c bentoml retrieve Service:20201014125939_6C7163 --target_dir /bentoservice' returned a non-zero code: 1
```

Initially, I thought this had something to do with my docker build environment - but then I was able to recreate the same error on my windows laptop.

To debug further - On my windows laptop - I added an .aws credentials file - 'C:/Users/my_userid/.aws/credentials' and retried the retrieve operation. In this case - it went past the above error - but now fails with the following:

`botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden`

See full stack trace below:
```
[2020-10-14 14:42:11,774] DEBUG - Setting debug mode: ON for current session
[2020-10-14 14:42:12,575] DEBUG - Connecting YataiService gRPC server at: http://yatai-url
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\pyvenvs\model_deployment\Scripts\bentoml.exe\__main__.py", line 7, in <module>
  File "c:\pyvenvs\model_deployment\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\pyvenvs\model_deployment\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\pyvenvs\model_deployment\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\pyvenvs\model_deployment\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\cli\click_utils.py", line 138, in wrapper
    return func(*args, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\cli\click_utils.py", line 115, in wrapper
    return_value = func(*args, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\cli\click_utils.py", line 99, in wrapper
    return func(*args, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\cli\bento_management.py", line 267, in retrieve
    safe_retrieve(bento_service_bundle_path, target_dir)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\saved_bundle\loader.py", line 227, in safe_retrieve
    with _resolve_remote_bundle_path(bundle_path) as local_bundle_path:
  File "C:\Program Files\Python38\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "c:\pyvenvs\model_deployment\lib\site-packages\bentoml\saved_bundle\loader.py", line 60, in _resolve_remote_bundle_path
    s3.download_fileobj(bucket_name, object_name, fileobj)
  File "c:\pyvenvs\model_deployment\lib\site-packages\boto3\s3\inject.py", line 678, in download_fileobj
    return future.result()
  File "c:\pyvenvs\model_deployment\lib\site-packages\s3transfer\futures.py", line 106, in result
    return self._coordinator.result()
  File "c:\pyvenvs\model_deployment\lib\site-packages\s3transfer\futures.py", line 265, in result
    raise self._exception
  File "c:\pyvenvs\model_deployment\lib\site-packages\s3transfer\tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\s3transfer\download.py", line 340, in _submit
    response = client.head_object(
  File "c:\pyvenvs\model_deployment\lib\site-packages\botocore\client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\pyvenvs\model_deployment\lib\site-packages\botocore\client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
```


**Environment:**
 - OS: Windows / RHEL
 - Python Version -  3.8.x
 - BentoML-0.9.1


Discussion:
I want to add that when I rolled back to v0.8.6 of BentoML - the errors go away. 

Resolving patch:
diff --git a/bentoml/cli/bento_management.py b/bentoml/cli/bento_management.py
--- a/bentoml/cli/bento_management.py
+++ b/bentoml/cli/bento_management.py
@@ -259,7 +259,7 @@ def retrieve(bento, target_dir):
 
         if get_bento_result.bento.uri.s3_presigned_url:
             bento_service_bundle_path = get_bento_result.bento.uri.s3_presigned_url
-        if get_bento_result.bento.uri.gcs_presigned_url:
+        elif get_bento_result.bento.uri.gcs_presigned_url:
             bento_service_bundle_path = get_bento_result.bento.uri.gcs_presigned_url
         else:
             bento_service_bundle_path = get_bento_result.bento.uri.uri


Test patch:
diff --git a/e2e_tests/yatai_server/test_sqlite_gcs.py b/e2e_tests/yatai_server/test_sqlite_gcs.py
--- a/e2e_tests/yatai_server/test_sqlite_gcs.py
+++ b/e2e_tests/yatai_server/test_sqlite_gcs.py
@@ -7,6 +7,7 @@
     local_yatai_server,
     get_bento_service_info,
     execute_bentoml_run_command,
+    execute_bentoml_retrieve_command,
 )
 
 logger = logging.getLogger('bentoml.test')
@@ -29,6 +30,11 @@ def test_yatai_server_with_sqlite_and_gcs():
             get_svc_result.bento.uri.type == BentoUri.GCS
         ), 'BentoService storage type mismatched, expect GCS'
 
+        retrieve_svc_result = execute_bentoml_retrieve_command(
+            f'{svc.name}:{svc.version}'
+        )
+        assert retrieve_svc_result.startswith(f'Service {svc.name} artifact directory')
+
         logger.info('Validate BentoService prediction result')
         run_result = execute_bentoml_run_command(bento_tag, '[]')
         logger.info(run_result)
diff --git a/e2e_tests/yatai_server/test_sqlite_s3.py b/e2e_tests/yatai_server/test_sqlite_s3.py
--- a/e2e_tests/yatai_server/test_sqlite_s3.py
+++ b/e2e_tests/yatai_server/test_sqlite_s3.py
@@ -7,6 +7,7 @@
     local_yatai_server,
     get_bento_service_info,
     execute_bentoml_run_command,
+    execute_bentoml_retrieve_command,
 )
 
 logger = logging.getLogger('bentoml.test')
@@ -32,6 +33,10 @@ def test_yatai_server_with_sqlite_and_s3():
         assert (
             get_svc_result.bento.uri.type == BentoUri.S3
         ), 'BentoService storage type mismatched, expect S3'
+        retrieve_svc_result = execute_bentoml_retrieve_command(
+            f'{svc.name}:{svc.version}'
+        )
+        assert retrieve_svc_result.startswith(f'Service {svc.name} artifact directory')
 
         logger.info('Validate BentoService prediction result')
         run_result = execute_bentoml_run_command(bento_tag, '[]')
diff --git a/e2e_tests/yatai_server/utils.py b/e2e_tests/yatai_server/utils.py
--- a/e2e_tests/yatai_server/utils.py
+++ b/e2e_tests/yatai_server/utils.py
@@ -35,6 +35,25 @@ def execute_bentoml_run_command(bento_tag, data, api="predict"):
     return stdout
 
 
+def execute_bentoml_retrieve_command(bento_tag):
+    dir_name = uuid.uuid4().hex[:8]
+    with TempDirectory() as temp_dir:
+        command = [
+            'bentoml',
+            'retrieve',
+            bento_tag,
+            '--target_dir',
+            f'{temp_dir}/{dir_name}',
+        ]
+        proc = subprocess.Popen(
+            command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ
+        )
+        stdout = proc.stdout.read().decode('utf-8')
+        print(stdout)
+        print(proc.stderr.read().decode('utf-8'))
+        return stdout
+
+
 @contextlib.contextmanager
 def modified_environ(*remove, **update):
     """