Repository: kserve/kserve

Issue:
Anonymous credentials for single model serving with s3
/kind feature

**Describe the solution you'd like**
[A clear and concise description of what you want to happen.]
Update [download from s3](https://github.com/kubeflow/kfserving/blob/4fd401e348174f954fee7596dc3d54f197f4aa8c/python/kfserving/kfserving/storage.py#L86) to configure boto3 to use anon credentials if anonymous credentials env is set to true.

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]
Would be nice if anon e2e test were also added.


Discussion:
May I be assigned this issue?
> May I be assigned this issue?

@mkbhanda I don't believe anyone is working on this issue so by all means go for it! :) 
Thanks @Aaronchoo!
1)  is there already an environment variable defined for Anon?
The only ones I found with grep are: "AZ_TENANT_ID", "AZ_CLIENT_ID", "AZ_CLIENT_SECRET", AZ_SUBSCRIPTION_ID",
python/kfserving/kfserving/storage.py:        headers_json = os.getenv(host_uri + _HEADERS_SUFFIX, "{}")
2) If the Anon environment variable is set and credentials as above, my guess is to use the credentials? Is that the desired behavior?
3) With rest to additional test cases: no anon or credentials set, anon set, anon & credentials set.
@Aaronchoo you will have to do the assign
/assign @mkbhanda 
Hi @mkbhanda!

To answer your questions:
1. The environment variable is [defined in s3_secret.go](https://github.com/kubeflow/kfserving/blob/master/pkg/credentials/s3/s3_secret.go#L35). You will need to redefine the variable in `storage.py`. The environmental variable is [appended on this line](https://github.com/kubeflow/kfserving/blob/master/pkg/credentials/s3/s3_secret.go#L137).
2. The current desired behaviour is to use anonymous credentials when the annotation is define AND set to `True`. We currently do this for multi-model serving so we should follow the same behaviour in `storage.py`.
3. There is a comment that sorta explained it in the [s3 storage documentation](https://github.com/kubeflow/kfserving/blob/master/docs/samples/storage/s3/README.md#create-s3-secret-and-attach-to-service-account)  (If this is not clear enough feel free to update the documentation) but to be clear I will write out the cases!
- Credentials set + anon set to True: Use anonymous credentials instead
- Credentials set + anon set to False: Use credentials
- Credentials set + anon not defined: Use credentials
- Credentials not set + anon set to True: Use anonymous credentials
- Credentials not set + anon set to False: Use credentials [boto3 searches for it](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials)
- Credentials not set + anon not defined: Use credentials [boto3 searches for it](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials)

Essentially always try to use credentials unless the anon env var is defined and set to `True`.

Hopefully this helps! Thanks :smile: 

Thank you for the detailed desired behavior and assigning @Aaronchoo !
@matty-rose and @mttcnnff on examining test_storage.py and test_s3_storage.py my understanding is that credentials are not considered as part of mocking. Am I missing something?

Moto is a popular library to interact with AWS infrastructure: https://github.com/spulec/moto and been around since 2014.

Based on @Aaronchoo careful explanation of scenarios,  there are 4 cases, env-with-anon-true (disregards secrets in env/config file etc), env-with-secret-credentials, env-clear (boto searches).  Would need to in addition create an S3 object each of type public and private to flex the access options. Can use a python fixture here. We may have some test case bloat!

The essence of my changes in storage.py are:
@staticmethod
    def _download_s3(uri, temp_dir: str):
        # anon environment variable defined in s3_secret.go 
        anon = ("True" == os.getenv("awsAnonymousCredential", "false").capitalize())
        config = None
        if anon:
            config = Config(signature_version=UNSIGNED)
        s3 = boto3.resource('s3', 
                            endpoint_url=os.getenv("AWS_ENDPOINT_URL", "http://s3.amazonaws.com"),
                            config=config)


Resolving patch:
diff --git a/python/kfserving/kfserving/storage.py b/python/kfserving/kfserving/storage.py
--- a/python/kfserving/kfserving/storage.py
+++ b/python/kfserving/kfserving/storage.py
@@ -25,6 +25,8 @@
 import zipfile
 from urllib.parse import urlparse
 
+from botocore.client import Config
+from botocore import UNSIGNED
 import boto3
 import requests
 from azure.storage.blob import BlockBlobService
@@ -81,9 +83,26 @@ def download(uri: str, out_dir: str = None) -> str:
         logging.info("Successfully copied %s to %s", uri, out_dir)
         return out_dir
 
+    @staticmethod
+    def get_S3_config():
+        # anon environment variable defined in s3_secret.go
+        anon = ("True" == os.getenv("awsAnonymousCredential", "false").capitalize())
+        if anon:
+            return Config(signature_version=UNSIGNED)
+        else:
+            return None
+
     @staticmethod
     def _download_s3(uri, temp_dir: str):
-        s3 = boto3.resource('s3', endpoint_url=os.getenv("AWS_ENDPOINT_URL", "http://s3.amazonaws.com"))
+        # Boto3 looks at various configuration locations until it finds configuration values.
+        # lookup order:
+        # 1. Config object passed in as the config parameter when creating S3 resource
+        #    if awsAnonymousCredential env var true, passed in via config
+        # 2. Environment variables
+        # 3. ~/.aws/config file
+        s3 = boto3.resource('s3',
+                            endpoint_url=os.getenv("AWS_ENDPOINT_URL", "http://s3.amazonaws.com"),
+                            config=Storage.get_S3_config())
         parsed = urlparse(uri, scheme='s3')
         bucket_name = parsed.netloc
         bucket_path = parsed.path.lstrip('/')


Test patch:
diff --git a/python/kfserving/test/test_s3_storage.py b/python/kfserving/test/test_s3_storage.py
--- a/python/kfserving/test/test_s3_storage.py
+++ b/python/kfserving/test/test_s3_storage.py
@@ -12,8 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
 import unittest.mock as mock
 
+from botocore.client import Config
+from botocore import UNSIGNED
 import kfserving
 
 
@@ -104,3 +107,37 @@ def test_full_name_key(mock_storage):
                                                [object_key])
 
     mock_boto3_bucket.objects.filter.assert_called_with(Prefix=object_key)
+
+
+AWS_TEST_CREDENTIALS = {"AWS_ACCESS_KEY_ID": "testing",
+                        "AWS_SECRET_ACCESS_KEY": "testing",
+                        "AWS_SECURITY_TOKEN": "testing",
+                        "AWS_SESSION_TOKEN": "testing"}
+
+
+def test_get_S3_config():
+
+    ANON_CONFIG = Config(signature_version=UNSIGNED)
+    DEFAULT_CONFIG = None
+
+    with mock.patch.dict(os.environ, {}):
+        config1 = kfserving.Storage.get_S3_config()
+    assert config1 == DEFAULT_CONFIG
+
+    with mock.patch.dict(os.environ, {"awsAnonymousCredential": "False"}):
+        config2 = kfserving.Storage.get_S3_config()
+    assert config2 == DEFAULT_CONFIG
+
+    with mock.patch.dict(os.environ, AWS_TEST_CREDENTIALS):
+        config3 = kfserving.Storage.get_S3_config()
+    assert config3 == DEFAULT_CONFIG
+
+    with mock.patch.dict(os.environ, {"awsAnonymousCredential": "True"}):
+        config4 = kfserving.Storage.get_S3_config()
+    assert config4.signature_version == ANON_CONFIG.signature_version
+
+    # assuming Python 3.5 or greater for joining dictionaries
+    credentials_and_anon = {**AWS_TEST_CREDENTIALS, "awsAnonymousCredential": "True"}
+    with mock.patch.dict(os.environ, credentials_and_anon):
+        config5 = kfserving.Storage.get_S3_config()
+    assert config5.signature_version == ANON_CONFIG.signature_version