Repository: kserve/kserve

Issue:
S3 storage-initializer, secret/annotations and AWS_REGION
/kind bug

**What steps did you take and what happened:**
Hi everybody! I am trying to deploy kfserving 0.6.0 standalone with the storage-initializer to fetch models from a local Swift cluster (exposing an S3 API compatible with v4 signature). I followed the [tutorial](https://github.com/kubeflow/kfserving/blob/master/docs/samples/storage/s3/README.md) and added the annotation for the `s3-region` as described (in my case, it needs to be `US` to avoid signature validation failures when contacting Swift). I checked after the k8s deploy that they were applied correctly to the container, and I found the following (as expected):

```
Init Containers:
  storage-initializer:
    [..]
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      AWS_ACCESS_KEY_ID:      <set to the key 'AWS_ACCESS_KEY_ID' in secret 'test-secret'>      Optional: false
      AWS_SECRET_ACCESS_KEY:  <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'test-secret'>  Optional: false
      S3_USE_HTTPS:           1
      S3_ENDPOINT:            thanos-swift.discovery.wmnet
      AWS_ENDPOINT_URL:       https://thanos-swift.discovery.wmnet
      AWS_REGION:             US
    [..]
```
But the storage-initializer container still failed to fetch the model from Swift, due to the following error:

```
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListObjects operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
```
The above can be easily reproduced via a python-venv with boto3, testing something like (of course in this case it is my internal endpoint, to repro one needs to use another one like AWS etc..):
```
import boto3

# Retrieve the list of existing buckets
s3 = boto3.client('s3', aws_access_key_id="XXXXXXX", 
                  aws_secret_access_key="XXXXXXXX",
                  endpoint_url="https://thanos-swift.discovery.wmnet")
response = s3.list_buckets()
```
The above doesn't work if I export `AWS_REGION="US"`, but it does if I export `AWS_DEFAULT_REGION="US"`. This seems to be supported by https://github.com/boto/boto3/issues/2574 (if you check the attached pull request the code mentions the AWS_DEFAULT_REGION env variable only).

I am wondering if we should just sed/replace AWS_REGION with AWS_DEFAULT_REGION (given that we use boto) and possibly release 0.6.1, or if there is another workaround that I haven't checked. Happy to send a pull request in case, or happy to see others doing it, no preferences :)

**What did you expect to happen:**
I expected the storage-initializer to honor the AWS_REGION setting and fetch from Swift/S3 correctly.

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version: 1.9.5
- Knative Version: 0.18.1
- KFServing Version: 0.6.0
- Kubeflow version: none
- Kubernetes version: (use `kubectl version`): 1.16
- OS (e.g. from `/etc/os-release`): Debian 10 (Buster)



Discussion:

Resolving patch:
diff --git a/python/kfserving/kfserving/api/creds_utils.py b/python/kfserving/kfserving/api/creds_utils.py
--- a/python/kfserving/kfserving/api/creds_utils.py
+++ b/python/kfserving/kfserving/api/creds_utils.py
@@ -56,7 +56,7 @@ def set_gcs_credentials(namespace, credentials_file, service_account):
 def set_s3_credentials(namespace, credentials_file, service_account,
                        s3_profile='default',  # pylint: disable=too-many-locals,too-many-arguments
                        s3_endpoint=None, s3_region=None, s3_use_https=None,
-                       s3_verify_ssl=None):  # pylint: disable=unused-argument
+                       s3_verify_ssl=None, s3_cabundle=None):  # pylint: disable=unused-argument
     """
     Set S3 Credentails (secret and service account).
     Args:
@@ -67,9 +67,10 @@ def set_s3_credentials(namespace, credentials_file, service_account,
                               is specified, will attach created secret with the service account,
                               otherwise will create new one and attach with created secret.
         s3_endpoint(str): S3 settings variable S3_ENDPOINT.
-        s3_region(str): S3 settings variable AWS_REGION.
+        s3_region(str): S3 settings variable AWS_DEFAULT_REGION.
         s3_use_https(str): S3 settings variable S3_USE_HTTPS.
         s3_verify_ssl(str): S3 settings variable S3_VERIFY_SSL.
+        s3_cabundle(str): S3 settings variable AWS_CA_BUNDLE.
     """
 
     config = configparser.ConfigParser()
@@ -99,6 +100,7 @@ def set_s3_credentials(namespace, credentials_file, service_account,
         's3_region': constants.KFSERVING_GROUP + "/s3-region",
         's3_use_https': constants.KFSERVING_GROUP + "/s3-usehttps",
         's3_verify_ssl': constants.KFSERVING_GROUP + "/s3-verifyssl",
+        's3_cabundle': constants.KFSERVING_GROUP + "/s3-cabundle",
     }
 
     s3_annotations = {}


Test patch: