Repository: kserve/kserve

Issue:
storage-initialiser fails to copy model from S3
Upon using a Secret and Service account that look like this:

```
apiVersion: v1
metadata:
  name: kfserving-aws-secret
  namespace: {namespace}
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.eu-east-1.amazonaws.com
    serving.kubeflow.org/s3-usehttps: "1"
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: eu-east-1
data:
  AWS_ACCESS_KEY_ID: {access_key_base64}
  AWS_SECRET_ACCESS_KEY: {secret_key_base64}
kind: Secret
type: Opaque
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: {namespace}
secrets:
  - name: kfserving-aws-secret  
```
...and an InferenceService like this:

```
apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: {model_name}
  namespace: {namespace}
spec:
  default:
    predictor:
      serviceAccountName: kfserving-sa
      tensorflow:
        storageUri: {model_base_path}    
```

...the model fails to serve, with the `storage-initialiser` pod throwing the following error:

```
[I 200226 14:09:42 initializer-entrypoint:13] Initializing, args: src_uri [s3://at-kubeflow-training/tf-job/ksc-ebce76a1] dest_path[ [/mnt/models]
[I 200226 14:09:42 storage:35] Copying contents of s3://at-kubeflow-training/tf-job/ksc-ebce76a1 to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
    Storage._download_s3(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
    client = Storage._create_minio_client()
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
    secure=use_ssl)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 150, in __init__
    is_valid_endpoint(endpoint)
  File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 301, in is_valid_endpoint
    if hostname[-1] == '.':
IndexError: string index out of range
```

This was using Kubeflow 1.0.0RC4, KFServing 0.2.2 and KNative 0.11.1


Discussion:
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| bug | 0.88 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
https://www.kubeflow.org/docs/aws/aws-e2e/ shows an example of the `kind: Secret` which uses

```
data:
  # echo -ne "AKIAxxx" | base64
  awsAccessKeyID: QUtJQVhxxxVXVjQ=
  awsSecretAccessKey: QzR0UnxxxVNOd0NQQQ==
```

I've just used the original AWS variable naming successfully (at least in terms of storage-initializer not failing with `IndexError: string index out of range`) with minio.

Let me post a typical minio setup (https://github.com/kubeflow/kfserving/tree/master/docs/samples/s3) in case someone needs it

```
apiVersion: v1
kind: Secret
metadata:
  name: kfserving-minio-secret
  namespace: anonymous
  annotations:
    serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000
    serving.kubeflow.org/s3-usehttps: "0"
    serving.kubeflow.org/s3-verifyssl: "0"
    serving.kubeflow.org/s3-region: us-east-1
type: Opaque
stringData:
  awsAccessKeyID: minio
  awsSecretAccessKey: minio123
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: anonymous
secrets:
  - name: kfserving-minio-secret
```
the default s3 secret key names expected by kfserving is configured here
https://github.com/kubeflow/kfserving/blob/master/config/default/configmap/inferenceservice.yaml#L89
Ok thanks for the input. Is it necessary to use Minio at all here? I'm not really that familiar with how Minio works exactly. On my cluster I have access directly to S3, and normally would use an endpoint like "s3.eu-east-1.amazonaws.com". Are you saying that I need to use Minio as a go-between?

Also, doesn't it make more sense to use by default the variable names  AWS_ACCESS_KEY_ID and   AWS_SECRET_ACCESS_KEY as that is what is commonly used by applications as environment variables for AWS credentials?
Minio is not a requirement, the idea is to support any S3 compatible storage. The above naming convention is for environment variable, here we are using it for the secret data key name, but you are free to configure what you want.
Yes I know that it is for a Secret, but every other application I've seen
that uses Secrets to pass AWS credentials have used the standard naming
convention. I realise I can change it myself, I just find it strange that
you use a different convention here.

On Thu, 27 Feb 2020 at 11:33, Dan Sun <notifications@github.com> wrote:

> Minio is not a requirement, the idea is to support any S3 compatible
> storage. The above naming convention is for environment variable, here we
> are using it for the secret data key name, but you are free to configure
> what you want.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/kubeflow/kfserving/issues/696?email_source=notifications&email_token=AGBE4ODQ3BR5HGQJTFDJYC3RE6JHJA5CNFSM4K4GNHZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND2P7Q#issuecomment-591898622>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGBE4OGHOYNBYBT6MFISHW3RE6JHJANCNFSM4K4GNHZQ>
> .
>

I think the name is copied from original kubeflow s3 example since we want to keep consistent, I just checked it is changed there too, so feel free to raise a PR to fix the default then. Thanks for brought this up. 

Resolving patch:
diff --git a/python/kfserving/kfserving/constants/constants.py b/python/kfserving/kfserving/constants/constants.py
--- a/python/kfserving/kfserving/constants/constants.py
+++ b/python/kfserving/kfserving/constants/constants.py
@@ -30,8 +30,8 @@
 DEFAULT_SA_NAME = "kfserving-service-credentials"
 
 # S3 credentials constants
-S3_ACCESS_KEY_ID_DEFAULT_NAME = "awsAccessKeyID"
-S3_SECRET_ACCESS_KEY_DEFAULT_NAME = "awsSecretAccessKey"
+S3_ACCESS_KEY_ID_DEFAULT_NAME = "AWS_ACCESS_KEY_ID"
+S3_SECRET_ACCESS_KEY_DEFAULT_NAME = "AWS_SECRET_ACCESS_KEY"
 S3_DEFAULT_CREDS_FILE = '~/.aws/credentials'
 
 # GCS credentials constants


Test patch: