Repository: kserve/kserve

Issue:
Calling kfserving.Storage.download(src_uri, dest_path) for an S3 URI without having the environment variable AWS_ENDPOINT_URL set will always fail.
/kind bug

Calling  kfserving.Storage.download(src_uri, dest_path) for an S3 URI *without* having the environment variable `AWS_ENDPOINT_URL` set will always fail.

**What steps did you take and what happened:**

The bug manifested from a use of an MLFLOW_SERVER prepackaged Seldon server where the model resided in S3 and the `envSecretRefName` included the `AWS_KEY`, `AWS_SECRET_KEY` and `S3_ENDPOINT` all properly defined with valid values.


**What did you expect to happen:**

The S3 file should be downloaded (in this use case above this means the Seldon model server loads the model file from S3 during the container intitialization).

**Anything else you would like to add:**

The root cause appears to be the bad default used when creating the Minio client:

```
   def _create_minio_client():
        # Remove possible http scheme for Minio
        url = urlparse(os.getenv("AWS_ENDPOINT_URL", "s3.amazonaws.com"))
        use_ssl = url.scheme == 'https' if url.scheme else bool(os.getenv("S3_USE_HTTPS", "true"))
        return Minio(url.netloc,...
```

The `netloc` will *always* be None _unless_ you happen to set the environment variable `AWS_ENDPOINT_URL`. This is because the default value is bad (missing the scheme), so the "s3.amazonaws.com" gets interpreted as the *path* instead of the *netloc*

```
>>> import os
>>> from urllib.parse import urlparse
>>> url = urlparse(os.getenv("AWS_ENDPOINT_URL", "s3.amazonaws.com"))
>>> print(url)
ParseResult(scheme='', netloc='', path='s3.amazonaws.com', params='', query='', fragment='')
```

**Environment:**

- Istio Version:
- Knative Version:
- KFServing Version:
- Kubeflow version:
- Minikube version:
- Kubernetes version: (use `kubectl version`): v1.14.9-eks-c0eccc
- OS (e.g. from `/etc/os-release`):



Discussion:
As discussed in slack, this was fixed in https://github.com/kubeflow/kfserving/pull/589
@axsaucedo Looking at the changes in that PR it doesn't look like it was fixed... The code in the #589 is identical to the broken code referenced in this issue. I've run into exactly the same thing. The default is also misleading as it suggests a scheme is not required.
@Jude188 you are right, I hadn't realised what you meant. I'll submit a fix.
Issue Label Bot is not confident enough to auto-label this issue.
                See [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) for more details.
                

Resolving patch:
diff --git a/python/kfserving/kfserving/storage.py b/python/kfserving/kfserving/storage.py
--- a/python/kfserving/kfserving/storage.py
+++ b/python/kfserving/kfserving/storage.py
@@ -208,8 +208,8 @@ def _download_local(uri, out_dir=None):
 
     @staticmethod
     def _create_minio_client():
-        # Remove possible http scheme for Minio
-        url = urlparse(os.getenv("AWS_ENDPOINT_URL", "s3.amazonaws.com"))
+        # Adding prefixing "http" in urlparse is necessary for it to be the netloc
+        url = urlparse(os.getenv("AWS_ENDPOINT_URL", "http://s3.amazonaws.com"))
         use_ssl = url.scheme == 'https' if url.scheme else bool(os.getenv("S3_USE_HTTPS", "true"))
         return Minio(url.netloc,
                      access_key=os.getenv("AWS_ACCESS_KEY_ID", ""),


Test patch: