Repository: kserve/kserve

Issue:
Support ali-cloud oss in storage-uri 
/kind feature

**Describe the solution you'd like**
There is already a closed issue #2128  requesting support for Aliyun OSS links for storage-uri. The reason for closing this issue was that "OSS is compatible with the S3 API". But it's not true. Storage-initializer will throw a error msg "Please use virtual hosted style to access" if use OSS links as S3 links.
![image](https://user-images.githubusercontent.com/32983350/236821407-be0960eb-1359-4476-85ab-8c8fe1e906f9.png)

This may be due to AliCloud OSS not supporting path-style URLs. As it's mentioned in [Compatibility with Amazon S3](https://www.alibabacloud.com/help/en/object-storage-service/latest/s3-compatibility-overview#concept-2158780), "For  security reasons, OSS supports only virtual hosted-style URLs in requests". But  S3 support both path-style and virtual-hosted–style URLs. 

Is it possible to add support for AliCloud OSS URLs?

**Anything else you would like to add:**
It seems KServe provide an environment variable S3_USER_VIRTUAL_BUCKET to specify URL style，but not used in storage.py. This may be a work-around if S3_USER_VIRTUAL_BUCKET can work.
![image](https://user-images.githubusercontent.com/32983350/236825380-01d05292-d23a-417c-90a8-2a9fb516aa82.png)
![image](https://user-images.githubusercontent.com/32983350/236824924-9bcb9694-36e0-4e50-9c71-15f37afd01da.png)




Discussion:
I think you specify the annotation `serving.kserve.io/s3-usevirtualbucket` on the s3 secret then the env environment should be injected to the storage initializer container
https://github.com/kserve/kserve/blob/ab71c1c3767c9d1e81e873702e5e46f29a57c65c/pkg/credentials/s3/utils.go#L98
I do add this annotation and  the env environment is injected to storage initializer container, but it does't work. Is there other env variables used except thoes in 
https://github.com/kserve/kserve/blob/ab71c1c3767c9d1e81e873702e5e46f29a57c65c/python/kserve/kserve/storage/storage.py#L114-L120
Maybe this env variable is not used to create s3 client?

I'm using KServe v0.10.0,  here is the error msg:
```
kubectl logs kserve-bloom-predictor-00001-deployment-67f67995fd-755nq -c storage-initializer

2023-05-09 07:37:09.618 1 root INFO [<module>():13] Initializing, args: src_uri [s3://asm-fluid-demo/models/bloom-560m] dest_path[ [/mnt/models]
2023-05-09 07:37:09.619 1 root INFO [download():63] Copying contents of s3://asm-fluid-demo/models/bloom-560m to local
2023-05-09 07:37:09.647 1 botocore.credentials INFO [load():1120] Found credentials in environment variables.
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kserve.Storage.download(src_uri, dest_path)
  File "/kserve/kserve/storage.py", line 83, in download
    Storage._download_s3(uri, out_dir)
  File "/kserve/kserve/storage.py", line 167, in _download_s3
    for obj in bucket.objects.filter(Prefix=bucket_path):
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/collection.py", line 81, in __iter__
    for page in self.pages():
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/collection.py", line 171, in pages
    for page in pages:
  File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SecondLevelDomainForbidden) when calling the ListObjects operation: Please use virtual hosted style to access.
```
Here is the InferreferenceService:
```yaml
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "kserve-bloom"
spec:
  predictor:
    timeout: 600
    minReplicas: 0
    serviceAccountName: s3-sa
    containers:
      - name: kserve-container
        image: cheyang/kserve-fluid:bloom-gpu
        resources:
          limits:
            cpu: "12"
            memory: 48Gi
          requests:
            cpu: "12"
            memory: 48Gi
        env:
          - name: STORAGE_URI
            # please update it accordingly
            value: "s3://asm-fluid-demo/models/bloom-560m"  # I simply replace "oss" as "s3" for oss link "oss://asm-fluid-demo/models/bloom-560m
            # value: "s3://${bucket}/models/bloom-7b1"
          - name: MODEL_NAME
            value: "bloom"
          - name: GPU_ENABLED
            value: "True"
```
Here is Secret for s3:
```yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: oss-s3creds
  annotations:
    serving.kserve.io/s3-endpoint: oss-cn-hangzhou-internal.aliyuncs.com # replace with your s3 endpoint e.g minio-service.kubeflow:9000
    serving.kserve.io/s3-usehttps: "1" # by default 1, if testing with minio you can set to 0
    serving.kserve.io/s3-region: "oss-cn-hangzhou"
    serving.kserve.io/s3-useanoncredential: "false" # omitting this is the same as false, if true will ignore provided credential and use anonymous credentials
    serving.kserve.io/s3-usevirtualbucket: "true"
type: Opaque
stringData: # use `stringData` for raw credential string or `data` for base64 encoded string
  AWS_ACCESS_KEY_ID: xxx
  AWS_SECRET_ACCESS_KEY: xxx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-sa
secrets:
- name: oss-s3creds
```


And I notice that error occur when calling the ListObjects operation, but according to [Compatibility with Amazon S3](https://www.alibabacloud.com/help/en/object-storage-service/latest/s3-compatibility-overview#concept-2158780), ListObject is not in Compatible S3 API operations List.
![image](https://user-images.githubusercontent.com/32983350/237039373-b1b9d82e-f70b-4ad7-9a54-4e0833dff4ee.png)

Hi @hclchimumu and @yuzisun , theoretically adding `serving.kserve.io/s3-usevirtualbucket: "true"` should work to enable oss via s3 SDK, however,  the current [implementation](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/storage/storage.py#L138-L144) of storage initializer didn't check this env variable (`S3_USER_VIRTUAL_BUCKET`) and set `{"addressing_style": "virtual"}` in the config.  We have fixed and tested it in our internal version of the storage initializer. Maybe I will create a PR to fix it.

Whether to have a dedicated `oss://` like https://github.com/kserve/kserve/pull/2128 (a PR raised by my colleague), that is another topic.


Resolving patch:
diff --git a/python/kserve/kserve/storage/storage.py b/python/kserve/kserve/storage/storage.py
--- a/python/kserve/kserve/storage/storage.py
+++ b/python/kserve/kserve/storage/storage.py
@@ -136,12 +136,21 @@ def _update_with_storage_spec():
 
     @staticmethod
     def get_S3_config():
+        # default s3 config
+        c = Config()
+
         # anon environment variable defined in s3_secret.go
-        anon = ("True" == os.getenv("awsAnonymousCredential", "false").capitalize())
+        anon = ("true" == os.getenv("awsAnonymousCredential", "false").lower())
+        # S3UseVirtualBucket environment variable defined in s3_secret.go
+        # use virtual hosted-style URLs if enabled
+        virtual = ("true" == os.getenv("S3_USER_VIRTUAL_BUCKET", "false").lower())
+
         if anon:
-            return Config(signature_version=UNSIGNED)
-        else:
-            return None
+            c = c.merge(Config(signature_version=UNSIGNED))
+        if virtual:
+            c = c.merge(Config(s3={"addressing_style": "virtual"}))
+
+        return c
 
     @staticmethod
     def _download_s3(uri, temp_dir: str):


Test patch:
diff --git a/python/kserve/kserve/storage/test/test_s3_storage.py b/python/kserve/kserve/storage/test/test_s3_storage.py
--- a/python/kserve/kserve/storage/test/test_s3_storage.py
+++ b/python/kserve/kserve/storage/test/test_s3_storage.py
@@ -141,20 +141,21 @@ def test_full_name_key_root_bucket_dir(mock_storage):
 
 
 def test_get_S3_config():
+    DEFAULT_CONFIG = Config()
     ANON_CONFIG = Config(signature_version=UNSIGNED)
-    DEFAULT_CONFIG = None
+    VIRTUAL_CONFIG = Config(s3={"addressing_style": "virtual"})
 
     with mock.patch.dict(os.environ, {}):
         config1 = Storage.get_S3_config()
-    assert config1 == DEFAULT_CONFIG
+    assert vars(config1) == vars(DEFAULT_CONFIG)
 
     with mock.patch.dict(os.environ, {"awsAnonymousCredential": "False"}):
         config2 = Storage.get_S3_config()
-    assert config2 == DEFAULT_CONFIG
+    assert vars(config2) == vars(DEFAULT_CONFIG)
 
     with mock.patch.dict(os.environ, AWS_TEST_CREDENTIALS):
         config3 = Storage.get_S3_config()
-    assert config3 == DEFAULT_CONFIG
+    assert vars(config3) == vars(DEFAULT_CONFIG)
 
     with mock.patch.dict(os.environ, {"awsAnonymousCredential": "True"}):
         config4 = Storage.get_S3_config()
@@ -165,3 +166,11 @@ def test_get_S3_config():
     with mock.patch.dict(os.environ, credentials_and_anon):
         config5 = Storage.get_S3_config()
     assert config5.signature_version == ANON_CONFIG.signature_version
+
+    with mock.patch.dict(os.environ, {"S3_USER_VIRTUAL_BUCKET": "False"}):
+        config6 = Storage.get_S3_config()
+    assert vars(config6) == vars(DEFAULT_CONFIG)
+
+    with mock.patch.dict(os.environ, {"S3_USER_VIRTUAL_BUCKET": "True"}):
+        config7 = Storage.get_S3_config()
+    assert config7.s3["addressing_style"] == VIRTUAL_CONFIG.s3["addressing_style"]