Repository: kserve/kserve

Issue:
Storage initializer v0.11.1 downloads model.pkl to /mnt/models/model/model.pkl instead of  /mnt/models/model.pkl 
/kind bug

**What steps did you take and what happened:**

We recently upgraded our KServe instance from v0.9.0 to newest v0.11.1 and encountered weird behaviour for storage initializer. We mainly serve models from mlflow registry. When trying to serve the wine example model from our mlflow registry the kserve-container crashes, because it cannot find model.pkl. The artifact in mlflow looks like this:
```
├── conda.yaml
├── MLmodel
├── model.pkl
├── python_env.yaml
└── requirements.txt
```

Storage initalizer v0.11.1 downloads all files to /mnt/models but downloading model.pkl to subdirectory model /mnt/models/model/model.pkl. So the volume after storage initalizer is done looks like:

```
├── conda.yaml
├── MLmodel
├── model
│     ├── model.pkl
├── python_env.yaml
└── requirements.txt
```
![image](https://github.com/kserve/kserve/assets/22219924/fbdceb6d-fde4-4516-94b1-01a914be786b)

This causes kserve-container to CrashLoopBackOff as it cannot find model.pkl where it is expected. 

Using the storage initializer v0.9.0 the artifact download is done the way kserve expects it. We therefore currently run the setup of kserve v0.11.1 using storage initalizer v0.9.0

**What did you expect to happen:**
storage initializer should download the model.pkl to the same directory as MLmodel

**What's the InferenceService yaml:**
A very basic InferenceService yaml
```
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: wine
  namespace: default
spec:
  predictor:
    model:
      modelFormat:
        name: mlflow
      name: ""
      protocolVersion: v2
      storageUri: s3://mlflow/4/1c8e8fe765c34c45baeba5030cb96753/artifacts/model
    serviceAccountName: sa-s3
```

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version: 1.16
- Knative Version: 1.4
- KServe Version: 0.11.1
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: 1.22.9
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/kserve/kserve/storage/storage.py b/python/kserve/kserve/storage/storage.py
--- a/python/kserve/kserve/storage/storage.py
+++ b/python/kserve/kserve/storage/storage.py
@@ -224,13 +224,12 @@ def _download_s3(uri, temp_dir: str):
             # Objects: churn, churn-pickle, churn-pickle-logs
             bucket_path_last_part = bucket_path.split("/")[-1]
             object_last_path = obj.key.split("/")[-1]
-            bucket_path_parent_part = bucket_path.rsplit("/", 1)[0]
 
             if bucket_path == obj.key:
                 target_key = obj.key.rsplit("/", 1)[-1]
                 exact_obj_found = True
-            elif object_last_path.startswith(bucket_path_last_part):
-                target_key = obj.key.replace(bucket_path_parent_part, "", 1).lstrip("/")
+            elif bucket_path_last_part and object_last_path.startswith(bucket_path_last_part):
+                target_key = object_last_path
             else:
                 target_key = obj.key.replace(bucket_path, "").lstrip("/")
 


Test patch:
diff --git a/python/kserve/kserve/storage/test/test_s3_storage.py b/python/kserve/kserve/storage/test/test_s3_storage.py
--- a/python/kserve/kserve/storage/test/test_s3_storage.py
+++ b/python/kserve/kserve/storage/test/test_s3_storage.py
@@ -260,3 +260,22 @@ def test_update_with_storage_spec_s3(monkeypatch):
 
     # revert changes
     os.environ = previous_env
+
+
+@mock.patch(STORAGE_MODULE + '.boto3')
+def test_target_startswith_parent_folder_name(mock_storage):
+    bucket_name = 'foo'
+    paths = ["model.pkl", "a/model.pkl", "conda.yaml"]
+    object_paths = ['test/artifacts/model/' + p for p in paths]
+
+    # when
+    mock_boto3_bucket = create_mock_boto3_bucket(mock_storage, object_paths)
+    Storage._download_s3(
+        f's3://{bucket_name}/test/artifacts/model', 'dest_path')
+
+    # then
+    arg_list = get_call_args(mock_boto3_bucket.download_file.call_args_list)
+    assert arg_list[0] == expected_call_args_list(
+        'test/artifacts/model', 'dest_path', paths)[0]
+    mock_boto3_bucket.objects.filter.assert_called_with(
+        Prefix='test/artifacts/model')