Repository: kserve/kserve

Issue:
Extract zip/archive files in GCS download
/kind bug

**What steps did you take and what happened:**

Created InferenceService with `storageUri` as a zip archive, e.g. `gs://bucket/folder/archive.zip` - no model files found.

Created InferenceService with `storageUri` as a folder containing a single archive, e.g. `gs://bucket/folder` - archive found and downloaded, but not extracted.

Used `kserve.storage.storage.Storage._download_gcs` directly, with either zip archive or gcs folder containing only a single archive - either downloads nothing, or downloads the zip but doesn't extract it.


**What did you expect to happen:**

I expected the zip to be downloaded and extracted.

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]

**Anything else you would like to add:**

https://github.com/kserve/kserve/pull/1836/files added code which should do this, but there are a couple of problems:

1. The uri is always used as a prefix, with a [trailing slash added](https://github.com/kserve/kserve/blob/cac5fb1407391eedee284009d31c5a45ef127b23/python/kserve/kserve/storage/storage.py#L250) - so specifying the uri as `gs://bucket/folder/archive.zip` results in any blobs with the prefix `folder/archive.zip/` being downloaded - there are none.
2. The folder itself is counted as a blob (I'm not sure if this is specific to the google cloud storage version I'm using - I created a fresh poetry project and just installed kserve (^0.11.2), and it chose google-cloud-storage 1.44.0), so the `file_count` is incremented for the folder as well which results in `file_count` of 2, so it [doesn't extract the archive](https://github.com/kserve/kserve/blob/cac5fb1407391eedee284009d31c5a45ef127b23/python/kserve/kserve/storage/storage.py#L272). I'm not sure including the folder in the count is the desired behaviour here - possibly this is an indentation issue?


**Environment:**

- Istio Version: 
- Knative Version: 
- KServe Version: 0.11.2
- Kubeflow version:
- Cloud Environment: 
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`): Debian GNU/Linux 10 (buster)



Discussion:
Similar issue on ubuntu 20.04 with charmed kubeflow(1.8) and kserve(0.11.0). Any insights?
I think if only files which are actually downloaded (e.g. not 'folders') are counted, it would fix this and not break anything else, so change (from [here](https://github.com/kserve/kserve/blob/cac5fb1407391eedee284009d31c5a45ef127b23/python/kserve/kserve/storage/storage.py#L262-L266)):
```python
if subdir_object_key.strip() != "" and not subdir_object_key.endswith("/"):
    dest_path = os.path.join(temp_dir, subdir_object_key)
    logging.info("Downloading: %s", dest_path)
    blob.download_to_filename(dest_path)
file_count += 1
```
to
```python
if subdir_object_key.strip() != "" and not subdir_object_key.endswith("/"):
    dest_path = os.path.join(temp_dir, subdir_object_key)
    logging.info("Downloading: %s", dest_path)
    blob.download_to_filename(dest_path)
    file_count += 1
```

As far as I can see, that would be safe.
Hi @gfkeith , feel free to create a PR to fix it, you can refer to [S3](https://github.com/kserve/kserve/blob/cac5fb1407391eedee284009d31c5a45ef127b23/python/kserve/kserve/storage/storage.py#L161) as well.

so far from y understanding the behaviour in KServe is `gs://bucket/folder/archive.zip` will be downloaded and unzipped (single zip file). 


Resolving patch:
diff --git a/python/kserve/kserve/storage/storage.py b/python/kserve/kserve/storage/storage.py
--- a/python/kserve/kserve/storage/storage.py
+++ b/python/kserve/kserve/storage/storage.py
@@ -302,7 +302,7 @@ def _download_gcs(uri, temp_dir: str):
                 dest_path = os.path.join(temp_dir, subdir_object_key)
                 logging.info("Downloading: %s", dest_path)
                 blob.download_to_filename(dest_path)
-            file_count += 1
+                file_count += 1
         if file_count == 0:
             raise RuntimeError("Failed to fetch model. No model found in %s." % uri)
 


Test patch:
diff --git a/python/kserve/kserve/storage/test/test_gcs_storage.py b/python/kserve/kserve/storage/test/test_gcs_storage.py
new file mode 100644
--- /dev/null
+++ b/python/kserve/kserve/storage/test/test_gcs_storage.py
@@ -0,0 +1,117 @@
+# Copyright 2024 The KServe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest.mock as mock
+import pytest
+from kserve.storage import Storage
+
+STORAGE_MODULE = "kserve.storage.storage"
+
+
+def get_call_args(call_args_list):
+    arg_list = []
+    for call in call_args_list:
+        args, _ = call
+        arg_list.append(args)
+    return arg_list
+
+
+def create_mock_dir(name):
+    mock_dir = mock.MagicMock()
+    mock_dir.name = name
+    return mock_dir
+
+
+def create_mock_dir_with_file(dir_name, file_name):
+    mock_obj = mock.MagicMock()
+    mock_obj.name = f"{dir_name}/{file_name}"
+    return mock_obj
+
+
+@mock.patch(STORAGE_MODULE + ".storage")
+def test_gcs_with_empty_dir(mock_storage):
+    gcs_path = "gs://foo/bar"
+
+    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [
+        create_mock_dir("bar/")
+    ]
+
+    with pytest.raises(Exception):
+        Storage.download(gcs_path)
+
+
+@mock.patch(STORAGE_MODULE + ".storage")
+def test_gcs_with_nested_sub_dir(mock_storage):
+    gcs_path = "gs://foo/bar/test"
+
+    mock_root_dir = create_mock_dir("bar/")
+    mock_sub_dir = create_mock_dir("test/")
+    mock_file = create_mock_dir_with_file("test", "mock.object")
+
+    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [
+        mock_root_dir,
+        mock_sub_dir,
+        mock_file,
+    ]
+    Storage.download(gcs_path)
+
+    arg_list = get_call_args(mock_file.download_to_filename.call_args_list)
+    assert "test/mock.object" in arg_list[0][0]
+
+
+@mock.patch(STORAGE_MODULE + ".storage")
+def test_download_model_from_gcs(mock_storage):
+    gcs_path = "gs://foo/bar"
+
+    mock_dir = create_mock_dir("bar/")
+    mock_file = create_mock_dir_with_file("bar", "mock.object")
+
+    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [
+        mock_dir,
+        mock_file,
+    ]
+    Storage.download(gcs_path)
+
+    arg_list = get_call_args(mock_file.download_to_filename.call_args_list)
+    assert "/mock.object" in arg_list[0][0]
+
+
+@mock.patch("os.remove")
+@mock.patch("os.mkdir")
+@mock.patch("zipfile.ZipFile")
+@mock.patch(STORAGE_MODULE + ".storage")
+def test_gcs_model_unpack_archive_file(
+    mock_storage, MockZipFile, mock_create, mock_remove
+):
+    gcs_path = "gs://foo/bar"
+    output_dir = "test/out_dir"
+
+    mock_dir = create_mock_dir("bar/")
+    mock_file = create_mock_dir_with_file("bar", "mock.zip")
+    MockZipFile.return_value = mock_file
+
+    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [
+        mock_dir,
+        mock_file,
+    ]
+    Storage.download(gcs_path, output_dir)
+
+    download_arg_list = get_call_args(mock_file.download_to_filename.call_args_list)
+
+    extract_arg_list = get_call_args(mock_file.extractall.call_args_list)
+
+    assert "/mock.zip" in download_arg_list[0][0]
+    assert output_dir == extract_arg_list[0][0]
+    assert mock_file.close.called
+    assert mock_remove.called
diff --git a/python/kserve/kserve/storage/test/test_storage.py b/python/kserve/kserve/storage/test/test_storage.py
--- a/python/kserve/kserve/storage/test/test_storage.py
+++ b/python/kserve/kserve/storage/test/test_storage.py
@@ -227,8 +227,13 @@ def test(_):
 def test_mock_gcs(mock_storage):
     gcs_path = "gs://foo/bar"
     mock_obj = mock.MagicMock()
-    mock_obj.name = "mock.object"
-    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [mock_obj]
+    mock_obj.name = "bar/"
+    mock_obj1 = mock.MagicMock()
+    mock_obj1.name = "bar/mock.object"
+    mock_storage.Client().bucket().list_blobs().__iter__.return_value = [
+        mock_obj,
+        mock_obj1,
+    ]
     assert Storage.download(gcs_path)