Repository: kserve/kserve

Issue:
Azure Blob Upgrade Bug Fix
<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://www.kubeflow.org/docs/about/contributing/ and developer guide https://github.com/kubeflow/kfserving/blob/master/docs/DEVELOPER_GUIDE.md
2. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
3. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
4. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What this PR does / why we need it**:

This PR addresses a bug which was introduced when we upgraded to Azure Blob 12.1.X or greater. Previously we thought that the container_client will filter blobs with a prefix but that's not the case. Per the [documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-list?tabs=python#use-a-hierarchical-listing) you need to implement your own logic. 

This PR introduces the use of a stack to iterate over the blob directories trees, and ensures blobs are saved without the prefix so it can work with inference services which expect certain artifacts to be in a standard format.

Typically you'll see storageUris which are structured like these: 
```
storageUri: https://<account>.blob.core.windows.net/private/models/iris-example/
storageUri: https://<account>.blob.core.windows.net/private/models/ner-tensorflow/1/
```

The prefix will be anything after the container name, the container name in this case is ```private```. Furthermore the artifacts would need to be written to the ```/mnt/models/``` directory without the prefix to ensure that the predictors can read the artifacts in the expected format.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #923 

**Special notes for your reviewer**:

This should be addressed and merged prior the next release.

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

**Release note**:
<!--  Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note

```

azure-storage-blob package update to 12.x
/kind feature

**Update to the current azure blob service to 12.x**
Update the current package from the legacy 2.1 version to the current 12.x series.

Hi, what would need to be done to do the upgrade? Is it only the files in the python kfserving package?


Discussion:
Hi @maganaluis. Thanks for your PR.

I'm waiting for a [kubeflow](https://github.com/orgs/kubeflow/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/kubeflow/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubeflow%2Fkfserving).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

[APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeflow/kfserving/pull/1750#" title="Author self-approved">maganaluis</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **yuzisun** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @yuzisun` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubeflow%2Fkfserving).

<details open>
Needs approval from an approver in each of these files:

- **[python/kfserving/OWNERS](https://github.com/kubeflow/kfserving/blob/master/python/kfserving/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["yuzisun"]} -->
Issue Label Bot is not confident enough to auto-label this issue.
                See [dashboard](https://label-bot-prod.mlbot.net/data/kubeflow/kfserving) for more details.
                
/cc @rakelkar 
@nyejon I think all we need to upgrade is here https://github.com/kubeflow/kfserving/blob/87b45e037ca279a7791e0b2a7da6d38898385a2b/python/kfserving/requirements.txt#L14, would you like to give a try and test out ?
We are using a custom SDK which requires Azure Storage Blob ```12.1.0``` within a custom Transformer, I don't see a workaround. Is there a reason why not to upgrade it? Otherwise we can create a PR? 
@maganaluis yes please!
@yuzisun I was waiting for approval from my org, I initiated PR https://github.com/kubeflow/kfserving/pull/1592

Resolving patch:
diff --git a/python/kserve/kserve/storage.py b/python/kserve/kserve/storage.py
--- a/python/kserve/kserve/storage.py
+++ b/python/kserve/kserve/storage.py
@@ -26,6 +26,7 @@
 import requests
 from pathlib import Path
 from azure.storage.blob import BlobServiceClient
+from azure.storage.blob._list_blobs_helper import BlobPrefix
 
 from botocore.client import Config
 from botocore import UNSIGNED
@@ -176,9 +177,22 @@ def _download_blob(uri, out_dir: str):  # pylint: disable=too-many-locals
         blob_service_client = BlobServiceClient(account_url, credential=token)
         container_client = blob_service_client.get_container_client(container_name)
         count = 0
-        blobs = container_client.list_blobs(prefix=prefix)
-        for blob in blobs:
-            dest_path = os.path.join(out_dir, blob.name)
+        blobs = []
+        max_depth = 5
+        stack = [(prefix, max_depth)]
+        while stack:
+            curr_prefix, depth = stack.pop()
+            if depth < 0:
+                continue
+            for item in container_client.walk_blobs(
+                            name_starts_with=curr_prefix):
+                if isinstance(item, BlobPrefix):
+                    stack.append((item.name, depth - 1))
+                else:
+                    blobs += container_client.list_blobs(name_starts_with=item.name,
+                                                         include=['snapshots'])
+        for blob in set(blobs):
+            dest_path = os.path.join(out_dir, blob.name.replace(prefix, "", 1).lstrip("/"))
             Path(os.path.dirname(dest_path)).mkdir(parents=True, exist_ok=True)
             logging.info("Downloading: %s to %s", blob.name, dest_path)
             downloader = container_client.download_blob(blob.name)


Test patch:
diff --git a/python/kserve/test/test_azure_storage.py b/python/kserve/test/test_azure_storage.py
--- a/python/kserve/test/test_azure_storage.py
+++ b/python/kserve/test/test_azure_storage.py
@@ -27,6 +27,7 @@ def create_mock_item(path):
 def create_mock_blob(mock_storage, paths):
     mock_objs = [create_mock_item(path) for path in paths]
     mock_container = mock.MagicMock()
+    mock_container.walk_blobs.return_value = mock_objs
     mock_container.list_blobs.return_value = mock_objs
     mock_container.download_blob.return_value = mock_objs[0]
     mock_svc = mock.MagicMock()
@@ -66,8 +67,8 @@ def test_blob(mock_storage, mock_makedirs):  # pylint: disable=unused-argument
 
     # then
     arg_list = get_call_args(mock_container.download_blob.call_args_list)
-    assert arg_list == [('simple_string/1/model.graphdef',),
-                        ('simple_string/config.pbtxt',)]
+    assert set(arg_list) == set([('simple_string/1/model.graphdef',),
+                                 ('simple_string/config.pbtxt',)])
 
     mock_storage.assert_called_with('https://kfserving.blob.core.windows.net',
                                     credential=None)
@@ -107,11 +108,14 @@ def test_deep_blob(mock_storage, mock_makedirs):  # pylint: disable=unused-argum
 
     # when
     mock_blob, mock_container = create_mock_blob(mock_storage, fq_item_paths)
-    kserve.Storage._download_blob(blob_path, "some/dest/path")
+    try:
+        kserve.Storage._download_blob(blob_path, "some/dest/path")
+    except OSError:  # Permissions Error Handling
+        pass
 
     # then
     actual_calls = get_call_args(mock_container.download_blob.call_args_list)
-    assert actual_calls == expected_calls
+    assert set(actual_calls) == set(expected_calls)
 
 
 @mock.patch('kserve.storage.os.makedirs')
@@ -119,7 +123,7 @@ def test_deep_blob(mock_storage, mock_makedirs):  # pylint: disable=unused-argum
 def test_blob_file(mock_storage, mock_makedirs):  # pylint: disable=unused-argument
 
     # given
-    blob_path = 'https://accountname.blob.core.windows.net/container/somefile'
+    blob_path = 'https://accountname.blob.core.windows.net/container/somefile.text'
     paths = ['somefile']
     fq_item_paths = paths
     expected_calls = [(f,) for f in fq_item_paths]
@@ -138,7 +142,7 @@ def test_blob_file(mock_storage, mock_makedirs):  # pylint: disable=unused-argum
 def test_blob_fq_file(mock_storage, mock_makedirs):  # pylint: disable=unused-argument
 
     # given
-    blob_path = 'https://accountname.blob.core.windows.net/container/folder/somefile'
+    blob_path = 'https://accountname.blob.core.windows.net/container/folder/somefile.text'
     paths = ['somefile']
     fq_item_paths = ['folder/' + p for p in paths]
     expected_calls = [(f,) for f in fq_item_paths]
@@ -158,7 +162,7 @@ def test_blob_no_prefix(mock_storage, mock_makedirs):  # pylint: disable=unused-
 
     # given
     blob_path = 'https://accountname.blob.core.windows.net/container/'
-    paths = ['somefile', 'somefolder/somefile']
+    paths = ['somefile.text', 'somefolder/somefile.text']
     fq_item_paths = ['' + p for p in paths]
     expected_calls = [(f,) for f in fq_item_paths]
 
@@ -168,4 +172,4 @@ def test_blob_no_prefix(mock_storage, mock_makedirs):  # pylint: disable=unused-
 
     # then
     actual_calls = get_call_args(mock_container.download_blob.call_args_list)
-    assert actual_calls == expected_calls
+    assert set(actual_calls) == set(expected_calls)