Repository: kserve/kserve

Issue:
On prem -cannot create inferenceservice using pvc://
/kind bug

**What steps did you take and what happened:**
Setup Kubeflow 1.3 on Kubernetes 1.20.5
Kserve 0.6 

Following this tutorial - https://github.com/kserve/kserve/tree/master/docs/samples/storage/pvc

1. created a PVC with name inf-pvc in namespace admin

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: inf-pvc
  namespace: admin
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: main
  volumeMode: Filesystem 

2. create an inferenceservice. 
[root@machine-003 kubeflow]# cat mnist-pvc.yaml 
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "inf-sample"
  namespace: "admin"
spec:
  predictor:
    sklearn:
      storageUri: "pvc://inf-pvc/export"


k describe po  inf-sample-predictor-default-00001-deployment-86878b884f-fgwrf  -n admin
[I 211211 06:04:10 initializer-entrypoint:13] Initializing, args: src_uri [/mnt/pvc/export] dest_path[ [/mnt/models]
[I 211211 06:04:10 storage:52] Copying contents of /mnt/pvc/export to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kserve.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 83, in download
    (_GCS_PREFIX, _S3_PREFIX, _LOCAL_PREFIX, _HTTP_PREFIX))
Exception: Cannot recognize storage type for /mnt/pvc/export
'gs://', 's3://', 'file://', and 'http(s)://' are the current available storage type.

Does pvc storage not work? 
https://github.com/kserve/kserve/blob/master/python/kserve/kserve/storage.py doesnt seem to allow pvc://? 


**What did you expect to happen:**
Should not see this error. 

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**
- Knative Version: v0.22.0
- KFServing Version: 0.6
- Kubeflow version: 1.6
- Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube version: None. Using kubernetes, 
- Kubernetes version: (use `kubectl version`): 1.20.5
- OS (e.g. from `/etc/os-release`):
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"




Discussion:
@pshegde Please check https://kserve.github.io/website/modelserving/storage/pvc/pvc, this error usually indicates that kserve is not able to locate the model, error message is a bit misleading..

Resolving patch:
diff --git a/python/kserve/kserve/storage.py b/python/kserve/kserve/storage.py
--- a/python/kserve/kserve/storage.py
+++ b/python/kserve/kserve/storage.py
@@ -44,6 +44,7 @@
 _URI_RE = "https?://(.+)/(.+)"
 _HTTP_PREFIX = "http(s)://"
 _HEADERS_SUFFIX = "-headers"
+_PVC_PREFIX = "/mnt/pvc"
 
 
 class Storage(object):  # pylint: disable=too-few-public-methods
@@ -51,6 +52,9 @@ class Storage(object):  # pylint: disable=too-few-public-methods
     def download(uri: str, out_dir: str = None) -> str:
         logging.info("Copying contents of %s to local", uri)
 
+        if uri.startswith(_PVC_PREFIX) and not os.path.exists(uri):
+            raise Exception(f"Cannot locate source uri {uri} for PVC")
+
         is_local = False
         if uri.startswith(_LOCAL_PREFIX) or os.path.exists(uri):
             is_local = True


Test patch: