Repository: kserve/kserve

Issue:
SDK should allow credentials to be set in its own namespace.
/kind bug

**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]
When we are trying to run `KFServing.set_credentials` on the KubeFlow Jupyter server. We encountered the below errors. It looks like the SDK try to find the control-plane config `inferenceservice-config` and check the configmap status. However, the KubeFlow Jupyter server doesn't allow RBAC permission outside of its namespace, so we should come up a solution that's not depended on where the control-plane is deployed.

Here are the logs when I try to set_credentials it under my own namespace.
```python
constants.INFERENCESERVICE_SYSTEM_NAMESPACE = namespace

def InferenceService(metadata, default_model_spec, canary_model_spec=None, canary_model_traffic=None):
    return V1alpha2InferenceService(api_version=constants.KFSERVING_GROUP + '/' + constants.KFSERVING_VERSION,
                             kind=constants.KFSERVING_KIND,
                             metadata=metadata,
                             spec=V1alpha2InferenceServiceSpec(default=default_model_spec,
                                                               canary=canary_model_spec,
                                                               canary_traffic_percent=canary_model_traffic))
KFServing = KFServingClient()
metadata = client.V1ObjectMeta(name=deploy_name, namespace=namespace)
default_model_spec = V1alpha2EndpointSpec(predictor=V1alpha2PredictorSpec(tensorflow=V1alpha2TensorflowSpec(storage_uri=model_base_path)))
kfsvc = InferenceService(metadata, default_model_spec)

KFServing.set_credentials(storage_type='S3',
                          namespace=namespace,
                          credentials_file='./coscreds.ini',
                          s3_endpoint=service_endpoint,
                          s3_region=cos_region,
                          s3_use_https='1',
                          s3_verify_ssl='1')
KFServing.create(kfsvc, watch=True, timeout_seconds=120)

---------------------------------------------------------------------------
ApiException                              Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in get_creds_name_from_config_map(creds)
    232             constants.INFERENCESERVICE_CONFIG_MAP_NAME,
--> 233             constants.INFERENCESERVICE_SYSTEM_NAMESPACE)
    234     except client.rest.ApiException as e:

~/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py in read_namespaced_config_map(self, name, namespace, **kwargs)
  17763         else:
> 17764             (data) = self.read_namespaced_config_map_with_http_info(name, namespace, **kwargs)
  17765             return data

~/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py in read_namespaced_config_map_with_http_info(self, name, namespace, **kwargs)
  17854                                         _request_timeout=params.get('_request_timeout'),
> 17855                                         collection_formats=collection_formats)
  17856 

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    333                                    response_type, auth_settings,
--> 334                                    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    335         else:

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in __call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    167                                      _preload_content=_preload_content,
--> 168                                      _request_timeout=_request_timeout)
    169 

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    354                                         _request_timeout=_request_timeout,
--> 355                                         headers=headers)
    356         elif method == "HEAD":

~/.local/lib/python3.6/site-packages/kubernetes/client/rest.py in GET(self, url, headers, query_params, _preload_content, _request_timeout)
    230                             _request_timeout=_request_timeout,
--> 231                             query_params=query_params)
    232 

~/.local/lib/python3.6/site-packages/kubernetes/client/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    221         if not 200 <= r.status <= 299:
--> 222             raise ApiException(http_resp=r)
    223 

ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Mon, 24 Feb 2020 19:43:11 GMT', 'Content-Length': '226'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"inferenceservice-config\" not found","reason":"NotFound","details":{"name":"inferenceservice-config","kind":"configmaps"},"code":404}



During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-37-b80412a092fc> in <module>
     30                           s3_region=cos_region,
     31                           s3_use_https='1',
---> 32                           s3_verify_ssl='1')
     33 KFServing.create(kfsvc, watch=True, timeout_seconds=120)

~/.local/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py in set_credentials(self, storage_type, namespace, credentials_file, service_account, **kwargs)
     74                                credentials_file=credentials_file,
     75                                service_account=service_account,
---> 76                                **kwargs)
     77         elif storage_type.lower() == 'azure':
     78             if credentials_file is None:

~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in set_s3_credentials(namespace, credentials_file, service_account, s3_profile, s3_endpoint, s3_region, s3_use_https, s3_verify_ssl)
     79     # Try to get S3 creds name from configmap, set default value then if cannot.
     80     s3_access_key_id_name = get_creds_name_from_config_map(
---> 81         's3AccessKeyIDName')
     82     if not s3_access_key_id_name:
     83         s3_access_key_id_name = constants.S3_ACCESS_KEY_ID_DEFAULT_NAME

~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in get_creds_name_from_config_map(creds)
    234     except client.rest.ApiException as e:
    235         raise RuntimeError(
--> 236             "Exception when calling CoreV1Api->read_namespaced_config_map: %s\n" % e)
    237 
    238     isvc_creds_str = isvc_config_map.data['credentials']

RuntimeError: Exception when calling CoreV1Api->read_namespaced_config_map: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Mon, 24 Feb 2020 19:43:11 GMT', 'Content-Length': '226'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"inferenceservice-config\" not found","reason":"NotFound","details":{"name":"inferenceservice-config","kind":"configmaps"},"code":404}
```

**What did you expect to happen:**
SDK should allow credentials to be set in its own namespace.

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version: 1.16
- Knative Version: 0.11
- KFServing Version: 0.2.2
- Kubeflow version: 1.0-rc
- Minikube version:
- Kubernetes version: (use `kubectl version`): 1.15
- OS (e.g. from `/etc/os-release`):



Discussion:
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| bug | 0.87 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
/cc @jinchihe 
@Tomcli I guess that may be caused by the KFServing is installed in `kubeflow` namespace, and in our constants, set the `INFERENCESERVICE_SYSTEM_NAMESPACE = 'kfserving-system'`, the configmap may be in `kubeflow` namespace. 

Could you please help tp check which namespace KFserving installed? and then check if have `inferenceservice-config` in that namespace?If this is root course, you can reset the constants to work around this by reset `constantss.INFERENCESERVICE_SYSTEM_NAMESPACE`.
So I'm able to change the `INFERENCESERVICE_SYSTEM_NAMESPACE`. However, the Jupyter notebook server won't let me access the configmap because we don't have any permission outside of our namespace. I think we need a new mechanism for the multi-users scenario where the s3/gcp credentials are created under the same namespace as the Jupyter notebook server.

```python
---------------------------------------------------------------------------
ApiException                              Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in get_creds_name_from_config_map(creds)
    232             constants.INFERENCESERVICE_CONFIG_MAP_NAME,
--> 233             constants.INFERENCESERVICE_SYSTEM_NAMESPACE)
    234     except client.rest.ApiException as e:

~/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py in read_namespaced_config_map(self, name, namespace, **kwargs)
  17763         else:
> 17764             (data) = self.read_namespaced_config_map_with_http_info(name, namespace, **kwargs)
  17765             return data

~/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py in read_namespaced_config_map_with_http_info(self, name, namespace, **kwargs)
  17854                                         _request_timeout=params.get('_request_timeout'),
> 17855                                         collection_formats=collection_formats)
  17856 

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    333                                    response_type, auth_settings,
--> 334                                    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    335         else:

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in __call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    167                                      _preload_content=_preload_content,
--> 168                                      _request_timeout=_request_timeout)
    169 

~/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py in request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    354                                         _request_timeout=_request_timeout,
--> 355                                         headers=headers)
    356         elif method == "HEAD":

~/.local/lib/python3.6/site-packages/kubernetes/client/rest.py in GET(self, url, headers, query_params, _preload_content, _request_timeout)
    230                             _request_timeout=_request_timeout,
--> 231                             query_params=query_params)
    232 

~/.local/lib/python3.6/site-packages/kubernetes/client/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    221         if not 200 <= r.status <= 299:
--> 222             raise ApiException(http_resp=r)
    223 

ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 27 Feb 2020 01:03:01 GMT', 'Content-Length': '370'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"inferenceservice-config\" is forbidden: User \"system:serviceaccount:anonymous:default-editor\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow\"","reason":"Forbidden","details":{"name":"inferenceservice-config","kind":"configmaps"},"code":403}



During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-28-2d8c49c13267> in <module>
     79                           s3_region=cos_region,
     80                           s3_use_https='1',
---> 81                           s3_verify_ssl='1')
     82 KFServing.create(kfsvc, watch=True, timeout_seconds=120)

~/.local/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py in set_credentials(self, storage_type, namespace, credentials_file, service_account, **kwargs)
     74                                credentials_file=credentials_file,
     75                                service_account=service_account,
---> 76                                **kwargs)
     77         elif storage_type.lower() == 'azure':
     78             if credentials_file is None:

~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in set_s3_credentials(namespace, credentials_file, service_account, s3_profile, s3_endpoint, s3_region, s3_use_https, s3_verify_ssl)
     79     # Try to get S3 creds name from configmap, set default value then if cannot.
     80     s3_access_key_id_name = get_creds_name_from_config_map(
---> 81         's3AccessKeyIDName')
     82     if not s3_access_key_id_name:
     83         s3_access_key_id_name = constants.S3_ACCESS_KEY_ID_DEFAULT_NAME

~/.local/lib/python3.6/site-packages/kfserving/api/creds_utils.py in get_creds_name_from_config_map(creds)
    234     except client.rest.ApiException as e:
    235         raise RuntimeError(
--> 236             "Exception when calling CoreV1Api->read_namespaced_config_map: %s\n" % e)
    237 
    238     isvc_creds_str = isvc_config_map.data['credentials']

RuntimeError: Exception when calling CoreV1Api->read_namespaced_config_map: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 27 Feb 2020 01:03:01 GMT', 'Content-Length': '370'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"inferenceservice-config\" is forbidden: User \"system:serviceaccount:anonymous:default-editor\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow\"","reason":"Forbidden","details":{"name":"inferenceservice-config","kind":"configmaps"},"code":403}
```
So I think we set this as `constants`, and allow user set env variables to update that? 
```
  credentials: |-
    {
       "gcs": {
           "gcsCredentialFileName": "gcloud-application-credentials.json"
       },
       "s3": {
           "s3AccessKeyIDName": "awsAccessKeyID",
           "s3SecretAccessKeyName": "awsSecretAccessKey"
       }
    }
```

comments? @yuzisun 
@Tomcli the secrets are created in user namespace, it just needs to access the configmap to get the secret data key names to create the secret. We could allow user to pass in this from the API.
I think we can try to get from  the `configmap`, if failed (no configmap or no credentials), then we will get from `constants` that allows user setting that.
Checked the code, we already defined that in the `constants `, we just need to enhance if cannot get `configmap`, and return None to get from `constants` then.

Resolving patch:
diff --git a/python/kfserving/kfserving/api/creds_utils.py b/python/kfserving/kfserving/api/creds_utils.py
--- a/python/kfserving/kfserving/api/creds_utils.py
+++ b/python/kfserving/kfserving/api/creds_utils.py
@@ -231,9 +231,11 @@ def get_creds_name_from_config_map(creds):
         isvc_config_map = client.CoreV1Api().read_namespaced_config_map(
             constants.INFERENCESERVICE_CONFIG_MAP_NAME,
             constants.INFERENCESERVICE_SYSTEM_NAMESPACE)
-    except client.rest.ApiException as e:
-        raise RuntimeError(
-            "Exception when calling CoreV1Api->read_namespaced_config_map: %s\n" % e)
+    except client.rest.ApiException:
+        logging.warning('Cannot get configmap %s in namespace %s.',
+                        constants.INFERENCESERVICE_CONFIG_MAP_NAME,
+                        constants.INFERENCESERVICE_SYSTEM_NAMESPACE)
+        return None
 
     isvc_creds_str = isvc_config_map.data['credentials']
     isvc_creds_json = json.loads(isvc_creds_str)


Test patch: