Repository: kserve/kserve

Issue:
Support headers in http(s) uri requests for inference service
/kind feature

**Describe the solution you'd like**
## InferenceService headers support in http(s) uri requests
Downloading model files from a http/https service may require authorization passed through the header. The problem can be solved by storing header details into secrets and extracting them to use when the storage initializer is downloading the model. This will be for non MMS use case.

### Summary
Inference service currently is unable to pass along credentials when making http/https GET requests. Http/https services that were normally unable to be queried due to lack of credentials can work by including the necessary headers. Anyone that requires authentication or additional details passed along with a query will benefit from this change.

### Motivation
There is no method for allowing credentials in the headers when making requests to an http/https service. There should be a flexible solution that makes use of k8s secrets/annotations to store this information.

### Goals
- Storage initializer will be able to download models (tar/zip) from a http/https service that requires header information. (ie: account_name and/or api_key)
- Stores headers in a k8s secret.
- Dynamically determines if a uri has corresponding headers to include.
	
### Non-Goals
- Model agent will be able to download models (tar/zip) from http/https services that require header information.

### Proposal

#### Option 1: Host URI in annotation and headers in data of secret

The idea is to store the host uri of the website in the annotation and the headers in the data of a k8s secret. These values will be stored as environment variables. More specific details will be shown below about the environment variable. In the storage initializer, a request to get the environment variable of the host uri which will return the list of headers. These headers will then get stored in the headers dictionary and added to the request.

##### Option 1.0.1: Prefixed values
**Secret.yml**
```
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    serving.kubeflow.org/https-host-uri: “example.com”
type: Opaque
data:
  header.account-name: “some_account_name”
  header.secret-key: “some_secret_key”
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
  - name: mysecret
```

##### Option 1.0.2: Dedicated fields for header
**Secret.yml**
```
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    serving.kubeflow.org/https-host-uri: “example.com”
type: Opaque
data:
  headers: |-
    account-name: “some_account_name”
    secret-key: “some_secret_key”
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
  - name: mysecret
```

The below shows different suggestions on how to store the headers in environment variables.

Note that both suggestions data will follow the same format <type>.<name>: <value>. This will allow for future flexibility in specifying what to do with the data element key. (ie proxy.proxy_key:proxy_value)

##### Option 1.1: Comma separated headers
Environment variables will be created as the following:
```
<https-host-uri-value>: <comma-separated-data-keys>
<data-element-key>: <data-element-value> # For each data element
```
And in this case:
```
example.com: “header.account-name,header.secret-key”
header.account-name: “some_account_name”
header.secret-key: “some_secret_key”
```

The benefits of this approach is obtaining the list of data keys can quickly be obtained by splitting the string by the comma. The drawback is that this list may be extremely long.

##### Option 1.2: Numbered headers count
Environment variables will be created as the following:
```
<https-host-uri-value>: <number-of-data-element>
<https-host-uri-#>:<data-element-key-#> # For each data element key
<data-element-key>: <data-element-value> # For each data element
```
And in this case:
```
example.com: 2
example.com-1: “header.account-name”
example.com-2: “header.secret-key”
header.account-name: “some_account_name”
header.secret-key: “some_secret_key”
```

The benefit of this approach is there will not be any issues with having too many headers/data keys. But this approach requires iterating through 0 to N (where N is the number of data keys) which can be costly.


Limitations for both cases:
- Does not support same uri but multiple credentials

#### Option 2: Modifying the inference service CRD
The storageURI can be modified to store potential headers. In the inference service, the storageURI can be modified to look like:
```
storageURI:
uri: “example.com”
headers:
account-name: “some_account_name”
secret-key: “some_secret_key”
```
The benefit is being able to explicitly set headers for each URI but changing the definition for inference service does not seem ideal.

I'm currently working on a PR for option 1.1


Discussion:

Resolving patch:
diff --git a/python/kfserving/kfserving/storage.py b/python/kfserving/kfserving/storage.py
--- a/python/kfserving/kfserving/storage.py
+++ b/python/kfserving/kfserving/storage.py
@@ -18,6 +18,7 @@
 import mimetypes
 import os
 import re
+import json
 import shutil
 import tarfile
 import zipfile
@@ -36,6 +37,7 @@
 _LOCAL_PREFIX = "file://"
 _URI_RE = "https?://(.+)/(.+)"
 _HTTP_PREFIX = "http(s)://"
+_HEADERS_SUFFIX = "-headers"
 
 class Storage(object): # pylint: disable=too-few-public-methods
     @staticmethod
@@ -232,12 +234,21 @@ def _download_from_uri(uri, out_dir=None):
         if filename == '':
             raise ValueError('No filename contained in URI: %s' % (uri))
 
-        with requests.get(uri, stream=True) as response:
+        # Get header information from host url
+        headers = {}
+        host_uri = url.hostname
+
+        headers_json = os.getenv(host_uri + _HEADERS_SUFFIX, "{}")
+        headers = json.loads(headers_json)
+
+        with requests.get(uri, stream=True, headers=headers) as response:
             if response.status_code != 200:
                 raise RuntimeError("URI: %s returned a %s response code." % (uri, response.status_code))
             if mimetype == 'application/zip' and not response.headers.get('Content-Type', '').startswith('application/zip'):
                 raise RuntimeError("URI: %s did not respond with \'Content-Type\': \'application/zip\'" % (uri))
-            if mimetype != 'application/zip' and not response.headers.get('Content-Type', '').startswith('application/octet-stream'):
+            if mimetype == 'application/x-tar' and not response.headers.get('Content-Type', '').startswith('application/x-tar'):
+                raise RuntimeError("URI: %s did not respond with \'Content-Type\': \'application/x-tar\'" % (uri))
+            if (mimetype != 'application/zip' and mimetype != 'application/x-tar') and not response.headers.get('Content-Type', '').startswith('application/octet-stream'):
                 raise RuntimeError("URI: %s did not respond with \'Content-Type\': \'application/octet-stream\'" % (uri))
 
             if encoding == 'gzip':


Test patch:
diff --git a/pkg/credentials/https/https_secret_test.go b/pkg/credentials/https/https_secret_test.go
new file mode 100644
--- /dev/null
+++ b/pkg/credentials/https/https_secret_test.go
@@ -0,0 +1,81 @@
+/*
+Copyright 2021 kubeflow.org.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package https
+
+import (
+	"github.com/google/go-cmp/cmp"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"testing"
+)
+
+var (
+	header1      = "username"
+	header2      = "password"
+	headerValue1 = "someUsername"
+	headerValue2 = "somePassword"
+	uriHost      = "example.com"
+)
+
+func TestHTTPSSecret(t *testing.T) {
+	scenarios := map[string]struct {
+		secret   *v1.Secret
+		expected []v1.EnvVar
+	}{
+		"noUriHost": {
+			secret: &v1.Secret{
+				Data: map[string][]byte{
+					header1: []byte(headerValue1),
+					header2: []byte(headerValue2),
+				},
+			},
+			expected: []v1.EnvVar{},
+		},
+		"noHeaders": {
+			secret: &v1.Secret{
+				ObjectMeta: metav1.ObjectMeta{},
+				Data: map[string][]byte{
+					HTTPSHost: []byte(uriHost),
+				},
+			},
+			expected: []v1.EnvVar{},
+		},
+		"secretEnvs": {
+			secret: &v1.Secret{
+				ObjectMeta: metav1.ObjectMeta{},
+				Data: map[string][]byte{
+					HTTPSHost: []byte(uriHost),
+					HEADERS:   []byte(`{` + NEWLINE + header1 + ColonSeparator + headerValue1 + NEWLINE + header2 + ColonSeparator + headerValue2 + NEWLINE + `}`),
+				},
+			},
+			expected: []v1.EnvVar{
+				{
+					Name:  uriHost + HeadersSuffix,
+					Value: `{` + NEWLINE + header1 + ColonSeparator + headerValue1 + NEWLINE + header2 + ColonSeparator + headerValue2 + NEWLINE + `}`,
+				},
+			},
+		},
+	}
+
+	for name, scenario := range scenarios {
+		envs := BuildSecretEnvs(scenario.secret)
+
+		if diff := cmp.Diff(scenario.expected, envs); diff != "" {
+			t.Errorf("Test %q unexpected result (-want +got): %v", name, diff)
+		}
+	}
+}