Repository: kserve/kserve

Issue:
Example to integrate with Fluid to Accelerate KServe Scaling via Model Artifact Caching
**Motivation**

In model serving use case, an application usually contains a model artifact and a servingruntime (main user container). 
Referring to the design of KServe, the user usually provides the model artifact via `storageUri` and defines the `modelFormat` which will leverage the matched servngruntime to serve the model. Before the servingruntime start serving the model, the `storage-initializer`(init container) will download the model artifact from remote storage (usually a cloud object store) to the shared local storage with the servingruntime container. 

As KServe support `serverless`, so every time when an inference service `scales up` from 0 to N, the storage-initializer will kick in to download the model artifact from the remote storage, and usually, the model artifact size varies from MBs to GBs which causes unnecessary bandwidth usage and increase the `cold start` time (A typical `1 GB` model usually take around `1 min+` to download from cloud storage like S3 via storage initializer). It will be very challenging for a user to wait 1 min (not including the image download and request process time) for the first response under serverless mode.

If we can pull the model artifact from a distributed cache (near the cluster) rather than download it from remote cloud storage every time the application scale up, it will help to improve the serverless performance. 

**Describe the solution you'd like**

Introduce [Fluid](https://fluid-cloudnative.github.io/) which is a cloud native distributed dataset orchestrator and accelerator for data-intensive applications as an `example and practice` to reduce the data access time and improve the serverless performance. 

Scopes of this topic:
	1. Add a doc about how to use Fluid to accelerate the Model Artifact access of KServe.
	2. Solve any issues during the integration if any.

**Anything else you would like to add:**

From my own testing, Fluid is able to reduce the access time of model artifact (1.3 GB for example) from 1 min+ to 5-20s (results vary from different runtime used in Fluid)

Fluid is able to work with KNative perfectly (find an example [here](https://github.com/fluid-cloudnative/fluid/blob/master/docs/en/samples/knative.md#demo---how-to-run-in-a-serverless-environment-with-knative-as-an-example)). As Kserve is built on top of KNative and [PVC](https://kserve.github.io/website/master/modelserving/storage/pvc/pvc/#copy-model-to-pv) is a supported storage type as well, it shouldn't be too hard to integrate Fluid with KServe.



Discussion:
cc @yuzisun @cheyang @wsxiaozhang
This seems very relevant for LLMs. Can we tag with [kserve/llm]?

Resolving patch:
diff --git a/docs/samples/fluid/docker/models.py b/docs/samples/fluid/docker/models.py
new file mode 100644
--- /dev/null
+++ b/docs/samples/fluid/docker/models.py
@@ -0,0 +1,94 @@
+# Copyright 2023, SAP SE or an SAP affiliate company and KServe contributors
+
+import os
+
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+from flask import Flask, request
+import logging
+from signal import signal, SIGINT
+
+app = Flask(__name__)
+logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s')
+
+model, tokenizer = None, None
+
+MODEL_URL = os.environ.get('MODEL_URL', '/mnt/models')
+MODEL_NAME = os.environ.get('MODEL_NAME', 'custom')
+LOAD_IN_8BIT = os.environ.get('LOAD_IN_8BIT', 'False') == 'True'
+GPU_ENABLED = os.environ.get('GPU_ENABLED', 'False') == 'True'
+
+
+def handler(signal_received, frame):
+    # SIGINT or  ctrl-C detected, exit without error
+    exit(0)
+
+
+def check_gpu():
+    app.logger.info(f"CUDA is avaiable: {torch.cuda.is_available()}")
+
+    # setting device on GPU if available, else CPU
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    app.logger.info(f'Using device: {device}')
+
+    # additional Info when using cuda
+    if device.type == 'cuda':
+        gpu_count = torch.cuda.device_count()
+        app.logger.info(f"Device Count: {gpu_count}")
+        for i in range(gpu_count):
+            app.logger.info(f"Device ID: {i}")
+            app.logger.info(f"Device Name: {torch.cuda.get_device_name(i)}")
+            app.logger.info('Memory Usage:')
+            app.logger.info(f'Allocated: {round(torch.cuda.memory_allocated(i)/1024**3,1)} GB')
+            app.logger.info(f'Cached: {round(torch.cuda.memory_reserved(i)/1024**3,1)} GB')
+
+
+@app.before_first_request
+def init():
+
+    check_gpu()
+
+    global model, tokenizer
+    if model is None or tokenizer is None:
+        app.logger.info(f"Loading Model from {MODEL_URL}")
+        app.logger.info(f"Loading Model in 8 bit: {LOAD_IN_8BIT}")
+
+        tokenizer = AutoTokenizer.from_pretrained(MODEL_URL)
+        model = AutoModelForCausalLM.from_pretrained(MODEL_URL, device_map="auto", load_in_8bit=LOAD_IN_8BIT)
+        app.logger.info("Model loaded successfully")
+
+
+@app.route("/v1/models/{}:predict".format(MODEL_NAME), methods=["POST"])
+def predict():
+    """
+    Perform an inference on the model created in initialize
+
+    Returns:
+        String response of the prompt for the given test data
+    """
+    global model, tokenizer
+    input_data = dict(request.json)
+
+    if "prompt" not in input_data:
+        return "Prompt not found", 400
+
+    prompt = input_data["prompt"]
+    result_length = input_data.get("result_length", 50)
+
+    if GPU_ENABLED:
+        input = tokenizer(prompt, return_tensors="pt").to("cuda")
+    else:
+        input = tokenizer(prompt, return_tensors="pt")
+
+    result = tokenizer.batch_decode(
+        model.generate(input["input_ids"], max_length=result_length)[0],
+        skip_special_tokens=True)
+
+    output = {"result": "".join(result)}
+    return output
+
+
+if __name__ == "__main__":
+    signal(SIGINT, handler)
+    app.run(host="0.0.0.0", debug=True, port=8080)
diff --git a/docs/samples/fluid/download_model.py b/docs/samples/fluid/download_model.py
new file mode 100644
--- /dev/null
+++ b/docs/samples/fluid/download_model.py
@@ -0,0 +1,25 @@
+# Copyright 2023, SAP SE or an SAP affiliate company and KServe contributors
+
+import argparse
+from pathlib import Path
+from huggingface_hub import snapshot_download
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model_name', default="bigscience/bloom-560m", help='model name from huggingface')
+    parser.add_argument('--model_dir', default="models", help='dir to download the model')
+    parser.add_argument('--revision', default="main", help='revision of the model')
+    args = vars(parser.parse_args())
+
+    model_name = args["model_name"]
+    revision = args["revision"]
+
+    model_dir = Path(args["model_dir"])
+    model_dir.mkdir(exist_ok=True)
+
+    snapshot_download(repo_id=model_name, revision=revision, cache_dir=model_dir)
+
+    # reference: https://aws.amazon.com/de/blogs/machine-learning/deploy-bloom-176b-and-opt-30b-on-amazon-sagemaker-with-large-model-inference-deep-learning-containers-and-deepspeed/ # noqa: E501
+    output_dir = list(model_dir.glob("**/snapshots/*"))[0]
+    print(f"export output_dir={output_dir}")


Test patch: