Repository: kserve/kserve

Issue:
`NCCL` and `flash_atten`  packages are missing in huggingface runtime
/kind bug

**What steps did you take and what happened:**
```
INFO 06-03 18:32:32 pynccl_utils.py:17] Failed to import NCCL library: Cannot find libnccl.so.2 in the system.
INFO 06-03 18:32:32 pynccl_utils.py:18] It is expected if you are not running on NVIDIA GPUs.
INFO 06-03 18:32:32 selector.py:81] Cannot use FlashAttention-2 backend because the flash_attn package is not found. Please install it for better performance. 
```
it seems NCCL library and flash_attn package are missing. kserve-container seems to be running this image id [docker.io/kserve/huggingfaceserver@sha256:a3375bcece636afae2ec11928c4fc295d712e00e54e0e69908f978ee543ba6f3](http://docker.io/kserve/huggingfaceserver@sha256:a3375bcece636afae2ec11928c4fc295d712e00e54e0e69908f978ee543ba6f3)
Here is the full log
```
2024-06-03 18:32:31.292 1 kserve INFO [storage.py:download():66] Copying contents of /mnt/models/ to local
INFO 06-03 18:32:31 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='/mnt/models/', speculative_config=None, tokenizer='/mnt/models/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/mnt/models/)
INFO 06-03 18:32:32 pynccl_utils.py:17] Failed to import NCCL library: Cannot find libnccl.so.2 in the system.
INFO 06-03 18:32:32 pynccl_utils.py:18] It is expected if you are not running on NVIDIA GPUs.
INFO 06-03 18:32:32 selector.py:81] Cannot use FlashAttention-2 backend because the flash_attn package is not found. Please install it for better performance.
INFO 06-03 18:32:32 selector.py:32] Using XFormers backend.
INFO 06-03 18:33:03 model_runner.py:175] Loading model weights took 12.5523 GB
INFO 06-03 18:33:04 gpu_executor.py:114] # GPU blocks: 826, # CPU blocks: 512
INFO 06-03 18:33:06 model_runner.py:937] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-03 18:33:06 model_runner.py:941] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 06-03 18:33:15 model_runner.py:1017] Graph capturing finished in 9 secs.
2024-06-03 18:33:15.381 1 kserve INFO [model_server.py:register_model():382] Registering model: kserve-native-vllm-llama-2-7b
2024-06-03 18:33:15.382 1 kserve INFO [model_server.py:start():252] Setting max asyncio worker threads as 5
2024-06-03 18:33:15.382 1 kserve INFO [model_server.py:serve():258] Starting uvicorn with 1 workers
2024-06-03 18:33:15.510 uvicorn.error INFO:     Started server process [1]
2024-06-03 18:33:15.510 uvicorn.error INFO:     Waiting for application startup.
2024-06-03 18:33:15.537 1 kserve INFO [server.py:start():63] Starting gRPC server on [::]:8081
2024-06-03 18:33:15.537 uvicorn.error INFO:     Application startup complete.
2024-06-03 18:33:15.537 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
```

**What did you expect to happen:**


**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version: 0.13.0rc1
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/vllm/vllm_completions.py b/python/huggingfaceserver/huggingfaceserver/vllm/vllm_completions.py
--- a/python/huggingfaceserver/huggingfaceserver/vllm/vllm_completions.py
+++ b/python/huggingfaceserver/huggingfaceserver/vllm/vllm_completions.py
@@ -144,10 +144,9 @@ async def create_completion(self, completion_request: CompletionRequest):
 
                 generators.append(
                     self.engine.generate(
-                        prompt,
+                        {"prompt": prompt, "prompt_token_ids": input_ids},
                         sampling_params,
                         f"{request_id}-{i}",
-                        prompt_token_ids=input_ids,
                     )
                 )
         except Exception as e:
@@ -175,7 +174,7 @@ async def create_completion(self, completion_request: CompletionRequest):
             )
 
         # Non-streaming response
-        final_res_batch: RequestOutput = [None] * len(prompts)
+        final_res_batch: List[RequestOutput] = [None] * len(prompts)
         try:
             async for i, res in result_generator:
                 final_res_batch[i] = res


Test patch: