Repository: kserve/kserve

Issue:
FileNotFoundError when using an s3 bucket as the model_dir with HuggingFace model server
/kind bug

First of all, I'd like to say thank you for the work on KServe! It's been delightful so far playing around with KServe. But we found a small bug while testing out the HuggingFace model server (which we're aware is a very new addition as well).

**What steps did you take and what happened:**
1. Created an InferenceService using the HuggingFace model server (yaml pasted below)
2. Specified an s3 bucket as the `model_dir` (I suspect this might happen for anything that's not a local dir)
3. Observed that the model is succesfully downloaded to a tmp directory and loaded, but then encountered the `FileNotFoundError` right after

Logs:
```
% k logs huggingface-predictor-00003-deployment-8659bb8b9-m945b
Defaulted container "kserve-container" out of: kserve-container, queue-proxy
INFO:root:Copying contents of s3://kserve-test-models/classifier to local
INFO:root:Downloaded object classifier/config.json to /tmp/tmpckx_trr1/config.json
...
INFO:root:Successfully copied s3://kserve-test-models/classifier to /tmp/tmpckx_trr1
INFO:kserve:successfully loaded tokenizer for task: 4
INFO:kserve:successfully loaded huggingface model from path /tmp/tmpckx_trr1
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/huggingfaceserver/huggingfaceserver/__main__.py", line 69, in <module>
    kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(
  File "/huggingfaceserver/huggingfaceserver/huggingface_model_repository.py", line 24, in __init__
    self.load_models()
  File "/kserve/kserve/model_repository.py", line 37, in load_models
    for name in os.listdir(self.models_dir):
FileNotFoundError: [Errno 2] No such file or directory: 's3://kserve-test-models/spam-classifier'
```


**What did you expect to happen:**

I expected that this would work, as the model was successfully downloaded and loaded. But I did find a tmp workaround below and I think I know where the issue is!

**What's the InferenceService yaml:**
```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface
spec:
  predictor:
    serviceAccountName: huggingface-sa
    containers:
    - args:
      - --model_name=spam-classifier
      # - --model_id=xyz (see workaround below)
      - --model_dir=s3://kserve-test-models/classifier
      - --tensor_input_names=input_ids
      image: kserve/huggingfaceserver:latest
      name: kserve-container
```

**Anything else you would like to add:**

A temporary workaround I found is to supply the `model_id` argument. It can have any value, as the `model_dir` will override it anyway during loading:

https://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/model.py#L91-L94

<details>
  <summary>I have verified that this workaround works (expand to see logs).</summary>

```
% k logs huggingface-predictor-00004-deployment-946b4d6c8-pk5nj -f
Defaulted container "kserve-container" out of: kserve-container, queue-proxy
INFO:root:Copying contents of s3://kserve-test-models/classifier to local
INFO:root:Downloaded object classifier/config.json to /tmp/tmppwjsica7/config.json
...
INFO:kserve:successfully loaded tokenizer for task: 4
INFO:kserve:successfully loaded huggingface model from path /tmp/tmppwjsica7
INFO:kserve:Registering model: classifier
INFO:kserve:Setting max asyncio worker threads as 5
INFO:kserve:Starting uvicorn with 1 workers
2024-02-09 18:57:33.228 uvicorn.error INFO:     Started server process [1]
2024-02-09 18:57:33.229 uvicorn.error INFO:     Waiting for application startup.
2024-02-09 18:57:33.234 1 kserve INFO [start():62] Starting gRPC server on [::]:8081
2024-02-09 18:57:33.234 uvicorn.error INFO:     Application startup complete.
2024-02-09 18:57:33.235 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
```
</details>

I think the issue is here:
https://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/__main__.py#L63-L72

1. `model.load()` will succeed, so we jump to line 68
2. It checks for `args.model_id`, which is empty, so we go inside the if block
3. It will try to instantiate `HuggingfaceModelRepository` with `model_dir`, which is pointing to an s3 bucket and not a local directory, thus causing the `FileNotFoundError`
4. This is how I came up with the workaround of passing `model_id`, so that the else block is executed instead (because the model did load succesfully, so doing `kserve.ModelServer().start([model] if model.ready else [])` won't be a problem)

**Environment:**

- Cloud Environment: aws
- Kubernetes version: (use `kubectl version`): v1.27.9-eks-5e0fdde
- OS (e.g. from `/etc/os-release`): Ubuntu 22.04.3 LTS



Discussion:

Resolving patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/__main__.py b/python/huggingfaceserver/huggingfaceserver/__main__.py
--- a/python/huggingfaceserver/huggingfaceserver/__main__.py
+++ b/python/huggingfaceserver/huggingfaceserver/__main__.py
@@ -62,11 +62,9 @@ def list_of_strings(arg):
                              kwargs=vars(args), engine_args=engine_args)
     try:
         model.load()
+        kserve.ModelServer().start([model] if model.ready else [])
     except ModelMissingError:
         logging.error(f"fail to locate model file for model {args.model_name} under dir {args.model_dir},"
                       f"trying loading from model repository.")
-    if not args.model_id:
         kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(
             [model] if model.ready else [])
-    else:
-        kserve.ModelServer().start([model] if model.ready else [])


Test patch: