Repository: modal-labs/modal-examples

Issue:
Mixtral tutorial doesn't work without huggingface access token
The tutorial for running mixtral on VLLM doesn't work since the model cannot be downloaded without a huggingface access token. This is because mixtral is now a gated model: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

```
Image build for im-1P0Aou6cl9H3BAwictbALw failed with the exception:                                           │
│ GatedRepoError('401 Client Error. (Request ID:                                                                 │
│ Root=1-66213747-475d6ad5261bb9eb4931c4fd;025f8bf1-0bb2-42ac-86a0-743e752004a0)\n\nCannot access gated repo for │
│ url https://huggingface.co/api/models/mistralai/Mixtral-8x7B-Instruct-v0.1/revision/main.\nRepo model          │
│ mistralai/Mixtral-8x7B-Instruct-v0.1 is gated. You must be authenticated to access it.')
```

Affected Tutorial: https://modal.com/docs/examples/vllm_mixtral
Affected Code: https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py

This can be fixed using a similar approach as done [here]( https://github.com/modal-labs/modal-examples/pull/539) (adding an env var for a `HF_TOKEN`) to the [function call](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py#L45-L49) where the model is downloaded.
Also the tutorial needs to be updated to inform the user that a huggingface access token is required.


Discussion:
Thanks for the thorough report! We just picked up this failure in our monitoring system, and it's great to have this issue to confirm it and suggest the fix.

Resolving patch:
diff --git a/06_gpu_and_ml/llm-serving/vllm_mixtral.py b/06_gpu_and_ml/llm-serving/vllm_mixtral.py
--- a/06_gpu_and_ml/llm-serving/vllm_mixtral.py
+++ b/06_gpu_and_ml/llm-serving/vllm_mixtral.py
@@ -20,6 +20,7 @@
 
 MODEL_DIR = "/model"
 MODEL_NAME = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+MODEL_REVISION = "1e637f2d7cb0a9d6fb1922f305cb784995190a83"
 GPU_CONFIG = modal.gpu.A100(memory=80, count=2)
 
 
@@ -33,10 +34,15 @@
 #
 # We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.
 #
-# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.
+# For this step to work on a [gated model](https://huggingface.co/docs/hub/en/models-gated)
+# like Mixtral 8x7B, the `HF_TOKEN` environment variable must be set.
+#
+# After [creating a HuggingFace access token](https://huggingface.co/settings/tokens)
+# and accepting the [terms of use](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1),
+# head to the [secrets page](https://modal.com/secrets) to share it with Modal as `huggingface-secret`.
 #
-# Tip: avoid using global variables in this function. Changes to code outside this function will not be detected and the download step will not re-run.
-def download_model_to_image(model_dir, model_name):
+# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.
+def download_model_to_image(model_dir, model_name, model_revision):
     from huggingface_hub import snapshot_download
     from transformers.utils import move_cache
 
@@ -44,6 +50,7 @@ def download_model_to_image(model_dir, model_name):
 
     snapshot_download(
         model_name,
+        revision=model_revision,
         local_dir=model_dir,
         ignore_patterns=["*.pt", "*.bin"],  # Using safetensors
     )
@@ -69,7 +76,12 @@ def download_model_to_image(model_dir, model_name):
     .run_function(
         download_model_to_image,
         timeout=60 * 20,
-        kwargs={"model_dir": MODEL_DIR, "model_name": MODEL_NAME},
+        kwargs={
+            "model_dir": MODEL_DIR,
+            "model_name": MODEL_NAME,
+            "model_revision": MODEL_REVISION,
+        },
+        secrets=[modal.Secret.from_name("huggingface-secret")],
     )
 )
 


Test patch: