TheBloke

/

Mixtral-8x7B-Instruct-v0.1-GGUF

like 560

Hi, I'm getting a download error when trying to deploy to SageMaker using the script that is in Deploy -> Amazon SageMaker.
Here are the script and the error log.

Deploy Script:

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF',
    'SM_NUM_GPUS': json.dumps(1)
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    image_uri=get_huggingface_llm_image_uri("huggingface",version="1.1.0"),
    env=hub,
    role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    container_startup_health_check_timeout=300,
  )

Error Log:

[huggingface_hub.utils._errors.EntryNotFoundError: No .bin weights found for model TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF and revision None.
and](Error: DownloadError
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/hub.py", line 96, in weight_files
    filenames = weight_hub_files(model_id, revision, extension)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/hub.py", line 37, in weight_hub_files
    raise EntryNotFoundError()

huggingface_hub.utils._errors.EntryNotFoundError: No .safetensors weights found for model TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF and revision None.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 115, in download_weights
    utils.weight_files(model_id, revision, extension)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/hub.py", line 101, in weight_files
    pt_filenames = weight_hub_files(model_id, revision, extension=".bin")
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/hub.py", line 37, in weight_hub_files
    raise EntryNotFoundError(

huggingface_hub.utils._errors.EntryNotFoundError: No .bin weights found for model TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF and revision None.

Is that script expected to work?