Text Generation
Transformers
Safetensors
English
mixtral
conversational
text-generation-inference
4-bit precision
awq

Doesn't load on Runpod

#2
by IronicKeyboard - opened

I'm using the template for "TheBloke Local LLMs One-Click UI and API" template using an L40. When I attempt to load the model, I get the following error:
Traceback (most recent call last):

File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 213, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)
File "/workspace/text-generation-webui/modules/models.py", line 87, in load_model

output = load_func_maploader
File "/workspace/text-generation-webui/modules/models.py", line 311, in AutoAWQ_loader

model = AutoAWQForCausalLM.from_quantized(
File "/usr/local/lib/python3.10/dist-packages/awq/models/auto.py", line 50, in from_quantized

model_type = check_and_get_model_type(quant_path, trust_remote_code)
File "/usr/local/lib/python3.10/dist-packages/awq/models/auto.py", line 25, in check_and_get_model_type

raise TypeError(f"{config.model_type} isn't supported yet.")
TypeError: mixtral isn't supported yet.

Sign up or log in to comment