Exllama not working
faulty config.json
these are missing! i inserted all of those but still get an error i cannot sort of. maybe someone knows what the issue is.
"pad_token_id": 0,
"hidden_size": 8192,
"intermediate_size": 22016,
"num_attention_heads": 64,
"num_hidden_layers": 48,
"rms_norm_eps": 1e-05,
errors:
model = ExLlama(config)
device = self.config.device_map.map(key)
raise ValueError("Unknown key: " + key)
May be related to this.
This model is not compatible with ExLlama. ExLlama only supports Llama models and Llama-derivatives, like Mistral, Yi, Deepseek Coder, etc. Not StarCoder, which is what this model is.
My more recent models have a file table which shows ExLlama compatibility, but I guess I didn't have that when I made this model, and I've not updated the README for it yet. This is a pretty old model now.