Issue loading quantized version of ESM2 model
#1
by
sanjanatule
- opened
Thank you for amazing blog post. I am facing issue loading ESM models in quantised format. I get the error " KeyError: 'inv_freq'".
Appreciate your help.
Here is the snipped of my code:
Quantization definition
bnb_config = BitsAndBytesConfig(
load_in_4bit = True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type=“nf4”,
bnb_4bit_compute_dtype=torch.bfloat16,
)
load the qunatized model
esm_model_name = “facebook/esm2_t30_150M_UR50D”
model = AutoModelForTokenClassification.from_pretrained(esm_model_name, quantization_config = bnb_config,trust_remote_code=True,device_map=‘cuda’)
model