FP!6 version Available?

#1
by eduardem - opened

I am wondering if the FP16 version of this model is available.

From my previous tests, GPTQ quantization introduces too much noise for the language related tasks

{'error': 'Could not load model FPHam/Jackson_The_Formalizer_V2_13b_GPTQ with any of the following classes: (<class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>,). See the original errors:\n\nwhile loading with LlamaForCausalLM, an error is thrown:\nTraceback (most recent call last):\n File "/src/transformers/src/transformers/pipelines/base.py", line 279, in infer_framework_load_model\n model = model_class.from_pretrained(model, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/src/transformers/src/transformers/modeling_utils.py", line 3236, in from_pretrained\n raise EnvironmentError(\nOSError: FPHam/Jackson_The_Formalizer_V2_13b_GPTQ does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack.\n\n\n'}

Sign up or log in to comment