| license: apache-2.0 | |
| tags: | |
| - finetuned | |
| - quantized | |
| - 4-bit | |
| - gptq | |
| - transformers | |
| - safetensors | |
| - llama | |
| - text-generation | |
| - en | |
| - license:mit | |
| - model-index | |
| - autotrain_compatible | |
| - endpoints_compatible | |
| - has_space | |
| - text-generation-inference | |
| - region:us | |
| model_name: miqu-1-70b-sf-GPTQ | |
| base_model: 152334H/miqu-1-70b-sf | |
| inference: false | |
| model_creator: 152334H | |
| pipeline_tag: text-generation | |
| quantized_by: MaziyarPanahi | |
| # Description | |
| [MaziyarPanahi/miqu-1-70b-sf-GPTQ](https://huggingface.co/MaziyarPanahi/miqu-1-70b-sf-GPTQ) is a quantized (GPTQ) version of [152334H/miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf) | |
| ## How to use | |
| ### Install the necessary packages | |
| ``` | |
| pip install --upgrade accelerate auto-gptq transformers | |
| ``` | |
| ### Example Python code | |
| ```python | |
| from transformers import AutoTokenizer, pipeline | |
| from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig | |
| import torch | |
| model_id = "MaziyarPanahi/miqu-1-70b-sf-GPTQ" | |
| quantize_config = BaseQuantizeConfig( | |
| bits=4, | |
| group_size=128, | |
| desc_act=False | |
| ) | |
| model = AutoGPTQForCausalLM.from_quantized( | |
| model_id, | |
| use_safetensors=True, | |
| device="cuda:0", | |
| quantize_config=quantize_config) | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| pipe = pipeline( | |
| "text-generation", | |
| model=model, | |
| tokenizer=tokenizer, | |
| max_new_tokens=512, | |
| temperature=0.7, | |
| top_p=0.95, | |
| repetition_penalty=1.1 | |
| ) | |
| outputs = pipe("What is a large language model?") | |
| print(outputs[0]["generated_text"]) | |
| ``` |