File size: 3,619 Bytes
dd993f5 6dc8db3 dd993f5 6dc8db3 f758e3c 6dc8db3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
---
language:
- en
license: llama2
model_name: OpenHathi-7B-Hi-v0.1-Base-gptq
base_model: meta-llama/Llama-2-7b-chat-hf
inference: false
model_creator: SarvamAI
model_type: llama
pipeline_tag: text-generation
prompt_template: '[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as
possible, while being safe. Your answers should not include any harmful, unethical,
racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses
are socially unbiased and positive in nature. If a question does not make any sense,
or is not factually coherent, explain why instead of answering something not correct.
If you don''t know the answer to a question, please don''t share false information.
<</SYS>>
{prompt}[/INST]
'
quantized_by: cmeraki
---
# OpenHathi Base GPTQ
- Model creator: [Sarvam AI](https://huggingface.co/sarvamai)
- Original model: [sarvamai/OpenHathi-7B-Hi-v0.1-Base](https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base/)
<!-- description start -->
## Description
This repo contains GPTQ model files for [Sarvam's OpenHathi](https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base/).
Files are made using AutoGPTQ with following config.
```
quantization_config : {"bits": 4,
"group_size": 128,
"damp_percent": 0.1,
"desc_act": true,
}
```
We use a custom [dataset](cmeraki/wiki_en_hi) which has both Hindi and English wiki articles. We truncate to max_length=1024 and model may not perform well beyond that context size.
<!-- description end -->
<!-- prompt-template start -->
## Prompt template
This is a base model not tuned for any instructions. Feel free to use any format. Alpaca/Vicuna works fine.
<!-- prompt-template end -->
## Oobagooba
Standard oobagooba works with exllama2 / autogptq loader
## Using in code
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_name_or_path = "cmeraki/OpenHathi-7B-Hi-v0.1-Base-gptq"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
device_map="auto",
trust_remote_code=False,
revision="main")
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
prompt = "do aur do"
prompt_template=f'''[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
{prompt}[/INST]
'''
print("\n\n*** Generate:")
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
print(tokenizer.decode(output[0]))
# Inference can also be done using transformers' pipeline
print("*** Pipeline:")
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
max_new_tokens=512,
do_sample=True,
temperature=0.7,
top_p=0.95,
top_k=40,
repetition_penalty=1.1
)
print(pipe(prompt_template)[0]['generated_text'])
```
<!-- README_GPTQ.md-use-from-python end -->
<!-- README_GPTQ.md-compatibility start -->
|