RichardErkhov/TechxGenus_-_Yi-9B-Coder-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Yi-9B-Coder - GGUF

• Model creator: https://huggingface.co/TechxGenus/
• Original model: https://huggingface.co/TechxGenus/Yi-9B-Coder/

Name	Quant method	Size
Yi-9B-Coder.Q2_K.gguf	Q2_K	3.12GB
Yi-9B-Coder.IQ3_XS.gguf	IQ3_XS	3.46GB
Yi-9B-Coder.IQ3_S.gguf	IQ3_S	3.64GB
Yi-9B-Coder.Q3_K_S.gguf	Q3_K_S	3.63GB
Yi-9B-Coder.IQ3_M.gguf	IQ3_M	3.78GB
Yi-9B-Coder.Q3_K.gguf	Q3_K	4.03GB
Yi-9B-Coder.Q3_K_M.gguf	Q3_K_M	4.03GB
Yi-9B-Coder.Q3_K_L.gguf	Q3_K_L	4.37GB
Yi-9B-Coder.IQ4_XS.gguf	IQ4_XS	4.5GB
Yi-9B-Coder.Q4_0.gguf	Q4_0	4.69GB
Yi-9B-Coder.IQ4_NL.gguf	IQ4_NL	4.73GB
Yi-9B-Coder.Q4_K_S.gguf	Q4_K_S	4.72GB
Yi-9B-Coder.Q4_K.gguf	Q4_K	4.96GB
Yi-9B-Coder.Q4_K_M.gguf	Q4_K_M	4.96GB
Yi-9B-Coder.Q4_1.gguf	Q4_1	5.19GB
Yi-9B-Coder.Q5_0.gguf	Q5_0	5.69GB
Yi-9B-Coder.Q5_K_S.gguf	Q5_K_S	5.69GB
Yi-9B-Coder.Q5_K.gguf	Q5_K	5.83GB
Yi-9B-Coder.Q5_K_M.gguf	Q5_K_M	5.83GB
Yi-9B-Coder.Q5_1.gguf	Q5_1	6.19GB
Yi-9B-Coder.Q6_K.gguf	Q6_K	6.75GB
Yi-9B-Coder.Q8_0.gguf	Q8_0	8.74GB

Original model description:

tags: - code - llama library_name: transformers pipeline_tag: text-generation license: apache-2.0

Yi-Coder

We've fine-tuned Yi-9B with an additional 0.5 billion high-quality, code-related tokens for 3 epochs. We used DeepSpeed ZeRO 3 and Flash Attention 2 to accelerate the training process. It achieves 69.5 pass@1 on HumanEval-Python. This model operates using the Alpaca instruction format (excluding the system prompt).

Usage

Here give some examples of how to use our model:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
PROMPT = """### Instruction
{instruction}
### Response
"""
instruction = <Your code instruction here>
prompt = PROMPT.format(instruction=instruction)
tokenizer = AutoTokenizer.from_pretrained("TechxGenus/Yi-9B-Coder")
model = AutoModelForCausalLM.from_pretrained(
    "TechxGenus/Yi-9B-Coder",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=2048)
print(tokenizer.decode(outputs[0]))

With text-generation pipeline:

from transformers import pipeline
import torch
PROMPT = """### Instruction
{instruction}
### Response
"""
instruction = <Your code instruction here>
prompt = PROMPT.format(instruction=instruction)
generator = pipeline(
    model="TechxGenus/Yi-9B-Coder",
    task="text-generation",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
result = generator(prompt, max_length=2048)
print(result[0]["generated_text"])

Note

Model may sometimes make errors, produce misleading contents, or struggle to manage tasks that are not related to coding. It has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.