QuantFactory/calme-2.3-phi3-4b-GGUF
This is quantized version of MaziyarPanahi/calme-2.3-phi3-4b created using llama.cpp
Original Model Card
![Phi-3 Logo](/QuantFactory/calme-2.3-phi3-4b-GGUF/resolve/main/phi-3-instruct.webp)
MaziyarPanahi/calme-2.3-phi3-4b
This model is a fine-tune (DPO) of microsoft/Phi-3-mini-4k-instruct
model.
⚡ Quantized GGUF
All GGUF models are available here: MaziyarPanahi/calme-2.3-phi3-4b-GGUF
🏆 Open LLM Leaderboard Evaluation Results
Detailed results can be found here
** Leaderboard 2**
Metric | Value |
---|---|
Avg. | 23.38 |
IFEval (0-Shot) | 49.26 |
BBH (3-Shot) | 37.66 |
MATH Lvl 5 (4-Shot) | 2.95 |
GPQA (0-shot) | 9.06 |
MuSR (0-shot) | 7.75 |
MMLU-PRO (5-shot) | 31.42 |
** Leaderboard 1**
Metric | Value |
---|---|
Avg. | 70.26 |
AI2 Reasoning Challenge (25-Shot) | 63.48 |
HellaSwag (10-Shot) | 80.86 |
MMLU (5-Shot) | 69.24 |
TruthfulQA (0-shot) | 60.66 |
Winogrande (5-shot) | 72.77 |
GSM8k (5-shot) | 74.53 |
MaziyarPanahi/calme-2.3-phi3-4b
is the best-performing Phi-3-mini-4k model on the Open LLM Leaderboard. (03/06/2024).
Prompt Template
This model uses ChatML
prompt template:
<|im_start|>system
{System}
<|im_end|>
<|im_start|>user
{User}
<|im_end|>
<|im_start|>assistant
{Assistant}
How to use
You can use this model by using MaziyarPanahi/calme-2.3-phi3-4b
as the model name in Hugging Face's
transformers library.
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
from transformers import pipeline
import torch
model_id = "MaziyarPanahi/calme-2.3-phi3-4b"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
# attn_implementation="flash_attention_2"
)
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True
)
streamer = TextStreamer(tokenizer)
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
# this should work perfectly for the model to stop generating
terminators = [
tokenizer.eos_token_id, # this should be <|im_end|>
tokenizer.convert_tokens_to_ids("<|assistant|>"), # sometimes model stops generating at <|assistant|>
tokenizer.convert_tokens_to_ids("<|end|>") # sometimes model stops generating at <|end|>
]
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
)
generation_args = {
"max_new_tokens": 500,
"return_full_text": False,
"temperature": 0.0,
"do_sample": False,
"streamer": streamer,
"eos_token_id": terminators,
}
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])
- Downloads last month
- 113
Model tree for QuantFactory/calme-2.3-phi3-4b-GGUF
Base model
microsoft/Phi-3-mini-4k-instruct
Quantized
this model
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard63.480
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard80.860
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard69.240
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard60.660
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard72.770
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard74.530
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard49.260
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard37.660
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard2.950
- acc_norm on GPQA (0-shot)Open LLM Leaderboard9.060