RachidAR/Phi-3-mini-4k-ins-June2024-Q5_K_M-imat-GGUF (June 2024 Update)

This model was converted to GGUF format from microsoft/Phi-3-mini-4k-instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Release Notes

This is an update over the original instruction-tuned Phi-3-mini release based on valuable customer feedback. The model used additional post-training data leading to substantial gains on instruction following and structure output. We also improve multi-turn conversation quality, explicitly support <|system|> tag, and significantly improve reasoning capability. We believe most use cases will benefit from this release, but we encourage users to test in their particular AI applications. We appreciate the enthusiastic adoption of the Phi-3 model family, and continue to welcome all feedback from the community.

The table below highlights improvements on instruction following, structure output, and reasoning of the new release on publich and internal benchmark datasets.

Benchmarks Original June 2024 Update
Instruction Extra Hard 5.7 6.0
Instruction Hard 4.9 5.1
Instructions Challenge 24.6 42.3
JSON Structure Output 11.5 52.3
XML Structure Output 14.4 49.8
GPQA 23.7 30.6
MMLU 68.8 70.9
Average 21.9 36.7

Chat Format

Given the nature of the training data, the Phi-3 Mini-4K-Instruct model is best suited for prompts using the chat format as follows. You can provide the prompt as a question with a generic template as follow:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
Question?<|end|>
<|assistant|>

For example:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
How to explain Internet for a medieval knight?<|end|>
<|assistant|> 

where the model generates the text after <|assistant|> . In case of few-shots prompt, the prompt can be formatted as the following:

<|system|>
You are a helpful travel assistant.<|end|>
<|user|>
I am going to Paris, what should I see?<|end|>
<|assistant|>
Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."<|end|>
<|user|>
What is so great about #1?<|end|>
<|assistant|>

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048
Downloads last month
5
GGUF
Model size
3.82B params
Architecture
phi3

5-bit

Inference Examples
Unable to determine this model's library. Check the docs .

Model tree for RachidAR/Phi-3-mini-4k-ins-June2024-Q5_K_M-imat-GGUF

Quantized
(98)
this model