README.md

---
base_model: arcee-ai/Virtuoso-Small-v2
library_name: transformers
license: apache-2.0
tags:
- llama-cpp
- gguf-my-repo
---

# Triangle104/Virtuoso-Small-v2-Q6_K-GGUF
This model was converted to GGUF format from [`arcee-ai/Virtuoso-Small-v2`](https://huggingface.co/arcee-ai/Virtuoso-Small-v2) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/arcee-ai/Virtuoso-Small-v2) for more details on the model.

---
Virtuoso-Small-v2 (14B) is our next-generation, 14-billion-parameter language model that builds upon the original Virtuoso-Small architecture. This version is distilled from Deepseek-v3, leveraging an expanded dataset of 5B+ tokens worth of logits.
Model Details

    Architecture Base: Qwen-2.5-14B
    Parameter Count: 14B
    Tokenizer:
        Initially integrated with Deepseek-v3 tokenizer for logit extraction.
        Final alignment uses the Qwen tokenizer, using specialized “tokenizer surgery” for cross-architecture compatibility.
    Distillation Data:
        ~1.1B tokens/logits from Deepseek-v3’s training data.
        Logit-level distillation using a proprietary “fusion merging” approach afterwards for maximum fidelity.
    License: Apache-2.0

Background on Deepseek Distillation

Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, we apply a full logit-level replication. This ensures more precise transference of knowledge, including advanced reasoning in:

    Technical and scientific queries
    Complex code generation
    Mathematical problem-solving

How to Use

Below is a sample code snippet using transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "arcee-ai/Virtuoso-Small-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Provide a concise summary of quantum entanglement."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training & Fine-Tuning

    Initial Training: Began with Qwen-14B, calibrated for large-scale text ingestion.
    Distillation & Merging:
        Trained on ~1.1B tokens worth of Deepseek-v3 logits.
        Employed “fusion merging” to retain as much teacher expertise as possible.
        Final step included DPO to improve alignment and reduce model hallucinations.
    Continuous Development: Additional R1 distillations are in progress to further enhance performance and specialization.

Limitations

    Context Length: 128k Tokens
    Knowledge Cut-off: Training data may not reflect the latest events or developments, leading to gaps in current knowledge beyond June 2024.

Ethical Considerations

    Content Generation Risks: Like any language model, Virtuoso-Small-v2 can potentially generate harmful or biased content if prompted in certain ways.

License

Virtuoso-Small-v2 (14B) is released under the Apache-2.0 License. You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.

If you have questions or would like to share your experiences using these models, please connect with us on social media. We’re excited to see what you build—and how these models help you innovate!

---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo Triangle104/Virtuoso-Small-v2-Q6_K-GGUF --hf-file virtuoso-small-v2-q6_k.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo Triangle104/Virtuoso-Small-v2-Q6_K-GGUF --hf-file virtuoso-small-v2-q6_k.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/Virtuoso-Small-v2-Q6_K-GGUF --hf-file virtuoso-small-v2-q6_k.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo Triangle104/Virtuoso-Small-v2-Q6_K-GGUF --hf-file virtuoso-small-v2-q6_k.gguf -c 2048
```