Triangle104/Bellatrix-Tiny-3B-R1-Q6_K-GGUF

This model was converted to GGUF format from prithivMLmods/Bellatrix-Tiny-3B-R1 using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Bellatrix is based on a reasoning-based model designed for the DeepSeek-R1 synthetic dataset entries. The pipeline's instruction-tuned, text-only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks. These models outperform many of the available open-source options. Bellatrix is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions utilize supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF).

    Use with transformers

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via:

pip install --upgrade transformers

import torch from transformers import pipeline

model_id = "prithivMLmods/Bellatrix-Tiny-3B-R1" pipe = pipeline( "text-generation", model=model_id, torch_dtype=torch.bfloat16, device_map="auto", ) messages = [ {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"}, {"role": "user", "content": "Who are you?"}, ] outputs = pipe( messages, max_new_tokens=256, ) print(outputs[0]["generated_text"][-1])

Note: You can also find detailed recipes on how to use the model locally, with torch.compile(), assisted generations, quantization, and more at huggingface-llama-recipes.

    Intended Use

Bellatrix is designed for applications that require advanced reasoning and multilingual dialogue capabilities. It is particularly suitable for:

Agentic Retrieval: Enabling intelligent retrieval of relevant information in a dialogue or query-response system.
Summarization Tasks: Condensing large bodies of text into concise summaries for easier comprehension.
Multilingual Use Cases: Supporting conversations in multiple languages with high accuracy and coherence.
Instruction-Based Applications: Following complex, context-aware instructions to generate precise outputs in a variety of scenarios.

    Limitations

Despite its capabilities, Bellatrix has some limitations:

Domain Specificity: While it performs well on general tasks, its performance may degrade with highly specialized or niche datasets.
Dependence on Training Data: It is only as good as the quality and diversity of its training data, which may lead to biases or inaccuracies.
Computational Resources: The model’s optimized transformer architecture can be resource-intensive, requiring significant computational power for fine-tuning and inference.
Language Coverage: While multilingual, some languages or dialects may have limited support or lower performance compared to widely used ones.
Real-World Contexts: It may struggle with understanding nuanced or ambiguous real-world scenarios not covered during training.

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Bellatrix-Tiny-3B-R1-Q6_K-GGUF --hf-file bellatrix-tiny-3b-r1-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Bellatrix-Tiny-3B-R1-Q6_K-GGUF --hf-file bellatrix-tiny-3b-r1-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Bellatrix-Tiny-3B-R1-Q6_K-GGUF --hf-file bellatrix-tiny-3b-r1-q6_k.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Bellatrix-Tiny-3B-R1-Q6_K-GGUF --hf-file bellatrix-tiny-3b-r1-q6_k.gguf -c 2048