Instructions to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF",
	filename="BAR-2x7B-Tool-Use.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Use Docker

docker model run hf.co/RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Ollama
How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with Ollama:
```
ollama run hf.co/RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
```

Unsloth Studio

How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with Docker Model Runner:
```
docker model run hf.co/RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M
```

Lemonade

How to use RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.BAR-2x7B-Tool-Use-GGUF-Q4_K_M

List all available models

lemonade list

BAR-2x7B-Tool-Use — GGUF (first-of-its-kind FlexOlmo conversion)

This is the first GGUF conversion of allenai/BAR-2x7B-Tool-Use, one of AllenAI's BAR-family Mixture-of-Experts models released on 2026-04-19 based on the new FlexOlmo architecture.

⚠ Requires patched llama.cpp

The FlexOlmo architecture is not yet supported in upstream llama.cpp. To run this GGUF you need a build with FlexOlmo support, currently in flight as a PR.

Build from the support branch:

git clone https://github.com/RhinoWithAcape/llama.cpp.git
cd llama.cpp
git checkout feature/flex-olmo-arch  # (or wait for upstream merge)
cmake -B build -DGGML_CUDA=OFF
cmake --build build -j --target llama-cli llama-quantize llama-completion

Once upstream PR lands, any standard llama.cpp / Ollama install will work directly.

What FlexOlmo is

Per transformers.models.flex_olmo, FlexOlmoDecoderLayer is Olmo2's hybrid post-norm decoder layer with the dense FFN swapped for OlmoE-style top-k MoE routing. Specifically:

Attention with q_norm and k_norm (Olmo2-style)
post_attention_layernorm and post_feedforward_layernorm (post-norm pattern, no input_layernorm)
Top-k MoE FFN with softmax routing (OlmoE-style)
No sliding-window attention

Model details

Field	Value
Architecture	`FlexOlmoForCausalLM` (Olmo2 hybrid + OlmoE MoE)
Total parameters	~11.6 B
Active parameters per token	~11 B (top-2 of 2 experts → effectively dense forward)
Layers	32
Hidden size	4096
Attention heads	32 (no GQA; head dim 128)
Experts	2 routed, top-2 selected
Vocab	100,278
Context	64K (rope_theta 500000)

Quants

Quant	Size	Status
Q4_K_M	~6.7 GB	✅ uploaded — recommended for consumer GPUs (12GB+)
Q5_K_M	~8 GB	rolling out (uploaded as ready)
Q6_K	~10 GB	rolling out
Q8_0	~12 GB	rolling out
F16	~22 GB	available on request

Usage — Ollama

(Once your llama.cpp / Ollama has FlexOlmo support):

hf download RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF \
  BAR-2x7B-Tool-Use.Q4_K_M.gguf Modelfile --local-dir ./bar
cd ./bar
ollama create bar-2x7b-tool:Q4_K_M -f Modelfile
ollama run bar-2x7b-tool:Q4_K_M "Hello"

Usage — llama.cpp (with FlexOlmo support built)

./build/bin/llama-completion \
    -m BAR-2x7B-Tool-Use.Q4_K_M.gguf \
    -p "Q: A train travels 60 miles in 1.5 hours. What's its average speed in mph?\nA:" \
    -n 60 --temp 0.3

Sample output:

"To find the average speed, we need to divide the distance by the time. So, the average speed is 60 miles / 1.5 hours = 40 mph."

Validation

Q4_K_M was tested on three prompts during development:

Factual: "The capital of France is" → "Paris."
Reasoning: 60mi/1.5h speed problem → "40 mph" with steps
Code/explanation: fibonacci docstring → coherent explanation of memoization

All clean stops on the model's EOS, no looping or degeneration.

License

Apache 2.0 (matching the source release at AllenAI). This conversion is a derivative work — same license applies.

Conversion details

Source: allenai/BAR-2x7B-Tool-Use (downloaded 2026-04-29)
Tools: patched llama.cpp on feature/flex-olmo-arch branch
Steps: convert_hf_to_gguf.py → llama-quantize
Architecture support added in: PR (pending) <link>

Acknowledgments

AllenAI for the BAR family release and the FlexOlmo architecture
llama.cpp maintainers — the existing Olmo2 + OlmoE handlers gave us the right primitives to combine
This conversion produced for the Zenith swarm project — autonomous engineering collective

Citation

If you use this GGUF, please cite AllenAI's BAR release:

@misc{allenai2026bar,
  title  = {BAR: Beam-and-Adjust-Routing models},
  author = {{Allen Institute for AI}},
  year   = {2026},
  month  = {April},
  url    = {https://huggingface.co/allenai/BAR-2x7B-Tool-Use}
}

Downloads last month: 29

GGUF

Model size

12B params

Architecture

flex_olmo

Hardware compatibility

4-bit

Model tree for RhinoWithAcape/BAR-2x7B-Tool-Use-GGUF

Base model

allenai/BAR-2x7B-Tool-Use

Quantized

(1)

this model