Harsha001/elephant-1-ROCmFPX

elephant-1 — ROCmFPX GGUF

baselabs/elephant-1, Base Labs' 9B decision model (post-trained from Qwen3.5-9B), quantized to the ROCmFPX formats for AMD GPUs. Give it a state and a set of questions; it returns a probability for every option of every question from one shared prefill.

These files run with llama-decide, a small tool in HarshaVardhan3002/ROCmFPX that reads the same request format as Base Labs' own runtime. A normal chat server is not the way to use this model.

Start with Q4_0_ROCMFP4_STRIX. It is the closest 4-bit file to the original.

These files only load in a ROCmFPX build of llama.cpp. They have not been tested on AMD hardware yet.

Not an official release of Base Labs or Baseten. All credit for the model goes to Base Labs.

Files and measured quality

Measured against the BF16 model (perplexity 7.5591) on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the next-token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16. These numbers use ordinary left-to-right attention on plain text; they show how much each file changes the weights, not decision accuracy, which was not measured.

File	Size	PPL	KLD	Same top
elephant-1-Q8_0_ROCMFPX.gguf	9.2 GB	7.5678	0.0054	96.47%	8-bit, closest to BF16
elephant-1-Q6_0_ROCMFPX.gguf	7.4 GB	7.5805	0.0067	95.96%	6-bit, almost as close as 8-bit
elephant-1-Q4_0_ROCMFP4_STRIX.gguf	5.1 GB	7.7795	0.0478	88.63%	Recommended. 4-bit recipe tuned for Strix Halo (gfx1151)
elephant-1-Q4_0_ROCMFP4_FAST.gguf	4.8 GB	7.8060	0.0496	88.54%	smallest, speed-first 4-bit (one scale per block)

Not published: Q3_0_ROCMFPX (4.9 GB, PPL 7.9236, KLD 0.0750, same top 86.74%) is larger than Q4_0_ROCMFP4_FAST and further from BF16.

Decisions

The GGUF files carry the decision settings from the original checkpoint (qwen35.decision.*: block-bidirectional attention on the full-attention layers and the 256 answer labels), so llama-decide needs no extra files.

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-decide -j
build/bin/llama-decide -m elephant-1-Q4_0_ROCMFP4_STRIX.gguf -ngl 999 -f request.json

request.json: a state (text or JSON) and questions, as in Base Labs' card:

{
  "state": {
    "task": "Pick up the red mug from the table and place it on the shelf.",
    "observations": ["The gripper closed around the handle of the red mug.",
                     "The arm lifted the mug 12 cm above the table.",
                     "The mug slipped and fell back onto the table, upright and intact."],
    "gripper_force_newton": 4.1
  },
  "questions": {
    "success": {"type": "noul", "instructions": "Was the task completed successfully?"},
    "failure_mode": {"type": "choice", "instructions": "What is the main reason for the outcome?",
      "criteria": {"grasp_slip": "The object slipped out of the gripper.", "collision": "The arm or the object hit something.",
                   "wrong_object": "The robot handled the wrong object.", "no_failure": "Nothing went wrong."}},
    "progress": {"type": "score", "instructions": "How far did the robot get with the task?",
      "criteria": ["did not start", "grasped the object", "lifted the object", "placed the object on the shelf"]}
  }
}

Answer from elephant-1-Q4_0_ROCMFP4_STRIX.gguf (CPU backend): success 0.020; failure_mode grasp_slip 0.996; progress most likely "grasped the object" (0.719), then "lifted the object" (0.223).

type	criteria	answer fields
noul: yes or no	optional	noul: P(yes)
choice: one of named options	{name: description}	choice, probabilities
score: ordered levels	list of levels, lowest first	score, probabilities

Options: --debias N (average over N cyclic shifts of the answer labels), --causal (plain causal attention instead of the trained block-bidirectional mask). llama-decide was checked against Base Labs' PyTorch runtime on this model (probabilities within 0.004).

How these were made

1. convert_hf_to_gguf.py --outtype bf16 from the fork, which also writes the decision settings.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (calibration_datav3), 512-token chunks.
3. llama-quantize --imatrix, one run per type (ROCmFPX fb08d7c merged onto upstream llama.cpp).
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels; each quant was scored with the fork's CPU backend, which runs the ROCmFPX types directly. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

The 8-bit file is not as close to BF16 as a stock Q8_0 usually is: ROCmFPX block scales have a fixed smallest step, so 8-bit blocks of small weights keep fewer than 8 bits (ROCmFPX#118).

License

MIT, same as the base model.