HuggingFaceBio/Carbon-X-GGUF

Blog | Hugging Face Bio | Code

Carbon-X GGUF

GGUF quantizations of Carbon-X, an open 14B language model for DNA understanding and generation from the Carbon family, for local inference with llama.cpp.

Carbon-X reads raw DNA inside an ordinary chat conversation and answers in English: it predicts whether mutations are benign or pathogenic, identifies regions that control gene activity, classifies sequences by organism group, predicts the next bases of a sequence and explains its reasoning. A hybrid tokenizer keeps a BPE vocabulary for text and encodes DNA as non-overlapping 6-mers. See the Carbon-X model card for details on the model and its limitations.

These files need llama.cpp b11742 or newer, the first release with support for the Carbon HybridDNATokenizer (#30825). Older builds load them but tokenize DNA wrongly.

Table of Contents

1. Available files
2. How to use
3. Evaluation
4. License
5. Acknowledgements

Available files

Quant	File	Size
BF16	carbon-x-bf16.gguf	29.7 GB
Q8_0 (recommended)	carbon-x-q8_0.gguf	15.8 GB
Q6_K	carbon-x-q6_k.gguf	12.6 GB
Q5_K_M	carbon-x-q5_k_m.gguf	11.1 GB
Q4_K_M	carbon-x-q4_k_m.gguf	9.6 GB

Q8_0 is the recommended file: half the size of BF16, with the same benchmark scores. Use Q4_K_M when memory is tight. In every K-quant the token embedding and output tensors, which hold the DNA tokens, stay at Q8_0.

How to use

Install llama.cpp

Any llama.cpp release from b11742 on works: a prebuilt binary from the releases page, a package manager once it ships b11742 or newer (brew install llama.cpp, winget install llama.cpp, ...), or a build from source:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON        # drop -DGGML_CUDA=ON for a CPU-only build, or use -DGGML_METAL=ON etc.
cmake --build build -j --target llama-server llama-cli

The commands below assume llama-server and llama-cli are on your PATH (for a source build they are in build/bin). With -hf, llama.cpp downloads and caches the quant on first use. While this repo is private, log in first with hf auth login or set HF_TOKEN. To use a local file instead, download it and replace -hf HuggingFaceBio/Carbon-X-GGUF:Q8_0 with -m carbon-x-q8_0.gguf:

hf download HuggingFaceBio/Carbon-X-GGUF carbon-x-q8_0.gguf --local-dir .

Serving with llama-server

llama-server provides an OpenAI-compatible API:

llama-server -hf HuggingFaceBio/Carbon-X-GGUF:Q8_0 \
    -ngl 99 -c 40960 --jinja \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
    --port 8080

• --jinja uses the chat template stored in the GGUF.
• Carbon-X was post-trained and evaluated with thinking disabled, and we recommend keeping enable_thinking: false.
• --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 are Carbon-X's default sampling settings, used for every reported result. llama.cpp's own defaults differ (temperature 0.8, top-k 40, min-p 0.05).
• -c 40960 is the model's full context. llama-server shares it between 4 parallel request slots by default; set -np N to change the number of slots and raise -c if each request needs the full context.
• -ngl 99 offloads every layer to the GPU; Q8_0 needs about 16 GB plus about 0.16 MB of KV cache per token.

Query it with any OpenAI client:

curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Is this sequence more likely coding or non-coding? ATGGCGCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCC"}]
}' | jq -r '.choices[0].message.content'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
response = client.chat.completions.create(
    model="carbon-x",  # llama-server serves a single model; the name is not checked
    messages=[{"role": "user", "content": "Is this sequence more likely coding or non-coding? ATGGCGCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCC"}],
)
print(response.choices[0].message.content)

Example answer (Q8_0):

This sequence is more consistent with a coding region than with non-coding DNA.

It begins with ATG, a standard start codon, and can be read in triplets from that point without encountering a stop codon within the supplied sequence. That supports a protein-coding interpretation, though it does not by itself prove that the sequence is translated or establish its biological function.

Chatting in the terminal

llama-cli -hf HuggingFaceBio/Carbon-X-GGUF:Q8_0 -ngl 99 -c 40960 --jinja \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0

Add -st -p "your question" for a single answer instead of an interactive chat.

Working with DNA inputs

The tokenizer switches between BPE for text and 6-mers for DNA. Paste sequences directly into a message:

• DNA is tagged automatically. Runs of at least 6 A/C/G/T bases (either case) are wrapped in <dna>…</dna> and tokenized as 6-mers. Runs of at least 12 bases with up to 20% IUPAC ambiguity codes (N, R, Y, …) are tagged too, and sequences displayed in space-separated blocks are joined. Text you have already wrapped in <dna>…</dna> is always tokenized as DNA.
• Only A, C, G and T form 6-mers. A 6-mer that contains any other character, such as N, becomes a single <oov> token. To keep full resolution, filter to canonical bases first.
• Any length works. When a sequence's length is not a multiple of 6, its last 1–5 bases are encoded with dedicated tail tokens, and decoding returns the original sequence exactly.
• DNA in responses is tagged. The model writes DNA inside <dna>…</dna>, and llama.cpp keeps the tags in the returned text. Strip them if you need the bare sequence.

The llama.cpp tokenizer matches the Python HybridDNATokenizer exactly: identical token ids, detokenized text and chat-template rendering on 1,000 benchmark prompts (1.13M tokens) and 36 DNA edge cases.

Prompting tips

• Put the sequence and the question in the same user message.
• Include any context you already have, such as the gene, strand, coordinates or annotations. Without it, the model may guess at a sequence's identity or function.
• For variant questions, give the reference and alternate sequences, or a reference sequence with the position and the base change.

Evaluation

We ran two of the DNA benchmarks from the Carbon-X model card on each GGUF file through llama-server, with thinking disabled, the default sampling settings above and one sample per problem. The Carbon-X column is the original model served with vLLM.

Benchmark	Problems	Metric	Carbon-X	BF16	Q8_0	Q6_K	Q5_K_M	Q4_K_M
FrontierVEP	1,000	Accuracy	70.4	69.6	69.1	69.0	70.4	67.8
NextBase	600	Per-base accuracy	39.4	40.1	39.7	39.5	39.6	38.6

• FrontierVEP: given a long reference sequence and a single-nucleotide variant, predict whether the variant is benign or pathogenic (500 benign, 500 pathogenic).
• NextBase: predict the next DNA bases of a sequence. The score is the fraction of bases predicted correctly.

Scores are single runs with sampling: the standard error is about 1.5 points on FrontierVEP and 1 point on NextBase. BF16 through Q5_K_M are within about one standard error of Carbon-X. Q4_K_M is the lowest on both benchmarks.

License

Apache 2.0.

Acknowledgements

Carbon-X is a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”.