cuntinum/Kinetic-Dense-76B

Kinetic Dense 76B

76 billion parameters. Fully dense. Every parameter fires on every token.

Website | Documentation | Discord | GitHub

---

Introduction

Kinetic Dense 76B is a large-scale fully dense language model developed by Cuntinum. Unlike mixture-of-experts architectures where only a fraction of parameters are active per token, Kinetic Dense activates all 76 billion parameters for every single token — maximizing reasoning depth per forward pass.

The model combines two attention mechanisms across 192 layers:

• DeltaNet recurrence (144 layers) — O(1) memory per token, enabling efficient long-context processing without KV cache explosion
• Grouped Query Attention (48 layers) — full bidirectional attention for global information mixing at regular intervals

This hybrid design achieves the long-context efficiency of recurrent models while retaining the pattern-matching strength of attention-based architectures.

---

Model Details

Developer	Cuntinum Inc.
Parameters	76B (all active)
Architecture	DeltaNet + GQA Hybrid (192 layers)
Hidden dimension	5,120
Intermediate dimension	17,408
Attention	24 query heads, 4 KV heads, head dim 256
Context length	262,144 tokens
Vocabulary	248,320 tokens
Precision	bfloat16
License	Apache 2.0

---

Architecture

Input Token
    │
    ▼
Embedding (248,320 × 5,120)
    │
    ├── × 144 DeltaNet Layers ─── O(1) recurrent state, no KV cache
    │        └── Fixed 786 KB state per layer regardless of context
    │
    ├── × 48  GQA Layers ──────── Every 4th layer, full attention
    │        └── Compressed KV cache (4 heads), RoPE positional encoding
    │
    ├── MLP (SiLU gate): 5,120 → 17,408 → 5,120
    │
    ▼
RMSNorm → LM Head → Logits (248,320)
    │
    ├── MTP Head ×3 ──── Multi-token prediction (speculative draft)
    │
    ▼
Output Token(s)

Why DeltaNet + GQA?

Property	DeltaNet Layers (75%)	GQA Layers (25%)
Memory per token	O(1) — fixed state	O(n) — KV cache grows
Context scaling	Free (state is constant)	Linear with context
Strength	Sequential reasoning, state tracking	Global pattern matching
Role	Efficient bulk processing	Periodic full-attention mixing

The result: 75% of the model processes 262K context with constant memory, while 25% provides the full-attention capability when the model needs to attend to distant tokens.

---

Performance

Capabilities

• Code generation — Strong across Python, TypeScript, Rust, Go, C++
• Reasoning — Multi-step mathematical and logical reasoning
• Long context — 262K native window with efficient DeltaNet processing
• Vision — Built-in image understanding and OCR
• Instruction following — Aligned for conversational and task completion

Benchmarks

We encourage the community to benchmark this model independently. Compatible with:

• Open LLM Leaderboard
• lm-evaluation-harness
• bigcode-evaluation-harness

Results will be updated here as they become available.

---

Quickstart

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "cuntinum/Kinetic-Dense-76B"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Write a Python function that finds all prime factors of a number."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.05,
    )

response = tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

vLLM (Recommended for Production)

# Full precision, 4-GPU tensor parallel
vllm serve cuntinum/Kinetic-Dense-76B \
    --tensor-parallel-size 4 \
    --dtype bfloat16 \
    --max-model-len 262144 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --enable-prefix-caching

# Single GPU with 4-bit quantization
vllm serve cuntinum/Kinetic-Dense-76B \
    --quantization bitsandbytes \
    --load-format bitsandbytes \
    --max-model-len 131072 \
    --trust-remote-code

OpenAI-Compatible API

Once served with vLLM, use any OpenAI-compatible client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="cuntinum/Kinetic-Dense-76B",
    messages=[{"role": "user", "content": "Explain DeltaNet attention in simple terms."}],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)

---

Hardware Requirements

Configuration	VRAM Required	Throughput
4× H100 80GB (bf16, TP=4)	~152 GB	~50-70 tok/s
2× H100 80GB (bf16, TP=2)	~152 GB	~30-40 tok/s
8× A100 80GB (bf16, TP=8)	~152 GB	~40-60 tok/s
1× H100 80GB (NF4)	~45 GB	~15-20 tok/s
4× A6000 48GB (bf16, TP=4)	~152 GB	~25-35 tok/s

Minimum: 1× GPU with 48+ GB VRAM (with 4-bit quantization) Recommended: 4× H100 80GB for full bf16 serving

---

Inference Recommendations

Parameter	Recommended	Notes
Temperature	0.6	Balanced creativity/coherence
Top-P	0.9	Standard nucleus sampling
Top-K	50	Optional, use with top-p
Repetition penalty	1.05	Prevents loops
Max tokens	4096+	Model handles long outputs well
Reasoning effort	Low	Best quality at low reasoning overhead

---

Chat Template

The model uses a standard chat template:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Reasoning is supported via <think>...</think> tags (hidden from user by default).

---

Intended Use

Recommended:

• Software engineering and code generation
• Complex multi-step reasoning
• Long document analysis and summarization
• Research and development
• Building AI-powered applications

Not recommended:

• Safety-critical applications without additional guardrails
• Generation of harmful content
• Any use that violates applicable laws

---

Training

Kinetic Dense 76B was trained end-to-end by Cuntinum using a proprietary multi-stage pipeline on NVIDIA H100 80GB and AWS p4d A100 80GB clusters. The training process includes:

1. Pre-training — Large-scale language modeling on curated data
2. Architecture scaling — Proprietary method to reach 192 layers at 76B parameters
3. Supervised fine-tuning — High-quality reasoning traces and instruction data
4. Direct preference optimization — Alignment with human preferences
5. Knowledge consolidation — Final stage training for coherent long-form generation

---

Limitations

• Primarily trained on English data; other languages may have reduced quality
• Safety aligned through direct preference optimization — additional application-specific guardrails may still be appropriate
• DeltaNet layers process sequentially (recurrent) which affects prefill speed on very long prompts
• Like all LLMs, may produce incorrect or fabricated information

---

License

This model is released under the Apache 2.0 License.

Free for commercial and research use.

---

Citation

@misc{kinetic-dense-76b,
  title={Kinetic Dense 76B: A Fully Dense Language Model with DeltaNet Recurrence},
  author={Igwiloh, Nnaa},
  year={2026},
  publisher={Cuntinum Inc.},
  howpublished={\url{https://huggingface.co/cuntinum/Kinetic-Dense-76B}}
}

---

Contact

• Organization: Cuntinum
• Discord: discord.gg/CcXcU7qex
• Issues: GitHub Issues