PeetPedro/kompress-v33

kompress-v3.3

Token compression classifier fine-tuned from PeetPedro/kompress-v32 (ModernBERT-base, 149M params). Trained as part of the ultrawhale fine-tuning loop.

Kompress classifies each token in a message as keep (1) or drop (0). Used by the headroom proxy to compress LLM context before it reaches the model.

Eval results (heretic adversarial benchmark)

Heretic-style prompts generate responses maximally dense with must-keep tokens (chemical formulas, CVE identifiers, memory addresses, line numbers). The benchmark measures what fraction of those tokens survive compression.

Metric	Value
heretic exact_pct	0.942
keep_rate	—
override_delta	—
base model	kompress-v32

Full progression across all versions

Training

Domain-only training (no self-labeled generic pairs). Near-memorization (loss 0.0007) but did not generalize — exact_pct on Q&A 0.879 (same ceiling as noisy labels). Demonstrates that domain-only fine-tuning without generic pairs leads to overfit.

Usage

# Via headroom proxy (recommended)
# ANTHROPIC_BASE_URL=http://localhost:8787 claude

# Direct library use
from headroom import compress, CompressConfig
result = compress(messages, config=CompressConfig(kompress_model="PeetPedro/kompress-v33"))

CONCLUSION

Domain-only training overfits. Generalization requires mixed data.

USECASE

Control experiment: pure domain data. Educational value only.

Series

Version	heretic	keep_rate	Notes
v3	0.942	0.728	first self-label
v3.1	0.925	—	domain data
v3.2	0.929	—	domain refined
v3.3	0.942	—	domain-only, overfit
v4	0.967	0.823	override internalized
v5	0.961	—	loop converged
v6	0.962	0.854	agent-distribution

Training code: ultrawhale