Contrastive-LM/CLM-v0.1-8B

CLM-v0.1-8B

| 📄 Blog | 💻 Code | 🗣️ Discord |

Contrastive Language Model (CLM) is a new class of System One model trained with a contrastive learning objective that connects states and actions. CLM-8B consists of two small projection heads (a state head and an action head) on top of a frozen Qwen3-8B encoder trained with a bidirectional InfoNCE loss.

• Training: pre-trained on ~60M Nemotron Q&A pairs, mid-trained on ~30M synthetic hard negatives, post-trained on ~1M agentic trajectories.
• Zero-shot: on par with Jev on computer-use, gaming and tool-calling tasks, with up to 9× lower latency.
• Fine-tuned as a verifier: SOTA on DeepSWE (81.6%) and Terminal-Bench 2.1 (87.6%), 4–6× faster than Jev.
• State & Action Caching: states and actions are encoded separately, so action embeddings can be reused. With ~1k candidates, CLM is 13× faster than Jev.

Usage

With the clm package

git clone https://github.com/Contrastive-LM/CLM.git && cd CLM
pip install -r requirements.txt

# 1. encoder: Qwen3-8B, last-token pooling
vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling \
     --enable-prefix-caching --max-model-len 2048 --gpu-memory-utilization 0.35 --port 8090

# 2. API + playground at http://localhost:8700/ (fetches CLM_v0.1-8B.pt into ~/.cache/clm/)
clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings

Ask typed questions about a state:

from clm import CLMClient, Choice, Noul, Score

client = CLMClient()  # http://127.0.0.1:8700 by default
r = client.system_one(
    state="Customer: my invoice was charged twice and nobody answers the phone!",
    questions={
        "urgency": Noul(instructions="Is this urgent?"),
        "department": Choice(instructions="Which team should handle this?",
                             criteria={"billing": "Charges, invoices, refunds",
                                       "technical": "Bugs and outages"}),
        "frustration": Score(instructions="How frustrated is the customer?",
                             criteria=["Calm", "Frustrated", "Very angry"]),
    },
)
print(r.answers["department"].choice)         # billing
print(r.answers["department"].probabilities)  # {'billing': 0.93878, 'technical': 0.06122}

Or rank free-form candidates (best-of-N solutions, tool names, next moves):

from clm import Engine

engine = Engine(emb_url="http://127.0.0.1:8090/v1/embeddings")
engine.rank("What causes tides on Earth?",
            ["The Moon's gravitational pull.", "Photosynthesis in plants.", "Because the Earth is round."])
# [{'rank': 1, 'candidate': "The Moon's gravitational pull.", 'prob': 0.993}, ...]

Fine-tuning

Only the heads are trained, so fine-tuning is cheap. This checkpoint is the starting point for the DeepSWE and Terminal-Bench heads.

hf download Contrastive-LM/CLM-v0.1-8B CLM_v0.1-8B.pt --local-dir ckpts
python train/finetune.py --task clm --hf-dataset Contrastive-LM/deepswe-clm-train-embeddings-8k \
    --init-ckpt ckpts/CLM_v0.1-8B.pt --out-dir runs/deepswe \
    --holdout-tasks heads/deepswe/heldout_tasks.json --batch 512 --seed 1234

See the fine-tuning guide.

Playground

clm-serve also serves a web playground at http://localhost:8700/.

Limitations

• Encoder-locked: the heads require Qwen3-8B last-token-pooled embeddings.
• No generation: CLM only scores the candidates you give it, and its probabilities are relative to that set.
• Verifier results need fine-tuning: the SOTA agentic-benchmark numbers come from fine-tuned heads, not this checkpoint zero-shot.
• Generalization: CLM-8B is one rung of our scaling ladder. A multimodal CLM-35B, trained with more data, compute and parameters for stronger generalization, is coming in early October.

Citation

@misc{kwok2026contrastivelanguagemodels,
  title={Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making},
  author={Jacky Kwok and Hangoo Kang and Tarun Suresh and Jon Saad-Falcon and Marco Pavone and Christopher Ré and Azalia Mirhoseini},
  year={2026},
  note={Notion Blog},
  url={https://contrastive-lm.notion.site}
}