Tuned Lenses

11 tuned lenses trained across 4 models and 3 corpora.

Lenses

Subdirectory	Base model	Training corpus
tuned-lens--olmo-3-1025-7b--dolmino	OLMo-3-1025-7B	Dolmino (OLMo-3 pretraining mix)
tuned-lens--olmo-3-1025-7b--fineweb-edu	OLMo-3-1025-7B	FineWeb-Edu
tuned-lens--olmo-3-1025-7b--pile-val	OLMo-3-1025-7B	The Pile (validation split)
tuned-lens--mistral-7b-v0.3--slimpajama	Mistral-7B-v0.3	SlimPajama-6B
tuned-lens--mistral-7b-v0.3--fineweb-edu	Mistral-7B-v0.3	FineWeb-Edu
tuned-lens--mistral-7b-v0.3--pile-val	Mistral-7B-v0.3	The Pile (validation split)
tuned-lens--gemma-2-9b--fineweb-code-math	Gemma-2-9B	FineWeb-Edu + Code + Math blend
tuned-lens--gemma-2-9b--fineweb-edu	Gemma-2-9B	FineWeb-Edu
tuned-lens--gemma-2-9b--pile-val	Gemma-2-9B	The Pile (validation split)
tuned-lens--qwen3-8b--fineweb-edu	Qwen3-8B	FineWeb-Edu
tuned-lens--qwen3-8b--pile-val	Qwen3-8B	The Pile (validation split)

Each subdirectory contains config.json and params.pt.

Usage

from tuned_lens import TunedLens
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.3")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.3")

lens = TunedLens.from_model_and_pretrained(
    model,
    "chutommy/tuned-lenses",
    subfolder="tuned-lens--mistral-7b-v0.3--fineweb-edu",
)

Models and corpora

Models: Mistral-7B-v0.3, Gemma-2-9B, OLMo-3-1025-7B, Qwen3-8B

Corpora:

FineWeb-Edu — education-filtered web text (HuggingFaceFW/fineweb-edu, sample-10BT)
pile-val — The Pile validation split (EleutherAI/the_pile)
dolmino — OLMo-3's mid-training data mixture (allenai/dolma3_dolmino_mix-100B-1025)
slimpajama — SlimPajama-6B, proxy for Mistral-7B's closed pretraining data (DKYoon/SlimPajama-6B)
fineweb-code-math — 70/20/10 blend of FineWeb-Edu, github-code-clean, and OpenWebMath, approximating Gemma-2's pretraining distribution