aimlcompanion/llm-training-lab

LLM Training Lab checkpoints

Checkpoints and files for the LLM Training Lab track on AI-ML Companion. Each folder belongs to one lab page. Learners use them to "start from our checkpoint" when a full training run would take too long on free hardware.

lltTinyStories

Used by the lab "Your First Real Pretraining Run".

File	What it is
step2000.pt	Our seed-0 baseline GPT (11,474,240 parameters: width 320, 8 layers, 8 heads, tied output layer) after 2,000 of 3,000 training steps on TinyStories, trained on Kaggle T4 GPUs in fp16 with a GradScaler. A PyTorch dict with model (the state dict), step and cfg.
tokenizer.json	The byte-level BPE tokenizer trained in the same lab, needed to read and write text with this model.

Load it the safe way, as the lab notebook does:

import torch
ckpt = torch.load("step2000.pt", map_location="cpu", weights_only=True)
model.load_state_dict(ckpt["model"])   # model built from the lab's GPT code with ckpt["cfg"]

The lab then trains steps 2,000 to 3,000 on your own GPU session and compares your result with ours.

lltDomainAdaptation

Used by the lab "Adapt a Model to Hindi with Continued Pretraining, Fine-Tuning or RAG".

Folder cpt-r5-lr3e-5/ holds Qwen/Qwen3-0.6B-Base after full-parameter continued pretraining on Hindi web text, the run the lab page reports as its main result:

Setting	Value
Training text	FineWeb2 Hindi (hin_Deva), with 5% of the rows English from FineWeb-Edu (replay)
Steps	320, about 2.6M tokens
Peak learning rate	3e-5
Hardware	One Kaggle T4, fp16
Files	model.safetensors (fp16, 596,049,920 parameters), the config files and the unchanged Qwen3 tokenizer

On the lab's held-out texts, Hindi bits per byte fell 2.1% and English bits per byte rose 2.7% against the base model. The lab page has the full panel and the runs it compares.

Load it as the lab notebook does:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo, sub = "aimlcompanion/llm-training-lab", "lltDomainAdaptation/cpt-r5-lr3e-5"
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, dtype="float16")

lltQLoRA

Used by the lab "Fine-Tune a 9B Model on a Free GPU With QLoRA".

Folder qlora-9b-final/ holds a LoRA adapter for Qwen/Qwen3.5-9B-Base, trained with QLoRA on one Kaggle T4:

Setting	Value
Adapter	rank 16 on all 248 linear layers, 43,278,336 parameters
Base during training	NF4 with double quantization, fp16 compute
Data	3,000 PolyAI/banking77 training messages as prompt and intent-name pairs
Training	1 epoch, 188 steps, batch 16, learning rate 2e-4
Files	adapter_model.safetensors, adapter_config.json (the base model is not included)

On the 3,080 Banking77 test messages, exact-match intent accuracy was 89.09%. The base model, with the intent list in its prompt, scored 29.6% on the first 500 test messages.

Load it on a 4-bit base as the lab notebook does:

model.load_adapter("aimlcompanion/llm-training-lab", adapter_name="ours", subfolder="lltQLoRA/qlora-9b-final")

lltRewardModels

Used by the lab "Train a Reward Model, Then Measure Where It Breaks".

Folder rm-s0/ holds a Bradley-Terry reward model: HuggingFaceTB/SmolLM2-360M with a one-number head (LlamaForSequenceClassification, num_labels=1), every weight trained on one Kaggle T4:

Setting	Value
Training data	3,000 pairs of HuggingFaceH4/ultrafeedback_binarized train_prefs (shuffle seed 0, chats of at most 1,024 tokens)
Training	1 epoch, 375 steps of 8 pairs, AdamW learning rate 1e-5, 3% warmup then cosine, fp16 autocast, seed 0
Input	The SmolLM2-360M-Instruct chat template (saved in the folder) on a user turn and an assistant turn, right padding with `<
Files	model.safetensors (fp16), the config, the tokenizer and the chat template

On the first 1,000 test_prefs pairs that fit 1,024 tokens, pairwise accuracy is 0.628. It is 0.723 when the chosen reply is longer and 0.496 when it is shorter, so it leans heavily on reply length.

File bon_pool.json holds the lab's best-of-n pool: 64 held-out UltraFeedback prompts, 64 replies each sampled from SmolLM2-360M-Instruct (fp32, temperature 1, up to 256 new tokens), and the score of every reply from Skywork/Skywork-Reward-V2-Qwen3-1.7B in its own chat format.

Load them as the lab notebook does:

from transformers import AutoModelForSequenceClassification
from huggingface_hub import hf_hub_download
rm = AutoModelForSequenceClassification.from_pretrained("aimlcompanion/llm-training-lab", subfolder="lltRewardModels/rm-s0")
pool_file = hf_hub_download("aimlcompanion/llm-training-lab", "lltRewardModels/bon_pool.json")

Data and licences

• Training data: roneneldan/TinyStories (CDLA-Sharing-1.0).
• lltDomainAdaptation training data: HuggingFaceFW/fineweb-2 and HuggingFaceFW/fineweb-edu (ODC-By 1.0). Base model: Qwen3-0.6B-Base (Apache-2.0).
• lltQLoRA training data: PolyAI/banking77 (CC-BY-4.0). Base model: Qwen3.5-9B-Base (Apache-2.0).
• lltRewardModels training data: HuggingFaceH4/ultrafeedback_binarized (MIT; its preference labels come from GPT-4, so OpenAI's terms on model outputs may apply). Base model: SmolLM2-360M (Apache-2.0). Pool replies: SmolLM2-360M-Instruct (Apache-2.0); gold scores: Skywork-Reward-V2-Qwen3-1.7B (Apache-2.0).
• These checkpoints: Apache-2.0.
• These are teaching models. The TinyStories model writes simple children's stories, the Hindi model is a 0.6B base model, not an assistant, the QLoRA adapter only classifies banking intents, and the reward model is a small, length-biased teaching model. They are not meant for any other use.