Danau5tin/ai-trains-ai-trainer

ai-trains-ai trainer agent (LoRA adapter, step 34)

An RL-trained agent whose job is to write RL training jobs for smaller models. This is the LoRA adapter from the ai-trains-ai project — the trainer agent that sits in the outer loop of two nested RL loops.

Derived from Qwen/Qwen3.6-35B-A3B (Apache-2.0). This repository contains only the LoRA adapter weights (rank 8, alpha 32); you need the base model to use it.

What this agent does

Given a task spec ("teach a small model to do X"), the agent works in a sandboxed workspace with file tools and writes a complete prime-rl training job: a verifiers environment + rubric, a dataset, and a hyperparameter config. Each job it submits is validated, then GRPO-trains a small Qwen (0.6B or 1.7B) on real GPUs; the trained model's improvement on a hidden eval flows back as the agent's reward.

It is designed to operate inside the project's harness (workspace layout, file tools, validation probe, task-spec format) — see the GitHub repo to run full episodes. Sampled bare, it is still Qwen3.6-35B-A3B plus a strong prior for writing verifiers environments and prime-rl configs.

Which checkpoint this is

This is the step-34 checkpoint of the 54-step pilot-7 arc (end of the pilot-7b leg) — the peak of held-out generalisation:

• On a held-out task family (incident triage — never trained on), mean reward went 0.399 (untrained base) → 0.545 at this checkpoint (n=10 per arm, so noisy; by step 54 it had eased back to 0.49).
• Over the arc, episode reward climbed from ~0.0 to a ~0.63 peak, with ~1,750 real GPU training jobs behind it.
• Behavioural shifts: the agent learned to stop picking the weaker 0.6B base model (1.7B share of jobs 42% → 95% across the arc) and started genuinely using the hyperparameter config surface (21% → ~78% of episodes).

Training details

Base model	Qwen/Qwen3.6-35B-A3B
Method	LoRA (rank 8, alpha 32) + importance-sampling GRPO, async off-policy
Stack	Tinker + tinker-cookbook
Steps	34 outer-loop steps (pilot-7 → pilot-7b, warm-started)
Reward	validation efficiency + job quality (absolute post-training score + uplift over the best untrained baseline) + a small train-speed tie-breaker
Sampling during training	temperature 1.0, Qwen3.5 renderer / chat template

Full write-ups — including the pilots that failed — are in the repo's retros.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-35B-A3B", torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base, "Danau5tin/ai-trains-ai-trainer")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-35B-A3B")

Or serve with vLLM LoRA support (--enable-lora --lora-modules trainer=<path>). For real episodes, drive it through the harness in the GitHub repo.

Honesty notes

• The holdout numbers are n=10 per arm — directionally solid (a rise then a plateau/dip), but individual means are noisy.
• The "hidden" evals the reward was computed against are now published in the GitHub repo (evals/), so post-release comparisons of new models against this adapter on those files are contaminated by definition.
• This adapter was trained through Tinker's managed API; the inner loop (the jobs the agent writes) is all open-source stack on rented GPUs.

License

The adapter weights are released under Apache-2.0 (matching the base model). The harness code in the GitHub repo is MIT.