rl-test

Checkpoints from an ongoing, experimental reinforcement-learning run on a Qwen3.5-9B model that writes grounded prompts for short commercial videos. A new iteration is added about every 1.5 to 3.5 hours. Quality has not been evaluated, so treat these as work-in-progress.

What is in this repository

Path	Contents	Use it for
hf/iter_XXXXXXX/	Hugging Face weights, config, tokenizer and processor (about 18 GB)	Inference
iter_XXXXXXX/	Full Megatron torch_dist checkpoint with optimizer state (about 136 GB)	Resuming training
rollout/global_dataset_state_dict_N.pt	Data-loader position for iteration N	Resuming training
latest_checkpointed_iteration.txt	Number of the newest full checkpoint	Resuming training

Iterations are never overwritten. For inference you only need one hf/iter_XXXXXXX/ folder. Its config, tokenizer, chat template and processor files are identical to the supervised (SFT) checkpoint the run started from.

Inference settings

Use the same settings as training. Other settings can change output quality.

Setting	Value	Note
System prompt	Required: system_prompt.txt, then the date (YYYY-MM-DD) on its own last line	Not in this repository; it is rl_code/prompts/system_prompt.txt in the private project dataset. The tool definitions are inside this text.
tools= argument	Do not pass it
Thinking	Leave on	Do not pass enable_thinking=False.
Sampling	do_sample=True, temperature=0.7, top_p=1.0, top_k=0	top_k=0 disables top-k; training did not use it.
max_new_tokens	6144 per turn
Total context	65536 tokens
dtype	bfloat16
Images	Downscale only if larger than 4,194,304 pixels (4096 image tokens), keeping aspect ratio
Tool loop	At most 8 tool calls and 2 image_search calls	Each turn, act on the first <tool_call> or <answer>. Return tool results as a user message wrapped in <tool_response>. Put each image right after its IMG_###: label, with at most 5 images per search.

The stop tokens come from the included generation_config.json. The loop itself is implemented as VideoSearcherEnv in rl_code/video_searcher_rl.py in the private project dataset; use that code if you need the full agent, including its tools and final-message handling.

First turn example

Requires transformers>=5.12, torch, accelerate and huggingface_hub. Put system_prompt.txt in the working directory and set ITER to any folder listed under hf/.

import time
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForImageTextToText, AutoTokenizer

ITER = "iter_0000036"
BRIEF = "Your video brief here."

root = snapshot_download("KL-PHD/rl-test", allow_patterns=[f"hf/{ITER}/*"])
path = f"{root}/hf/{ITER}"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForImageTextToText.from_pretrained(path, dtype=torch.bfloat16, device_map="auto").eval()

system = open("system_prompt.txt").read().rstrip("\n") + "\n" + time.strftime("%Y-%m-%d")
messages = [{"role": "system", "content": system}, {"role": "user", "content": BRIEF}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=6144, do_sample=True, temperature=0.7, top_p=1.0, top_k=0)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

Resuming training

Download one iter_XXXXXXX/ folder and its rollout/global_dataset_state_dict_N.pt into a directory, write N to latest_checkpointed_iteration.txt there, and pass that directory as --load. This needs slime at commit 540ea91 and 4 GPUs (saved with tensor-parallel size 2).