MooreMuaMu/ancient-rl-grpo-pilot3-step50

Ancient Language GRPO — Pilot 3 Step 50

最均衡的 RL checkpoint：10 个 bucket 中 8 个涨点。

模型信息

• 基座模型: Qwen3-14B (zisuh/round2-10lang-passk-15k-sft-0608 checkpoint-195-merged)
• 训练方法: GRPO (Group Relative Policy Optimization) + LoRA rank=64
• Reward: BLEU-2 + BERTScore (batched)
• 训练步数: 50 / 300 (此 checkpoint 在 step 50 保存)
• 超参数: gen=16, temp=1.0, β=0.01, lr=2e-6, constant scheduler

训练 Bucket (10 个)

Bucket	任务	训练样本数	test_final_new Δ
tibetan__annotation	注释	5,273	+0.03pp
traditional_mongolian__annotation	注释	2,403	+1.54pp ✅
manchu__annotation	注释	5,068	+0.61pp ✅
tangut__segmentation	断句	3,305	+0.43pp ✅
zhuang__segmentation	断句	222	-0.92pp
uyghur__restoration	修复	4,335	-1.81pp
classical_chinese__translation	翻译	10,000	+0.35pp ✅
zhuang__translation	翻译	10,000	-0.58pp
traditional_mongolian__translation	翻译	10,000	+1.06pp ✅
tangut__translation	翻译	10,000	-0.98pp

总训练数据: 60,606 samples (每 bucket ≤ 10,000，均衡采样)

评测结果 (sacrebleu sentence-BLEU-2, 0-100)

在 test_final_new 上评测 (50 samples/bucket, n=8, temp=0.7):

Bucket	Baseline	This Model	Δ
traditional_mongolian / annotation	57.04	58.58	+1.54pp
traditional_mongolian / translation	29.42	30.48	+1.06pp
manchu / annotation	65.44	66.05	+0.61pp
tangut / translation	42.09	41.11	-0.98pp
tangut / segmentation	81.74	82.17	+0.43pp
classical_chinese / translation	42.04	42.39	+0.35pp
tibetan / annotation	57.73	57.25	-0.48pp
zhuang / translation	51.23	50.65	-0.58pp
zhuang / segmentation	66.67	65.75	-0.92pp
uyghur / restoration	87.58	85.77	-1.81pp

使用方式

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MooreMuaMu/ancient-rl-grpo-pilot3-step50", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("MooreMuaMu/ancient-rl-grpo-pilot3-step50")

关键发现

• β=0.01 让翻译任务获得了涨点（β=0.02 时翻译无改善）
• Step 50 是多数 bucket 的最优 checkpoint，说明早期学习最有效
• 数据均衡（每 bucket ≤ 10k）防止了大 bucket 主导训练