Rendred/vgc-critique-qlora

VGC Team-Critique QLoRA (Qwen2.5-7B)

A LoRA adapter that turns Qwen2.5-7B-Instruct into a competitive Pokémon (VGC) team critic. Given a team, it names concrete weaknesses — coverage gaps, missing speed control, redundant typings, dead abilities, and threats the team can't answer — in the style of a Smogon "Rate My Team" reply.

Format scope: gen9vgc2026regi (Showdown ladder). The metagame shifts each regulation, so judgments are calibrated to this format only.

Results

Evaluated on 310 held-out tournament teams (Regulation SVI, never seen in training — training data is Showdown ladder replays). Half kept clean, half given ONE known injected flaw so "did it find the flaw?" is checkable without human grading.

metric	base Qwen2.5-7B	+ this adapter
groundedness (claims that verify)	88.3%	93.8%
flaw-detection recall	70.8%	81.2%
unsupported claims about Top-8 teams	12.6%	5.1%
points per critique	10.3	5.9

Recall by injected-flaw type: speed-control 34/34, redundant-typing 90/90, no-threat-answer 74/74, removed-Fake-Out 5/52. The fourth is a known label-generation issue (inconsistent severity ranking), not a model limit.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

BASE = "Qwen/Qwen2.5-7B-Instruct"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb,
                                             device_map="auto")
model = PeftModel.from_pretrained(model, "Rendred/vgc-critique-qlora")   # <- this adapter

team = """- Incineroar @ Assault Vest (Ability: Intimidate) | Moves: Fake Out, Knock Off, Flare Blitz, U-turn
- ..."""
prompt = ("You are an experienced competitive Pokemon player writing a "
          "'Rate My Team' critique. Identify the team's real weaknesses and "
          f"explain why each matters. Be specific.\n\nTeam:\n{team}")
text = tok.apply_chat_template([{"role":"user","content":prompt}],
                               tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=400, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Training

QLoRA (4-bit NF4), r=16, α=32, on q/k/v/o + gate/up/down. 250 steps, effective batch 16, lr 2e-4 cosine, on a single T4. Trained on 6,377 grounded (team, critique) pairs (1,775 real + 4,602 synthetic flaw-injected). Every training label is verified against the type chart, base stats, and Smogon usage stats — no ungrounded matchup claims.

Recommended: gate the output. Even at 93.8% groundedness, a small fraction of generated claims are unsupported. The source project ships a verify.py that drops any claim the computed facts contradict before display.

Source, dataset pipeline, and eval harness: see the project repository.