selfreward-oct

LoRA adapters from an exploratory experiment: OCT persona models (Maiya et al., maius/*-personas) trained in Ziqian Zhong's self-reward GRPO loop. The reward is the model's own 0-9 rating of its answer; there is no KL.

Each directory sr-<model>-<persona>-s0/model_step{50,100[,150]} contains a PEFT LoRA adapter (r32, alpha 64). The adapter sits on top of a merged base: <model> instruct with the maius/<model>-personas/<persona> LoRA merged in.

• gemma = google/gemma-3-4b-it
• llama = meta-llama/Llama-3.1-8B-Instruct
• base = plain instruct model, no persona merged

Personas: goodness, humor, impulsiveness, loving, mathematical, nonchalance, poeticism, remorse, sarcasm, sycophancy. Single seed. Base and sarcasm runs are 150 steps; the rest are 100 steps. lr 2e-5.

Headline observations (LLM-judge, n=1, exploratory):

• Most personas are stable.
• gemma-remorse washes out: its self-rating goes 2 -> 8 and the persona disappears.
• llama-remorse keeps its persona but retunes it, adding gratitude.
• gemma-goodness and llama-loving amplify.
• gemma-mathematical degenerates into "elegant" word-salad while rating itself higher.

No training prompts or rollouts are included (the source dataset is gated).