Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via
  Past-Future
Authors: name: Yidong Wang, name: Xin Wang, name: Cunxiang Wang, name: Junfeng Fang, name: Qiufeng Wang, name: Jianing Chu, name: Xuran Meng, name: Shuxun Yang, name: Libo Qin, name: Yue Zhang, name: Wei Ye, name: Shikun Zhang

Self-Rewarding Language Models propose an architecture in which the Large
Language Models(LLMs) both generates responses and evaluates its own outputs
via LLM-as-a-Judge prompting, dynamically improving its generative capabilities
through iterative Direct Preference Optimization (DPO). However, our analysis
reveals a critical limitation in existing Self-Rewarding paradigms: the
synchronized improvement of chosen and rejected responses progressively narrows
the representational difference between contrasting samples, undermining
effective preference learning. We propose Temporal Self-Rewarding
Language Models that strategically coordinate past, present, and future model
generations to sustain learning signals. Our dual-phase framework introduces:
(1) Anchored Rejection - fixing rejected responses using the past
initial model's outputs and (2) Future-Guided Chosen - dynamically
curating chosen samples using next-generation model predictions. Extensive
experiments across three model families (Llama, Qwen, Mistral) and different
model sizes (Llama3B/8B/70B) demonstrate significant improvements when trained
with our method compared to Self-Rewarding using same computation resources.
For example, Llama3.1-8B reaches a 29.44 win rate on AlpacaEval 2.0 with our
method, outperforming the Self-Rewarding baseline (19.69) by 9.75. Notably, our
method also demonstrates superior out-of-distribution generalization across
mathematical reasoning (GSM8K), knowledge-based QA (ARC, TruthfulQA), and code
generation (HumanEval) tasks, even though we do not specifically collect such
training data.