Title: Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
Benchmark: GRUE - General Reinforced-language Understanding Evaluation
Summary: The paper investigates the viability of reinforcement learning for language model alignment with human preferences. It introduces the RL4LMs library, the GRUE benchmark for RL evaluation on NLP tasks, and the NLPO algorithm, which improves stability and performance in LM training compared to previous methods like PPO
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: alignment, NLP, LLM as a Judge, reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Aligning pre-trained large language models with human preferences through reinforcement learning methods.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: As language generation problems where the model is given a language input (prompt) and needs to produce a target string, evaluated by reward functions rather than supervised target strings.
Task item definition: Language input (task-specific prompt) and a corresponding target string or reference used for reward calculation.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size detail: The sizes of the train and validation splits vary depending on the specific task within the GRUE benchmark. For instance, IMDB has 25k training and 5k validation examples, while CNN/Daily Mail has 287k training and 13k validation examples.
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: Mean and variance, standard deviations
Metric subscores: Yes
Metric subscores detail: Subscores for different aspects like fluency, sentiment, and task-specific metrics (e.g., BLEU, METEOR)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors compare the trends observed with automated metrics to human judgments and find a general correlation when the generated text is above a certain naturalness threshold. They also acknowledge instances where human feedback suggests potential reward hacking not detected by automated metrics.
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment