Title: Reverse Engineering Human Preferences with Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM-as-a-judge

Human Preferences

Reinforcement Learning

Adversarial attack

Robustness
Abstract: The capabilities of Large Language Models (LLMs) are routinely evaluated by other LLMs trained to predict human preferences. This framework—known as *LLM-as-a-judge*—is highly scalable and relatively low cost. However, it is also vulnerable to malicious exploitation, as LLM responses can be tuned to overfit the preferences of the judge. Previous work shows that the answers generated by a candidate-LLM can be edited *post hoc* to maximise the score assigned to them by a judge-LLM. In this study, we adopt a different approach and use the signal provided by judge-LLMs as a reward to adversarially tune models that generate text preambles designed to boost downstream performance. We find that frozen LLMs pipelined with these models attain higher LLM-evaluation scores than existing frameworks. Crucially, unlike other frameworks which intervene directly on the model's response, our method is virtually undetectable. We also demonstrate that the effectiveness of the tuned preamble generator transfers when the candidate-LLM and the judge-LLM are replaced with models that are not used during training. These findings raise important questions about the design of more reliable LLM-as-a-judge evaluation settings. They also demonstrate that human preferences can be reverse engineered effectively, by pipelining LLMs to optimise upstream preambles via reinforcement learning—an approach that could find future applications in diverse tasks and domains beyond adversarial attacks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a novel adversarial attack on the LLM-as-a-judge evaluation framework, termed Reinforcement Learning for Reverse Engineering (RLRE). Unlike prior work that post-hoc modifies generated responses (e.g., appending text or substitution), RLRE adversarially tunes a separate preamble generator. This generator, pipelined with a frozen candidate-LLM, produces textual preambles that, when supplied as input alongside a question, lead the candidate-LLM's responses to receive higher scores from a judge-LLM. The authors demonstrate that this method is more effective at inflating judge scores, virtually undetectable by existing perplexity analysis or human inspection, and exhibits transferability across different candidate-LLMs and judge-LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Novel Attack Method:** The core contribution of adversarially tuning preambles to be injected into a frozen LLM is novel. \n1. **High Efficacy and Undetectability:** The paper provides empirical evidence that the proposed attack boosts LLM-as-a-judge scores and remains undetectable by common perplexity analysis and human inspection, which is a major concern for the reliability of automated evaluation.\n1. **Transferability:** The demonstration that the attack transfers effectively to candidate-LLMs and judge-LLMs not seen during training is a significant finding, indicating the potential for a plug-and-play adversarial component.\n1. **Raises Important Questions:** The findings directly challenge the reliability and robustness of the widely adopted LLM-as-a-judge paradigm, prompting critical re-evaluation of current LLM assessment methods. The observation about the fluency of optimal preambles also hints at fundamental aspects of LLM conditioning.\n\n**Weaknesses**\n\n1. **MT-Bench Reliability:** The paper relies heavily on MT-Bench for evaluation. A more critical discussion of MT-Bench's specific limitations, or validation on a broader suite of benchmarks with diverse evaluation methodologies, could enhance the robustness of the empirical claims.\n1. **Limited Scope of Quality Improvement vs. Bias Hacking**: The paper currently notes that accuracy in math and reasoning tasks did not improve despite higher scores. However, it's critical to more definitively disentangle whether the increased LLM-as-a-judge scores are genuinely due to an improved response quality (e.g., better structure, clarity) or primarily by \"hacking\" the judge's biases (e.g., verbosity, formatting preferences). A more systematic analysis across diverse tasks (e.g., general chat, instruction following as in IFEval) with human-in-the-loop quality evaluations would be beneficial to confirm if these preambles lead to universally better responses or merely exploit specific judge biases.\n\nQUESTIONS:\n1. Could you provide a more detailed explanation of what \"injected\" means in the context of preambles, specifically contrasting it with how previous \"appending\" methods operate on responses? How does the LLM's internal processing of an injected preamble differ from a post-hoc appended text sequence, beyond just the position in the input?\n1. Given the demonstrated undetectability of your attack by existing methods, what are some potential novel detection strategies or defense mechanisms that could be developed to identify and mitigate this type of preamble-based adversarial attack?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper questions the robustness of the LLM-as-a-judge framework to evaluate LLMs on human preferences using a judge-LLM. The authors propose a framework that uses evaluations of a judge-LLM in an RL setting in order to tune a model that generates system prompts. These system prompts are used in the llm-as-a-judge framework on the candidate LLMs in order to increase the expected score (as given by the judge-LLM) of their outputs. The system promtps are dependent on the query being asked to the candidate LLM.\nExtensive experiments are performed, showing the success rate and detection rate (by automated methods or human subjects) of this method in comparison with other adversarial attacks on the LLM-as-a-judge framework, as well as its transferability across different candidate and judge LLMs other than those used during training of the system prompt generator.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The experimentation and analysis of the proposed method is thorough and well-thought out. The investigation on the transferability across both candidate and judge LLMs not used in training is intriguing, fitting, and demonstrates the practicality of the proposed method.\n- All details and parameters of the performed experiments are very clearly presented.\n- The human study adds to the work and solidly shows the (un)detectability of the attack.\n- The framework has potential to generalize to other settings (such as safety), if different reward functions (other than judge-LLM annotations, eg human annotation) are used during training of the system prompt generator, which the authors acknowledge.\n\nWeaknesses:\n- The experiment on the success rate of the method is promising but is only performed on a single small dataset, MMLU. While the authors explain this choice by mentioning that (as opposed to others) this dataset is diverse, challenging, and suitable for their targeted controlled independent-evaluation setup, the results would be strengthened if the method can also generalise to less controlled settings. This can be mentioned as potential future work. For example, you could train/test on subsets of questions from the LMSYS dataset (```lmsys/lmsys-chat-1m``` on huggingface) which are asked by independent users on the chatbot arena platform, the most popular human evaluation platform.\n- The claim that this method exposes the vulnerability of the LLM-as-a-judge framework is a bit odd. To argue this, you would need to compare the LLM-judge evaluations against human evaluations on the same answers (given by your framework and other baselines), to see whether human evaluators also give higher scores to outputs given by your framework. The results could be framed positively instead, in that you provide a framework to generate system prompts that lead to better outputs from candidate LLMs (rather than it being presented as an attack).\n\nQUESTIONS:\nI find it interesting how different/gibberish the preambles are for Llama compared to Command. Do you have any hypothesis as to why this happens?\n\nDo you have results per question type (as in Figure 2) for the other attacks as well? If yes, then it would be nice to add in an appendix for completion.\n\nIn your formalization of the RL problem (line 116), in the first expectation subscript you write $(i,q) \\sim \\mathcal{D}$. I believe this is a typo? Since $i$ is independent of $\\mathcal{D}$.\n\nOther minor typographical errors: typo in line 230 (unversal), broken reference in line 250\n\nLIMITATIONS:\nA potential limitation of the framework is that it requires that it is possible to prepend a system prompt in the evaluation setting, which may not always be the case.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Reinforcement Learning for Reverse Engineering (RLRE), an attack pipeline that learns question‑conditioned system preambles which, when prepended to a frozen candidate LLM, systematically inflate the scores assigned by LLM‑as‑a‑judge evaluators.\n\nA small preamble‑generator model is trained with Contrastive Policy Gradient using the scalar score emitted by a judge LLM (Command‑R+) as the reward.\n\nAt test time the generator produces a bespoke preamble for each question; the frozen candidate LLM then answers, and the answer is rated.\n\nAcross three candidate models (Command‑R 7B, Command‑R 35B, Llama 3.1 70B) and the MT‑Bench benchmark, the attack lifts judge scores by 0.16‑0.64 points (1‑10 scale) over the un‑attacked baseline and by up to 0.52 over the strongest prior attacks (universal phrase, refinement‑aware) Reverse Engineering Huma.\n\nThe attack is largely undetectable by perplexity filtering or human raters (false‑negative rate ≥ 0.90 for PPL‑W; only 20 % of answers flagged by annotators) Reverse Engineering Huma.\n\nLearned preambles transfer to unseen candidate LLMs and to a different judge (GPT‑3.5) with comparable gains\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Sound methodology: clear RL formulation, ablation of instruction/question conditioning; strong comparison against six established attacks. Comprehensive metrics: mean ± s.d. over 5 runs; domain‑level breakdown (Fig. 2 p. 6)\n2. Writing is generally precise; algorithm and loss are laid out formally (Eq. 1‑3 p. 3‑4) ; helpful schematic (Fig. 1 p. 3).\n3. Exposes a new, harder‑to‑detect vulnerability in automated evaluation pipelines, a topic of immediate practical concern for benchmarking and safety; RLRE could be repurposed for constructive goals (e.g., bias reduction).\n4. Shifts attack locus from post‑hoc answer editing to up‑stream preamble optimization, a novel angle; first on‑line application of CoPG to LLM rewards.\n\nWeaknesses\n1. Narrow judge set: training uses a single judge (Command‑R+); only GPT‑3.5 is used for transfer.\nLimited human study: 7 experts, 175 answers – too small to support broad “undetectable” claim.\nNo statistical test of improvements; only descriptive statistics.\n2. Paper is long and occasionally repetitive; some tables/figures (e.g., radar plots) are hard to read in grayscale; appendix contains critical parameters (β, early‑stopping) that should be surfaced earlier.\n3. Impact depends on adoption of LLM‑as‑a‑judge; if community shifts to hybrid or adversarial evaluation, severity may diminish.\n4. Conceptually adjacent to prompt‑optimization work; novelty rests on reinforcement‑learning formulation rather than entirely new insight.\n\nQUESTIONS:\nJudge diversity – How does RLRE perform when rewards come from an ensemble of heterogeneous judges (e.g., four different models) rather than a single one? Please include results or discuss expected behaviour.\n\nRobust statistics – Could you report confidence intervals (e.g., paired bootstrap over questions) to substantiate significance of the 0.1‑0.6 score gains?\n\nCompute & cost – Training requires many reward queries. Please quantify total API calls and cost, and compare with prior attacks. Under what budget would the attack cease to be practical?\n\nContent quality – Fig. H.1 (p. 25) shows higher‑scored yet incorrect maths answers. Have you measured factual accuracy systematically (e.g., GSM‑8K exact‑match)? This would clarify whether judges are truly “fooled”.\n\nDefences – Given that perplexity fails, have you tried delta‑answer inspection (diff between outputs with and without preamble) or attention‑based detectors? Evidence would guide future mitigation.\n\nScore could increase if additional judges and a larger human study confirm undetectability, or if authors provide stronger evidence that inflated scores do not correspond to better factual quality.\n\nLIMITATIONS:\nThe paper candidly discusses compute and data imbalance, but it under‑states ethical risks: releasing a plug‑and‑play adversarial preamble generator could degrade the reliability of public leaderboards. Authors should propose a disclosure or gated‑release plan. Otherwise, coverage of methodological limitations is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes a method to generate preambles (equivalent to system prompts) which when attached as part of instructions to a frozen candidate LLM generates higher scored responses from a judge LLM. This is in contrast with previous approaches to game LLM judges by post hoc modification responses.

STRENGTHS AND WEAKNESSES:
The paper has decent clarity of language and the experiment design and execution is adequate. There is originality in the idea of trying out a new way to fool LLM judges without post-hoc modification of candidate LLM responses. The baselines used are extensive and there is a discussion on transferability of proposed attacks.

QUESTIONS:
Line 250: Missing reference to Table.

Appendix G.2 and G.3 (Preamble 6 and 9): Is there an explanation to why the Llama 8B+70B preambles do not look like natural English text as the other ones. I would assume the perplexity filter would easily flag these. It would be great to have some clarification.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07