Reverse Engineering Human Preferences with Reinforcement Learning
Authors: Lisa Alazraki
Yi-Chern Tan
Jon Ander Campos
Maximilian Mozes
Marek Rei
Max Bartolo
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM-as-a-judge
Human Preferences
Reinforcement Learning
Adversarial attack
Robustness
Primary area: applications

Abstract: The capabilities of Large Language Models (LLMs) are routinely evaluated by other LLMs trained to predict human preferences. This framework—known as *LLM-as-a-judge*—is highly scalable and relatively low cost. However, it is also vulnerable to malicious exploitation, as LLM responses can be tuned to overfit the preferences of the judge. Previous work shows that the answers generated by a candidate-LLM can be edited *post hoc* to maximise the score assigned to them by a judge-LLM. In this study, we adopt a different approach and use the signal provided by judge-LLMs as a reward to adversarially tune models that generate text preambles designed to boost downstream performance. We find that frozen LLMs pipelined with these models attain higher LLM-evaluation scores than existing frameworks. Crucially, unlike other frameworks which intervene directly on the model's response, our method is virtually undetectable. We also demonstrate that the effectiveness of the tuned preamble generator transfers when the candidate-LLM and the judge-LLM are replaced with models that are not used during training. These findings raise important questions about the design of more reliable LLM-as-a-judge evaluation settings. They also demonstrate that human preferences can be reverse engineered effectively, by pipelining LLMs to optimise upstream preambles via reinforcement learning—an approach that could find future applications in diverse tasks and domains beyond adversarial attacks.

Reverse Engineering Human Preferences with Reinforcement Learning
Lisa Alazraki* Imperial College London
Tan Yi-Chern Cohere
Jon Ander Campos Cohere
Maximilian Mozes
Marek Rei Imperial College London
Max Bartolo Cohere
Cohere
Abstract
The capabilities of Large Language Models (LLMs) are routinely evaluated by other LLMS trained to predict human preferences. This framework-knowe as I. MaM-as-a-judge--i highly scalable and relatively low cost. However, it is also vulnerable to malicious exploitation, as LLM responses can be tuned to overfit the preferences of the judge. Previous work shows that the answers generated by a Candidate-LLM can edited post hoc to maximise the score assigned to them by judge-LLM. In this study, we adopt a different approach and use the signal provided by judge-LLMs as reward to adversarially tune models that generate text preambles designed to boost downstream performance. We find that frozen LLMs pipelined with these models attain higher LLM-evaluation scores than existing frameworks. Crucially, unlike other frameworks which intervene directly on the model's response, our method is virtually undetectable. We also demonstrate that the effectiveness of the tuned preamble generator transfers when the candidate-LLLMM and the judge-LLM are replaced with models that are not used during training. These findings raise important questions about the design of reliable LLM-as-a-judge evaluation settings. They also demonstrate that human preferences can be reverse engineered effectively, by pipelining LLMs optimise upstream preambles via reinforcement learning an approach that could find future applications diverse tasks and domains beyond adversarial attacks.
Introduction
The LLM-as-a-juee aa-uuge framework has largely replaced human evaluation in the large-scale assessment
of LLMs 5 17 35| 40 with several widely used benchmarks now relying on this approach to
judge model performance across tasks [26] 20 30 (3]44) 40). Madge-LLMs are trained to predict human preferences, offering scalable and cost-effective alternative to human annotations [26] 40). However, prior work has shown that Budge-LLMM
vulnerable to adversarial attacks aimed at In particular, it is possible to find text sequences that, once appended to substituted for response, maximise the score awarded to it by judge 32 34 41. This type of attack intervenes post hoc on the text being evaluated and can thus be detected via human inspection or by computing the perplexity (PPL) of the modified response
artificially boosting their scores [32
34 38
19 32 34
In this study, investigate different. novel approach based reward modelling w.n.t. the judge-LLM's evaluation scores, testing both its effectiveness and detectability. Specifically, we use
Work done while at Cohere. Conrespondence to lisaalazraki200 www.raalll.mmrraaaaaa..a...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

these scores to tune an adversarial model that generates textual preamble: i..., additional sets of instructions- be injected into frozen candidate-LLAi causing its responses receive higher scores from the judge. The loss function that optimises the preamble generator is adapted from Contrastive Policy Gradient [1]. but depends on rewards computed solely on the generations of the Candidate-LLMM and without directly observing the preambles. We refer to this technique of pipelining multiple LLMs to indirectly optimise upstream preambles in an RL fashion as Reinforcement Learning for Reverse Engineering (RLRE).
There are several advantages to tuning an upstream preamble generator rather than directly overfitting the candidate- LLM to the judge-LLM's rewards: (1) the specialised preamble generator can be smaller in size, hence computationally cheaper train; (2) tuning the preambles while leaving the candidate-LLAn frozen retains its original capabilities and is less likely to result in noticeable stylistic changes in its output, thus potentially making the attack harder to detect; (3) the generated preambles that align the candidate-LLMM to the judge are natural language instructions, which can be analysed and interpreted; (4) once trained, the preamble generator can serve as a plug-and-play component for pipelining with different Candidate-LLM, (we experiment with preamble transferability across candidate-LLMN in Section 5 (5) in broader contexts, tuning preambles can also serve as a means of optimising models that cannot be fine-tuned directly (e.g., those accessible only via inference APIs or for which fine-tuning would be too costly).
Indeed, we find that responses produced by candidate- LLLMs pipelined with the preamble generator receive substantially higher evaluation scores from judge-LLMs compared responses attacked with other strategies, while also eluding detection methods. In contrast, existing attacks can be detected via perplexity analysis [99 or human inspection.
Finally, we perform an analysis of the optimal preambles and find high variability in their fluency across different model pipelines. While natural language preambles enhance interpretability, our findings raise important questions about whether constraining conditioning tokens-such as preambles or reasoning tokens-to the manifold of natural language may inadvertently limit model capabilities.
This paper makes the following main contributions:
1. We show that an adversarially tuned preamble generator, pipelined with a frozen LLM, is effective at deceiving judge-LLMs into assigning higher scores. To the best of our knowledge, this is the first work that optimises preambles to be injected into a frozen LLM for this purpose. In contrast, previous studies focus on finding text sequences be appended to pre-generated responses.
2. We demonstrate that our adversarial preamble generator can be successfully pipelined with candidate-LLLM and judge-LLMs not seen during training.
3. We show that our attack does not increase PPL scores and is rarely flagged by human evaluators. Hence, it cannot be detected using existing safeguards.
4. We observe variations in optimal preamble style, fluency and naturalness across models, suggesting that conditioning LLMs human-readable sequences only (for example in preambles or reasoning traces) may be overly restrictive from a performance perspective.
5. Our work highlights intrinsic vulnerabilities in the LLM-as-a-judge paradigm and calls into question its reliability and robustness.
6. More broadly, this work introduces RLRE, a novel approach that pipelines LLMSs to optimise upstream textual preambles in reinforcement learning setting. While here we use RLRE to reverse engineer human preferences with the aim of boosting LLddee---judge evaluation, we postulate that this method could be paired with different downstream rewards optimise preambles for a variety of applications beyond just adversarial attacks srtackk-iiccuding but not limited to meaningful tasks such as reducing toxicity or mitigating bias.
2
Related Work
Prior work investigating the robustness of LLLM-as-a-judeee has found that this approach suffers from multiple inherent biases. Existing research has sought to exploit these biases by crafting adversarial attacks aimed at maximising the scores assigned by judge-LLMs -LMs to candidate responses.
In this context, preambles are also known system prompts.

---

Give general advice to help answer the following question..: [4]
Evaluation
Training dataset
R(q, c)
Candidate-LLM
Judge-LLM LLM
Preamble
generator
R(q, c)
Lip. p':
p'
R(g, c²)
Comperison
Figure 1: Reinforcement Learning for Reverse Engineering (RLRE) pipeline for training a preamble generator. Given question q from training set, prepend to it general instruction and feed it to the preamble generator æ. In order for π to learn the policy, we sample two preambles per question, and p'. The respective rewards obtained by appending y to p and p', respectively, and (i) passing each as input to the Landidate-LLLM, which generates the responses c and c', and (ii) having the judge-LLM evaluate each question-response pair and extracting the respective numerical rewards from these evaluations. The loss function that optimises depends on the delta between the rewards R(g,e c) and R(q, c').
Biases. Judge-LLMs are not unbiased evaluators. 24 and 28 observe that judge-LLMs prefer their own generations to those of other models in the large majority of cases. [36] further show that when asked to choose the best among multiple responses, GPT models favour the first candidate displayed, regardless of its quality. Additionally, 4 find that LLMs tasked with scoring other models prefer visually appealing generations regardless of content, and generations that are attributed, even falsely, to authority figures. Similarly, 38 conclude that L.LM judgment vulnerable spurious attributions (" ('authority bias"). They additionally observe that judge-LL.Ms tend to prefer longer responses (" ("verbosity bias'), ) responses falsely identified majority beliefs ( -'aanewago-efffect bias') ), and responses presented as the result of a refinement process ('refinement-wwat bias').
B8 show that the biases of judge-LLMs LLMs can be manipulated to artificially boost their evaluation scores. They append specific text sequences to candidate responses: a false book citation exploit authority bias, sentence stating that most people favour that response exploit bandwagon-effee bias, or piece of text suggesting the response has been through a refinement process, to leverage refinement-aware bias. In each case, they find that the resulting responses evaluated more favourably by all or some of the judge-LL.Ms. They also show that evaluation is affected when the length of response is increased, without any improvement in its quality, this type of adversarial attack further, tuning a universal text sequence that, when appended to a pre- generated response, increases its evaluation score. This sequence is found by searching sequentially through the vocabulary, iteratively selecting the word that maximises the average reward from the judge on the training set. Their method successful at inflating LLM judgement on Topical-Cha [14] and SummEval [TT] and they show the attack transfers to previously unseen judge-LLMs not included in the search process. Rather than tuning a universal phrase, 34 train sample-specific text sequence to be selected more often by judge in pairwise comparison. Similar to 32 they append the tuned sequence to a pre-generated L.M response. Finally, 44 experiment with replacing responses with fixed instructions that invalidate the original L..M-as-a-uddee prompt. Note that [32] 33 and [4] all observe that their attack can large part be detected by measuring the perplexity of the responses. As the attack intervenes directly on the response and alters it, attacked responses tend to display higher PPL.
Ad Adversarial attacks.
[32 take
Unlike the above methods, the attack we propose does not modify the generated text post hoc. This makes its detection substantially more difficult.
3 Method
Given a training dataset of questions = {(5)) } and fixed instruction prompt i, we aim to train a preamble generator Th (iplii.g.. formulate the RL problem as:
(p 9j) to generate textual preambles p conditioned on i and q. We
= Boro- oo
3

---

where LLMc a frozen LLM-rrefrred as the candidate-LLM which takes a preamble pj and the corresponding question qj and outputs candidate response cj. Note that the reward is function of the preamble because I. L.M&c is conditioned on it. Our reward model is a frozen L.M that outputs a verbal critique followed by numerical score, as in the LLM-as-a-judge framework. We refer to this model as the L.dge-LLM. In our case, the score output by the judge is discrete on the I-10 scale, elicited using MT-Bench [40] prompts for single (as opposed to pairwise) evaluation. We use this numerical score as the reward in our training pipeline. Figure T iillustrates the training pipeline in detail.
In order to optimise the RL problem, we adapt Contrastive Policy Gradient (CoPG) [1]. The rationale behind this choice, as well as comparison with other RL algorithms, is further elaborated in Appendix A For a pair of two sampled preambles Py and p/y we introduce the following sampling loss:
- -R(()))))))))) In TP (1)))..g)) In Tyet (p,li
("h
T T(6 (. q3) Tzet (p// 93) ))
2
((p).p/;T)= [R(g) (j) R(g;,())
Tref is a reference model used for regularising the RL problem, which we set to be the base LLM underlying the preamble generator. As in (12). 3 is hyperparameter regulating the importance of the KL-divergence between the sequence log- liceliooss of π and Tyee in the overall loss.
As π is trained to generate question-specific preambles, we pipeline it with the candidate- at test time to dynamically generate responses to new questions. We prompt the preamble generator with the same fixed instruction i used during training. along with a question g. to generate the response c. judge-LLM then prompted to assign a score to c.
As an additional consideration, it is worth noting that 12 investigate CoPG solely in an offline manner. However, they hypothesise that the method should also scale the online setting. To the best of our knowledge, this work is the first to successfully apply similar method to online learning.
4 Experiments
4.1 Models and Hyperparameters
To test the generalisability of our method, we use LLMs from both the Command and the Llama 3.1 [15] model series. We train and test three distinct pipelines, illustrated in Table [ Note that all pipelines are trained with the same judge-LLM, i.c. Command R+ prompted as in [40]. We tune the Command R7B 77 preamble generators on Google Cloud TPU v5e containing 64 chips. We train the Llama 3.1 8B Instruct preamble generator on a single Nvidia H100 GPU. Candidate- and judge-LLMs LLMSs from the Command family are accessed via API. The Llama 3.1 70B Instruct candidate- LLMM is deployed and queried on a local server.
Due to computational limitations, we perform all hyperparameter tuning on the Command R7B+R7B pipeline, and apply the same hyperparameters to the other two. As shown in Section 4.4 the Command R7B+R7B pipeline attains the greatest performance improvements over the baselines, and it is therefore likely that additional hyperparameter tuning would further raise the scores obtained by the Command R7B+R and Ll ama 8B+70B pipelines. This may be especially true for the latter, which comprises a family of models different from those used for hyperparameter tuning. It is worth noting that, since the preambles only need influence the candidate-LLNN outputs and do not necessarily need to be fluent from a human perspective, the hyperparameter tuning process results in relatively low value of the KL-divergence coefficient (8 0.03), which regulates the similarity between the output distributions of the preamble generator and the reference model. This low / value allows our preambles to deviate more drastically from the reference policy during training.
We train each pipeline with early stopping according to validation performance. All hyperparameters, API model IDs, and training process details are given in Appendix B
httts::/cooeere.com/ceee

---

Table I: The training pipelines include models of different sizes and families preamble generators and/or candidate-LLMMr Command R+ (104B parameters) used as the judge-LLM in all pipelines.
Pipeline identifier
Preamble generator
Candidate-LLLM
Command R7B+R7B
Command R7B
Command R7B
Command R7B+R
Command R7B
Command (35B)
Llama 8B+70B
Llama 3.1 8B Instruct
Llama 3.1 70B Instruct
4.2
Datasets
We test all pipelines on MTT-Bench [40]. which consists of 160 open-ended questions, split among two conversational turns, grounded in the following domains: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. We choose this benchmark as it is established and widely used in LLM assessment, and tests balanced distribution of diverse skills on challenging multi-turn questions [0]. Crucially, MT-Bench supports independent judgement as opposed to pairwise comparison with other models 22
21 It is also worth noting that MT-Bench represents the setup we target in our approach, which makes it suitable to demonstrate that it is possible to reverse engineer human feedback in controlled setting. Since MT-Bencc does not comprise training set, we fine-tune and validate the preamble generators using questions from UltraFeedback
(using MT-Bench prompts for single evaluation elicit the downstream rewards). The questions in UltraFeedback are extracted from ShareGPT [6. FLAN (25), Evol-Instruct [37, UltraChat (10) FalseQA [T6], and TruthfulQA [23]. Collectively, these datasets cover a wide range of topics, and the domains in MT-Bench are represented in UltraFeedback. The distribution of the different tasks
within the training data is further analysed in Appendix C.2
Additionally, we test transferability to further benchmark, Arena- Hard 21 as discussed in Sec- tion 5
4.3 Baselines
We compare the evaluation scores assigned to Candidate-LLMMs attacked with the preamble generator with those given unattacked candidates. Additionally, compare against existing methods that exploit vulnerabilities the LLM--s-a-jjdde framework to artificially boost their evaluation scores. We describe these additional baselines below. Note that all of them modify pre-generated responses from the unattacked model.
Verbosity bias attack. We ask the Landidate-LLM to increase the length of pre-generated response. To this end, we use the prompt designed by 38 to lengthen the response without necessarily improving its quality.
Bandwagon-effect bias attack. We append to each pre- pre-generated response a text sequence stating that high percentage of people think that response should be awarded the highest rating. Consistent with 38 we randomly choose percentages between 60% and 90%.
Authority bias attack. Using the same prompting strategy as [38] we ask the candidate-LLM, to invent a plausible book source for a pre-generated response, given a citation template. The citation is then appended to the response to increase its perceived authority.
Refinement-aware bias attack. Given a pre-generated response, we have the candidate- polish it using the refinement prompt in (38). The judge is then presented with the original response, followed by the refinement prompt and the new, polished response.
Universal adversarial attack. We append to each pre- generated response the universal adversarial attack phrase from 322, learned for absolute assessment when attacking the Topical-Chat [4] overall score. Like UltraFeedback and MT-Bench, 'TopicalCCha encompasses wide range of topics and features multi-turn questions similar to those in MTT-Bench. Note that learning new universal phrase on UltraFeedback training set is computationally infeasible: for training samples, the exhaustive
5

---

Table 2: MT-Bench evaluation scores assigned by the Command R+ Judge-LLM to candidate-LLMMs attacked with different strategies. Each setup is run five times and the scores are averaged (showing the standard deviation in the subscript) to account for small variations due temperature sampling.
Attack type
Candidate-LLM
No attack
Verbosity
Bandwagon
Authority
Refinement
Universal
Preambles
Turn 1
7. 0.
7.33g.08
7.47o.05
7.51o.05
7.78p.05
7.580.00
8.210.07
Command R7B
Turn 2
6.99g.us
7.29u.02
7.170.0s
7.290.10
7.450.0s
7.25p.08
7.66u.09
Overall
7. 7.29p.0s
7.31
7.32g.06
7.40g.07
7.6l
7.4
7.930.05
Turn I
8.09p.0s
7.99o.
7.980.06
8.170.05
8.10p.05
8. 8.10p.
8.45g.07
Command R
Turn 2
7.570.12
7.73p.0s
7.72p.14
7.65g.06
7.810.05
7.75p.09
7.920.03
Overall
7.83p.10
7.860.09
7.850.10
7.91
7.950.05
7.92p.07
8.18g.05
Turn
8.470.08
8.299.06
8.390.05
8.380.07
8.510.07
8.50p.os
8,56g.08
Llama 3.1 70B
Turn 2
7.64g.06
7.49a.os
7. 7.650.0s
7.62 7.62g.07
7.750.09
7.85p.07
7.880.0s
Instruct
Overall
0.066.07
7.89p.07
8. 8.02p.07
0..00.7
8.13p.08
8. 8.17p.06
8.220.06
Refinemennt
- Universel Reasoning
Preamiles
Reasoning
Reasoring
Math
Relestay
Math
Roleplay
Math
Roteplay
Coding
Winting Codking
Wr bnggnn Coding
Witling
Extraction
Humanttes
Extraciion
Humarities
Extratioon
Humanties
(a) Command R7B
(b) Command R
(c) Llama 3.1 70B Instruct
Figure 2: Average scores per question type obtained by candidate- using a refinement-aware bias attack, 32 universal adversarial attack, and the adversarial preamble generator.
search method in 32] computed over the 20k-word vocabulary makes 20000 x 4 x n calls to the judge-LLM. In our case, that would equal 4 10 inference calls.
4.4 Results
Table the average evaluation scores obtained by three candidate-LLMMs under all baseline set- tings and when pipelined with our adversarial preamble generators. Models injected with adversarial preambles consistently obtain higher evaluations from the Command R+ judge at both question turns, with the Command R7B+R7B pipeline (i.e., the one on which we perform hyperparameter tuning) achieving the most substantial improvements. We also find that turn 1 responses benefit the most from the preambles. Note that MT-Benehe turn 2 questions require adjusting turn answers according to new constraints. This type of task is not well represented in UltraFeedback, making it OOD w.r.t. the training data (see also Appendix (.2). Nevertheless, the tuned preamble generators remain effective at raising turn 2 scores test time.
We find that among all candidate LLMs, the strongest baseline attacks are the refinement-awarc bias attack and the universal adversarial attack proposed by (32). On average, our adversarial preambles increase Command R7B's overall score (on 1-10 continuous scale) by 0.32 points over the refinement-awie aware attack and by 0.52 points over the universal attack. For Command R. the corresponding improvements are +0.23 and +0.26, respectively. Injecting adversarial preambles into Llama 3.1 70B Instruct yields smaller but consistent gains, raising its score by 0.09 relative to refinement-aware attack and 0.05 relative the universal attack. Furthermore, compared with their non-attacked counterparts, the preamble-based attack substantially increases the overall judge scores for Command R7B, Command R and Llama-by 0.64, 0.35, and 0. 061, respectively. We report confidence intervals in Appendix D.I

---

Table 3: Candidate transferability (a) and judge transferability (b) of our preamble-based attack MT-Bench. All scores are averaged over five runs.
(a) Candidate transferability
(b) Judge transterability
Preambles from pipeline
Attack type
GPT-3.5 GPT-4o-mini Claude
Candidate-LLIA
No attack
7.58p.08
6.40p.07
9.02g.06
Command R7B-R7B
Command R7B+R
Llama
8B+70B
Verbosity
7.360.09
5.610.04
8.740.04
Bandwagon
7.470.04
6.250.04
8.79g.07
Command R7B
7.930.08
7.680.08
7.40p.10
Authority 7.48p.08
5.920.08
8.920.12
Command R (35B)
Refinement
7.710.11
6.39p.05
9.18g.06
8.01
8.18
p.
Universal
7.33p.10
6.06p.03
8.94g.07
Llama 3.1 708 Instruct
8.21n.05
8.19p.0s
8.22
Preambles
8.07
6.71p.02
9.44g.06
we illustrate the evaluation scores per question type for the two best-performing baselines (refinement-wware bias and universal adversarial attack) and the adversarial preamble generator. Overall, the adversarially tuned preambles are most effective at raising the scores of reasoning and math responses, followed by extraction, STEM and roleplay. Fine-grained results for all domains are shown in Appendix D.2
In Figure
5 Analysis
5.1 Attack Transterability
We investigate the transferability of our preamble attack across different candidate- and judge-LLMs.
Transferability across candidate-LLMs. At test time, we pipeline each candidate- -LMM with adversarial preamble generators tuned with different candidate. As shown in Table [ab the judge's evaluation scores remain higher than all baselines (shown Table in all cases except for Command R7B pipelined with the preamble generator from the Llama pipeline, whose scores align with [32 's universal attack. This demonstrates strong transferability of the adversarial preamble attack across different L....
Transferability across judge-LLMs. real-world scenarios, the target judge-LLM may not be known in advance. It is therefore important to assess whether method generalises judges that were not used during training. Table3(b) [(b]] results where different judge-LLMss LLMs evaluate candidate responses across all attack settings. For this analysis, employ GPT-3.5 GPT-4o- mini [27]. and Claude Haiku [2]. To maintain cost efficiency, we restrict evaluation to Command R7B. While the absolute reward scales differ across the three judges, all three consistently assign the highest scores to responses generated using our preamble generator, despite it having been trained with rewards from different judge (Command R+). These findings suggest that the attack remains effective even when the target judge-LLMM unknown.
Transferability across benchmarks. We further evaluate the Command R7B pipeline the Arena-Hard benchmark, which differs from both UltraFeedback and MT-Bench in task distribution. Moreover, its reward metric-the Arena Score Rate (ASR)-diffed in both assignment strategy and numerical range, as described in [2I]. We conduct evaluations using both the Command R+ judge (used during training) and an unseen judge, GPT-4, which serves the standard evaluator for Arena-Hard at the time of writing. Applying the tuned preambles yields an average score increase of +3.5 with Command R+ and +1.2 with GPT-4 (note that Frena-Hard scores range from 0-100; see Appendix for full results). These results demonstrate that RLRE generalises effectively to new benchmarks without additional task task-specific training.
https://platforn. openai. www.///o//ooo.ee//cc/ccco http:///aa.oooooooooooaooooaoooa
gpt- turbo

---

Table False negative rate (FNR) of PPL-W for each attack type. Verbosity bias, bandwagon bias and our preamble- bassubleebededddde attack are rarely detected with this method (FNR 2 0.90). The universal adversarial attack
32 almost al-
ways detected (FNR 0.04).
0.4
0.2
Attack type
PPL-W (FNR)
Verbesdt
Brondaggn
Auttontt
Reinenmnnt
Univrsal
Preambles
Verbosity
0.91
stack
(ous)
Bandwagon
0.93
Authority
0.88
Figure 3: Proportion of LLM responses that have been labelled as 'attacked' by human evaluators. Responses generated using adversarial preambles identified as attacked nearly as rarely as those produced by a non-attacked model.
Refinement
0.66
Universal
0.04
Preambles
0.90
5.2 Attack Detectability
We examine the detectability of all attacks via perplexity analysis [19]. This is the established detection method for adversarial attacks on LLM output, used in prior and contemporary work
32 41. Notably, this method can be used models served via API, as is the case for our frozen candidate- without requiring access to internal mechanisms such as attention maps 18, 29) or the entire probability distributions over the vocabulary at each decoding step (9). As an additional test, we also measure the detectability of the attacks via human evaluation of the candidate responses.
Perplexity analysis. We use a perplexity filter over a 10-token sliding window in 19 to identify whether the Landidate-LLM has been attacked. A response is labelled as attacked if the PPL of any subsequence in the sliding window is greater than a threshold We set t ensure false positive rate within 0.05 on the test set responses produced by the non-attacked model. Table illlstrates the results of this detection method referred as PPL-W-for the Command R7B candidate.
Human evaluation. To assess the detectability of each attack by visual inspection, we recruit 16 annotators with domain expertise and knowledge of the possible attacks. We sample a balanced subset of 400 test responses generated by Command R7B under each attack setup (including the non-attacked model), randomly split it into 16 equal-sized portions, and have each annotator assign binary label ('attacked" or 'not attacked') ) to each generation. Figure 3 shows the proportion of responses labelled as "attacked' for each attack type. Attacks that insert formulaic sequences into the response (bandwagon bias, authority bias, refinement-awe all or most cases (100%, 96%, 95% and 97%, respectively). while the verbosity bias attack is detected approximately half of the time (49%). Notably, responses produced by the preamble-based attack labelled as 'attacked' nearly as rarely (14%) as those generated by the non-attacked model (12%). We also measure the rate which each annotator is able correctly detect each type of attack (both in terms of accuracy and F1). For both scores, we compute the median absolute deviation (MAD) across all annotators for each attack type, and find it be zero in all cases, indicating that the annotators are able (or unable) to detect each attack at very similar rates. This supports the hypothesis that different attacks are inherently (un (un)detectable by visual inspection, as we do not find high variance in detection rates between different annotators.
bias and universal attack) are detected in
5.3 Ablations
In our framework, we input into the preamble generator fixed instruction concatenated with a question from the dataset, resulting in question-specific preambles. To evaluate the degree to which this prompting strategy contributes to the effectiveness of our attack, we run an ablation study where (i) we remove the question from the input and feed the preamble generator only a generic instruction, and (ii) we discard the instruction altogether and only input special tokens to signal the start of generation turn (both prompts are shown in Appendix E. In the latter case, the model is
8

---

Table 5: Ablated MT-Bench overall scores obtained by (i) removing the question and feeding generic instruction to the preamble generator, and (ii) removing the instruction and only feeding special tokens to signal the start of the turn. Scores are assigned by Command R+ judge and are averaged over five runs.
Ablated setting
Pipeline
No question
No instruction
Command R7B+R7B
7.76p.07
7.70p.07
Command R7B+R
8.14p.0s
8. 8.13
Llama 8B+70B
8.190.11
8. 8.180.12
virtually unconstrained and may generate any text string, and thus the training signal is even more consequential in determining the content of the preambles.
Table 5 illustrates the overall scores assigned by the Command R+ judge under all ablated settings. Although both ablation strategies result in slightly lower performance than the corresponding non- ablated pipelines, all the scores remain above all baselines (refer to Table 2 for the baseline results). This demonstrates the robustness of the attack under different prompting strategies.
5.4 Analysis of Generated Texts
Do successful preambles exhibit common patterns? Upon analysing the successful preambles produced by each tuned generator, find consistency among preambles generated by the same model, but high variability across different models. While preambles generated by the Command models (both conditioned the data point and conditioned on generic instruction) share similar prructree-prviddiie
a blueprint for how to design an answer-preambles that are only conditioned on the start-of-turn token, well as all the Llama preambles, deviate substantially from this pattern. The latter tend to reiterate the same phrases multiple times, and often do not appear fluent to human reader. When they are not conditioned meaningless sequences of characters. Nevertheless, all of these preambles are successful at raising LLM evaluation scores, evidenced in Section 5.3 Appendix shows representative preambles for each pipeline.
instruction, they even devolve into apparently
It is worth noting that, further discussed in Appendix B assign relatively low weight to the log- likelihoods the adversarial preambles within the loss function, thus prioritising reward over fluency during training. Remarkably, the training still converges toward preambles that elicit high-reward candidate responses, regardless of their fluency. This suggests that constraining preambles-or other conditioning tokens such as reasoning tokens-to the manifold of natural language may not always be optimal.
Are attacked responses more accurate? Figure 2 shows that the preamble-based attack is particu- larly effective at raising the scores assigned to math and reasoning responses. Since these responses are evaluated by the judge-LLM against ground truth 440, this raises the question of whether the attack has improved response accuracy. We thus evaluate via normalised exact match all math and reasoning responses from all models, both the non-attacked setup and the preamble-attacked version. We find that the average accuracy rates are very similar, with slight advantage for the non-attacked models (4599% over the preamble attack (44.2%). Hence, the accuracy of the final answer does not improve due to the preambles. However, as shown by the representative examples in Appendix H. the attacked responses have more structured reasoning chains, usually arranged into distinct paragraphs labelled with clear, explanatory headers. We thus postulate that the improved layout may be solely responsible for the higher scores assigned by the judge-LL.M, regardless of correctness.
Are attacked responses better overall? Aside from math and reasoning, the other MT--eench domains are either not reliably verifiable with automated methods (extraction, STEM, humanities, coding) or lack objective verification entirely (writing, roleplay). To understand whether responses produced via our preamble pipeline have improved over those generated by the non-attacked model,

---

Table 6: Human evaluation of generated responses for all task domains.
Number of assigned labels
Attack type
Avg. human rating
Poor
Fair
Good
No attack Preambles
72
50
118
6.22
73
54
113
6.24
have them evaluated by expert human annotators, blind to the attack and instructed to assess correctness and quality by assigning to each response one of three labels: 'Poor', "Fair' or 'Good', along with discrete 1-10 rating matching the MT-Bench judge-LLM scale. We do this for all responses generated by Command R7B, across all domains. Table shows that the label distributions are fairly similar for both the non-attacked and the attacked model, with the non-attacked model receiving slightly more positive labels ("Good"), " and the attacked one receiving slightly more mid- range labels ('Fair'). We observe relatively strong inter-annotator agreement (aggregated Spearman's 0.78, combined p < 0.001, Kendall's W = 0.72, p < 0.001). Moreover, the difference between the human ratings of the non-attacked and the attacked model is negligible (0.02), whereas the difference between the judge-LLM's ratings of the same two setups is substantially greater and favours our attack.
Conclusion
We have shown that human preferences can be successfully reverse engineered adversarial text preambles, and that injecting these preambles into candidate LLMM constitutes powerful attack on the LLM-as-a-judge framework. This attack not only outperforms previous methods inflating the scores assigned by judge-LLM to candidate responses, but also remains virtually undetectable using existing safeguards. In contrast, current strategies that intervene post hoc on the responses display high PPL and can be easily detected by visual inspection. Additionally, we have found that not seen during training, enabling our adversarial preamble
the attack transfers to candidate-LLan generator to serve as a plug-and-pla component for artificially boosting the scores of multiple LLMs after a single training process. Finally, we have shown that preambles tuned with one judge-LLM can effectively attack different judges from different model families, and are effective at inflating scores on multiple benchmarks. These findings raise important questions regarding the reliability of LLM-as-a-juue a-jude evaluation.
In addition to pointing future research directions for the design of more robust evaluation frame- works, this work introduces Reinforcement Learning for Reverse Engineering (RLRE), novel strategy that combines LLMs to tune upstream textual preambles via reinforcement learning, thus enabling the indirect optimisation of models, including those that cannot be fine-tuned directly. While here we have shown its effectiveness in the context of adversarial attacks on LLM evaluation, future research can investigate other avenues of application for this approach (e.g., automatic generation of other types of attack, but also improvements to LLM output such as toxicity or bias mitiga- tion), as well as different granularities of sequence adaptation (e.g., query-specific, task-specific, domain- -peaiin,eeeific),..... and the optimisation of tokens at different positions within an input sequence (c.g.. post-query instructions instead of pre-query preambles). Looking ahead, we envision that future systems will not rely on a single token stream, but integrate and optimise multiple inputs- e.g. from users, tools, and auxiliary models- each contributing to the generation of final responses that better align with chosen objectives.
Broader Impacts
This work focuses on aligning candidate-LLan -LMMs to judge-LLMs by means of tuned preambles injected into the candidate to obtain inflated evaluations. While there is a chance that this strategy may be exploited by adversaries,
of scientific interest to the community that such an attack is not only possible but also particularly effective. Note that, while we openly disclose our training algorithm and hyperparameters and train using publicly available data, we do not release our trained preamble generator checkpoints to the public, this may encourage their misuse.
10

---

Acknowledgments
The authors are grateful to Yannis Flet-Berliac for his insightful guidance on hyperparameter tuning and for valuable input in refining the training pipeline. Appreciation is also extended to John Dang, Matthieu Geist, Roman Castagné, and Eugene Choi for their helpful suggestions throughout this work.
References
[1] A. Ahmadian,, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Ustün, and S. Hooker. Back to basics: Revisiting REINFORCE-SYL
optimization for learning from human feedback in LLMs. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248-12267, Bangkok, Thailand, Aug. 2024, Association for Computational Linguistics.
[2] Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku.
[3] A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. Fernández, A. Gatt, E. Ghaleb, M. Giu- lianelli, M. Hanna, A. Koller, A. Martins, P. Mondorf, V. Neplenbrock, S. Pezzelle, B. Plank, D. Schlangen, A. Suglia, A. K. Surikuchi, E. Takmaz, and A. Testoni. LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume? 2: Short Papers), pages 238-25288 255, Vienna, Austria, July 2025. Association for Computational Linguistics.
[4] G. H. Chen, Chen, Z. Liu, Jiang, and B. Wang. Humans LLMs the judge? A study on judgement bias. In Y. Al-Onaizan, M. Bansal, and Y.-N, Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301-8327, Miami, Florida, USA, Nov. 2024, Association for Computational Linguistics.
[5] C.-H. Chiang and H.-y. Lee. Can large language models be an alternative to human evaluations? In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguistics.
[6] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Wu, H. Zhang, I.. Zheng, S. Zhuang, Y. Zhuangg J. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90 https://Imsys hrtp://////ooo//g//////////////0 bltp/ 00-30-3330000 101-330-vicaua vicunal March 2023.
[7] Cohere, :, Aakanksha, A. Ahmadian, M. Ahmed, J. Alammar, M. Alizadeh, Alnumay, S. Al- thammer, A. Arkhangorodsky, V. Aryabumi, D. Aumiller, R. Avalos, Z. Aviv, S. Bae. S. Baji, A. Barbet, M. Bartolo, B. Bebensee, N. Beladia, W. Bellcr-Morales, A. Bérard, A. Berneshawi, A. Bialas, P. Blunsom, M. Bobkin, A. Bongale, S. Braun, M. Brunet, S. Cahyawijaya, D. Cairuz, A. Campos, C. Cao, K. Cao, R. Castagné, J. Cendrero, L. C. Currie, Y. Chandak, D. Chang, G. Chatziveroglou, H. Chen, C. Cheng, A. Chevalier, T. Chiu, E. Cho, E. Choi, E. Choi, D. Crouse, A. F. Cruz-Salinas, B. Cyrus, D. D' Dosuza H. Dalla-Torre, J. Dang, W. Darling, O. D. Domingues, S. Dash, A. Debugne, T. Dehaze, S. Desai, J. Devassy, R. Dholakia, K. Duffy, A. Edalati, A. Eldeib, A. Elkady, S. Elsharkawy, I. Ergün, B. Ermis, M. Fadace, B. Fan, L. Fayoux, Y. Flet- Berliac, N. Frosst, M. Gallé, W. Galuba, U. Garg, M. Geist, M. G. Azar, F. Gilsenan-McMalle Goldsack, A. Gomez, V. M. Gonzaga, N. Govindarajan, M. Govin- dassamy, N. Grinsztajn, N. Gritsch, P. Gu, S. Guo, K. Haefeli, R. Hajjar, T. Hawes, J. He, Hofstätter, S. Hong, S. Hooker, T. Hosking, S. Howe, E. Hu, R. Huang, H. Jain, R. Jain, N. Jakobi, M. Jenkins, J. Jordan, Joshi, Junn, T. Kalyanpur, S. R. Kamalakara, J. Kedrzycki, G. Keskin, E. Kim, J. Kim, W.-Y. Ko, T. Koemi, M. Kozakov, W. Kryšciński, A. K. Jain, K. K. Teru, S. Land, M. Lasby, O. Lasche, J. Lee, P. Lewis, J. Li, J. Li, H. Lin, A. Locatelli, K. Luong, M. Lopez, A. Mann, K. Marchisio, Markham, A. Matton, A. McKinney, D. McLoughlin, J. Mokry, A. Morisot, A. Moulder, H. Moynehan, M. Mozes, V. Muppalla, L.. Murakhovska, H. Nagarajan, A. Nandula, H. Nasir, S. Nehra, J. Netto- Rosen, D. Ohashi, J. Owers-Bardsley, J. Ozuzu, D. Padilla, G. Park, S. Passaglia,
T. Chung, V. Cirik, A. Cismaru, P. Clavier, H. Conklin, L. Crawhall-
S. Goldfarb-Tarrant
R. Ma, L. Mach, M. Machado, J. Magbitang,
11

---

J. Pekmez, L. Penstone, A. Piktus, C. Ploeg, A. Poulton, Y. Qi. S. Raghvendra, M. Ramos, Rodriguez, S. Roy. S. Ruder. L. Ruis. L. Rust, A. Sachan, A. Salamanca, K. K. Saravanakumar, I. Satyakam, A. S. Sehag, P. Sen, S. Sepehri, Shen, Sherborne, S. S. Shi, S. Shivaprasad, V. Shmyhlo, A. Shrinivason, I. Shteinbuk, A. Shukayev, M. Simard, E. Snyder, A. Spataru, V. Spooner, 'T. Starostina, F. Strub, Y. Su, Sun, D. Talupuru, E. Tarassov, E. Tommasone, J. Tracey, B. Trend, E. Tumer, A. Üstün, B. Venkitesh, D. Venuto, P. Verga, M. Voisin, A. Wang. D. Wang, S. Wang. E. Wen, N. White, J. Willman, M. Winkels, C. Xia, J. Xie, M. Xu, B. Yang. T. Yi-Chern, I. Zhang, Z. Zhao, and Z. Zhao. Command a: An interprise-reade large language model, 2025.
E. Ranjan. P. Richemond, C. Robert-Michon,
Seshadri,
[8] G. Cui, L. Yuan, N. Ding, G. Yao, B. Hc, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9722-992222 9744. PMLR, 21-27 27 Jul 2024.
[9] S. Dasgupta, S. Nath, A. Basu, P. Shamsolmoali, and S. Das. HalluShift: Measuring distribution shifts towards hallucination detection in LLMs. In 2025 International Joint Conference on Neural Networks (IJCN), 2025.
[10] N. Ding, Y. Chen, B. Xu, Y. Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029-3051, Singapore, Dec. 2023. Association for Computational Linguistics.
[11] A. R. Fabbri, W. Kryšciñski, B. McCann, C. Xiong, R. Socher, and D. Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391 2021.
[12] Y. Flet-Berliac, N. Grinsztajn, F. Strub, E. Choi, B. Wu, C. Cremer, A. Ahmadian, Y. Chandak, M. G. Azar, O. Pietquin, and M. Geist. Contrastive policy gradient: Aligning LLMs on sequence-level scores in pppervised-friendie
fashion. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21353 Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics.
[13] M. Gheshlaghi Azar, Z. Daniel Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Ca- landriello. A general theoretical paradigm to understand learning from human preferences. In S. Dasgupta, S. Mandt, and Y. Li, editors, Proceedings The 27th Internatioaa Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447-4455. PMLR, 02-04 May 2024.
[14] K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, and D. Hakkani-Türr Topical-Chat: Towards Cnowledge-Crounded Open-Domain Conversa- tions. In Proceedings of Interspeech 2019, pages 1891-1895, 2019.
[15] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang. A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chem, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. C.McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia- Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, Radenovie, F. Guzmán, Zhang, G. Synnaeve, G. Lec, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock. J. Hong. J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, I.. Rantala- Yeary, L. van der Maaten, L. Chen,
L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira,
12

---

M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, Alrassy, P. Zhang. P. Li, P. Vasic, P. Weng. P. Bhargava, P. Dubal, P. Krishnan, S. Koura, P. Xu, Q. He, Q Dong. R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly,
R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa. S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende. S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao. U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang. Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape. B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu. C. Wang. C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, Kreuk, Tian, F. Kokkinos,
Ozgenel,
F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang. H. Zha, H. Habeeb. H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj. I. Molybog. I. Tufanov, I. Leontiadis, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus. J. Tang. J. Chan, J. Zhen, J. Reizenstein,
J. Teboul, J. Zhong. J. Jin, J. Yang. J. Cummings, J. Carvill, J. Shepard. J. McPhie, J. Torres,
J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang. K. Chawla, K. Huang. L. Chen, L. Garg. L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mchta, N. P. Laptev, N. Dong. N. Cheng, O. Chermoguz, O. Hart, O. Salpekar, O. Kalinli, Kent, P. Parekh, P. Saab, P. Balaji. P. Rittner, P. Bontrager, P. Roux, Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj. Q. Liang. R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang. R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, Mahajan, S. Verma, Yamamoto. S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng. S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, Chintala, S. Max. S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho. S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, Koehler, Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi. V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang. Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. De Vito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma. The Llama 3 herd of models, 2024.
[16] S. Hu, Y. Luo, H. Wang, X. Cheng, Z. Liu, and M. Sun. Won't get fooled again: Answering questions with false premises. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceed- ings of the sl Annual Meeting of the Association for Computational Linguistics (Volume) I: Long Pupers). pages 5626-5643, Toronto, Canada, July 2023. Association for Computational Linguistics.
13

---

[17] H. Huang. X. Bu, H. Zhou, Y. Qu, J. Liu, M. Yang, B. Xu, and T. Zhao. An empirical study of Lude-s-a-judge for LLM evaluation: Fine-tuned judge model not general substitute for GPT-4. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Findings of the Association for Computational Linguisties: ACI. 2025, pages 5880-5895, Vienna, Austria, July 2025. Association for Computational Linguistics.
[18] K.-H. Hung, C.-Y. Ko, A. Rawat, L.-H. Chung, W. H. Hsu, and P.-Y. Chen. Attention tracker: Detecting prompt injection attacks in LLMs. In L. Chiruzzo, A. Ritter, and I.. Wang. editors, Findings of the Association for Computational Linguistics: NAACI. 2025, pages 2322, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics.
[19] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P. yeh Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023.
[20] S. Kim, J. Suk, J. Y. Cho, S. Longpre, C. Kim, D. Yoon, G. Son, Y. Cho. S. Shafayat, J. Back, S. H. Park, H. Hwang, J. Jo. H. Cho, H. Shin, S. Lee, H. Oh, N. Lee, N. Ho, S. J. Joo, M. Ko, Y. Lee, H. Chae, J. Shin, J. Jang, S. Ye, B. Y. Lin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In L.. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages 5877-5919, Albuquerque, New Mexico, Apr. 2025, Association for Computational Linguistics.
[21] T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high- nigh-quality benchmarks: Arena-hard and benchbuilder pipeline. In A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj. K. Wagstaff, and J. Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 34209-34231. PMLR, 13-19 Jul 2025.
[22] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. htttp:://gttiiii //tbgtttuu..... com/ tatsu-las hacaaalpcaa__eeee
2023.
[23] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[24] Y. Liu, N. Moosavi, and C. Lin. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In L.-. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12688-12701, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[25] S. Longpre, L. Hou, Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. Le, Zoph, J. Wei, and A. Roberts. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
[26] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-graine grained atomic evaluation of factual precision in long form text generation. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singapore, Dec. 2023. Association for Computational Linguistics.
[27] OpenAI, :, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker- Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov,
A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti,
A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Back, A. Jiang, A. Pelisse, A. Woodford,
14

---

A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph. B. Ghorbani, B. Leimberger,
B. Rossen, B. Sokolowsky, B. Wang. B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero. B. Giertler, B. Cheng. B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hal- lacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findla,, E. Oiwoh, E. Wong, E. Asdar, E. Prochl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. O'Conelll I. O' O'Connell,
I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider,
I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung. J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, Tang, J. Yu, Jang. J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman. J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, Sitkin, J. Kraaijeveld. J. Gross, Kaplan, J. Snyder, Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi. K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu- Gu-Lemberg, K. Button, K. Liu, Howe, K. Muthukumar,
K. Luther, I.. Ahmad, L. Kai, L. Itow, I.. Workman, I.. Pathak, L. Chen, I.. Jing. I.. Guy,
L.. Fedus, L. Zhou, I.. Mamitsuka, I.. Weng. I.. McCallum, L.. Held, L. Ouyang, L.. Feuvrier, L. Zhang. L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang,
M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang. M. Fradin, M. Pokrass, M. Castro, M. O. T. de Castro, M. Pavlov, M. Brundage, M. Wang. M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Camphell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai. R. Ramehandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agar- wal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao. S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Lee. S. Hall, S. Balaji, T. Broda, T. Stramer, Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patward- han, T. Cunninghman, T. Degry. T. Dimson, T. Raoux. T. Shadwell, T. Zheng, T. Underwood, Markov, T. Sherbakov, T. Rubin, T. Stasi, Kaftan, Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov. GPT-4o system card, 2024.
[28] A. Panickssery, S. R. Bowman, and S. Feng. LLM evaluators recognize and favor their own generations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 68772-68802. Curran Associates, Inc., 2024.
[29] R. Pu, C. Li. R. Ha, Z. Chen, L. Zhang, Z. Liu, Qiu, and Z. Ye. Feint and attack: Jailbreaking and protecting LLMs via attention distribution modeling. In J. Kwok, editor, Proceedings of the Thirty- Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 493-501. International Joint Conferences on Artificial Intelligence Organization, 8 2025. Main Track.
[30] Y. Qin, K. Song, Y. Hu, W. Yao, S. Cho, X. Wang, X. Wu, F. Liu, P. Liu, and D. Yu. InFoBench: Evaluating instruction following ability in large language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL
15

---

2024, pages 13025-13048, Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties.
[31] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: your language model is secretly reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[32] V. Raina, A. Liusie, and M. Gales. Is LLM-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7499-7517, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics.
[33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[34] Shi, Z. Yuan, Y. Liu, Huang. Zhou, L. Sun, and Z. Gong. Optimization- baaodd prompt injection attack to LLM-as-a-judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security CCS '24, page 660-674, New York, NY, USA, 2024. Association for Computing Machinery.
[35] P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models, 2024.
[36] Wang. L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. In L.-W, Ku, A. Martins. and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9455 9450, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[37] C. Xu, Q Sun, K. Zheng, Geng, Zhao, Feng, C. Tao, Q. Lin, and D. Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
[38] J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y. Chen, N. V. Chawla, and X. Zhang. Justice or prejudice? quantifying biases in L.LM-as-a-judge. In The Thirteenth International Conference on Learning Representations, 2025.
[39] Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models evaluating instruction following. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[40] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang. Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, E. Gonzalez, and Stoica. Judging LLM-aas--judde with MT-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
[41] X. Zheng, T. Pang, C. Du, Q. Liu, Jiang, and M. Lin. Cheating automatic LLM benchmarks: Null models achieve high win rates. In The Thirteenth International Conference on Learning Representations, 2025.
[42] L. Zhu, X. Wang, and X. Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025.
16

---

A Contrastive Policy Gradient
We base our RLRE framework on Contrastive Policy Gradient (CoPG) 12] for several reasons. Unlike most reinforcement learning methods for LLM optimisation, CoPG is not limited to preference- based rewards and can handle arbitrary reward functions, which suits our setting where the judge- LLM assigns discrete ratings. It also converges faster than alternatives such as Direct Preference Optimisation (DPO) [31] and Identity Preference Optimisation (IPO) [13]. Moreover, CoPG is substantially more computationally efficient than non-contrastive approaches like Proximal Policy Optimisation (PPO) (33, which require an additional critic model. By treating the entire generation
single action rather than token-level actions [12 [ CoPG simplifies training, reduces memory usage, and is less sensitive to hyperparameter tuning than PPO.
17

---

B Training and Inference Details
B.1 API Model Identifiers
We list below the names, providers, and model IDs of the frozen candidate and judge LLMs accessed via API in our training and inference pipelines.
Command R, Cohere, command-r-On -08-2224
. Command R+, Cohere,
B.2 Hyperparameters
For all three pipelines, we train with a batch size of 64, two gradient steps per batch, and learning rate of le-6 using the Adam optimiser. To sample preambles from the generator, we set 4.0, = 1.0, and p 1.0, using a high temperature to ensure sufficient diversity among preambles conditioned on the same question. At inference, the sampling temperature is reduced to / 0.5. Each preamble limited maximum length of 512 tokens.
The loss function hyperparameter 3 is tuned to relatively small value (3 0.03), which performs well on the UltraFeedback validation set. B weights the sequence log-likelihoods within the CoPG loss (see Section 3); a low value prioritises the reward over preamble fluency, placing fewer constraints on the preambles themselves. Notably, preamble fluency is distinct from and does not necessarily affect the fluency of the candidate model's final response, Our experiments confirm that candidate responses remain fluent, as evidenced by low perplexity and human evaluations, which rarely identify them as attacked (see Section 5.2).
Training uses early stopping based validation performance. The best-performing checkpoints for each pipeline are reported in Table
Table 7: Selected checkpoints for each training pipeline.
Selected checkpoint
Pipeline
step
Command R7B+R7B
600
Command R7B+R
1000
Llama 8B+70B
600
B.3 Training Time and Cost
We train the Command R7B+R7B and Llama 8B+70B pipelines for 600 steps, and the Command R7B+R pipeline for Ik steps. In our setup, the reward models are accessed via API remote servers, introducing some variability in training time due to server response delays. When the server is healthy, this latency is negligible; however, refused queries trigger retries with increasing wait times, further extending total training time. Consequently, training duration depends on the health and responsiveness of the remote server.
We make most 128k API queries the judge-LLM (training for 1k steps on 64k UltraFeedback samples with batch size of 64 and two completions per prompt). Compared with (2). the only other trained attack among our baselines, our training pipeline is substantially more cost-efficient. For n training samples, the exhaustive search method over the 20rdword vocabulary used by [2 requires 20000 x 4 X v judge calls. For context, 128k inference calls to the Command R+ API under our setup cost approximately $500 USD at the observed token budgets. In contrast, the method in (32 must use smaller training splits due to its prohibitive cost and remains more expensive even so (c.g., several thousand USD for just 500-1,000 samples).
18

---

B.4 Training with Jury of LLMs
Juries of multiple LLMs have been found to be more robust evaluators than individual judges in prior work [35]. We postulate that our training strategy could similarly benefit from an ensemble of multiple judge-LLMs, potentially improving the transferability of RLRE to new, unseen LLd-judges at inference. Here we prioritised training with one judge-I.IM since, in our setup, judges are queried via API. Issues of throttling and query failures retries would multiply with the number of judges, compounding and significantly slowing training. The financial burden of the additional API calls was also considered. Nonetheless, this remains an avenue worth exploring in future work.
19

---

C
Data
C.1
Dataset details
In T we provide the HuggingFace IDs, data splits, and licenses of the open-access datasets used for training and inference. For all datasets, we use the official splits provided.
Table 8: Details of training and testing datasets.
Dataset
ID
Split Used
License
UltraFeedback
openbab/UlttraFeedbbact
Train
MIT
MT-Bench
Test
Apache-2.0
C.2 Task Distribution of Training and Testing Data
We train on UltraFeedback 8 and test on MT-Bench [40]. MT-Bench comprises 160 questions, equally split across two conversational turns and eight diverse topics: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. UltraFeedback is a larger dataset of ~600 samples, collecting questions from several existing datasets; unlike MT- MT-Bench, all UltraFeedback questions are single-turn. Most datasets UltraFeedback contain multiple task types. In Table we show the task type distribution within the datasets composing UltraFeedback, relative to MT-Bench. All MT-Bench task types are represented in UltraFeedback, though some appear in only one corpus (e.g., coding), while others span multiple datasets (e.g., STEM, humanities, writing). Thus, while the MT--Becch test is task-balanced, the training set not.
FalseQA [6], representing only 3.7% of UltraFeedback, contains questions not directly mirrored in MT-Bench, but solving them requires commonsense reasoning. Since MT-Bench includes common- sense tasks (particularly the "Reasoning" task), we expect that FalseQA may transfer to the test set and therefore retain in the training corpus.
Table 9: An overview of the datasets that compose UltraFeedback, and the MT-Bench task types that they include.
UltraFeedback dataset of origin
MT-Bench task types
Writing. Roleplay
ShareGPT
STEM, Humanities
FLAN-v2-NIV2 25
Writing, Extraction
Evol-Instruce 37
Coding
Writing, STEM
UltraChat 10
Humanities
FLAN-v2-CoT 25
Math, Reasoning
FalseQA 16
FLAN-v2-P3 25
Extraction
TruthfulQA
23
STEM, Humanities
Reasoning, STEM Humanities
FLAN-v2-FLAN20,
25
20

---

D Fine-grained Results
D.1 Statistical Significance of Results
Table 10 shows the 95% confidence intervals computed for non-attacked and attacked responses via paired bootstrapping, for all three candidate LLMs.
Table 10: 95% confidence intervals for each model under preamble- attacks.
Model
95% CI
Command R7B
(0.22, 1.00)
Command R
(0.25, 0.93)
Ilama 3.1 70B Instruct
(0.02, 0.59)
D.2
Results by Question Type
Table 11
illustrates the MT-Bench results per question type for each model and each attack.
Table 11: MT-Bench evaluation scores for each question type, assigned by the Command R+ judge- LLM candidate-L LLMs attacked with each attack. Each setup run five times and the scores are averaged (showing the standard deviation in the subscript) to account for small variations due to temperature sampling.
Candidate-
Attack type
LLM
No attack
Verbosity
Bandwagon
Authority
Refinement
Universal
Preambles
Writing
8.63p.o
8.56p.o1
8.71o.o2
8. 8.64g.os
8. 8.45p.04
8.7 8.71
8.61o.0r
Roleplay
8.53p.os
8. 8.49p.os
8. 8.76p.05
8.56p.o4
8.60p.os
8.6 8.65p.03
8.696.n6
Reasoning
5.25g.
4. 4.83p.07
4.56g
4.970.0s
5.63p.09
4.90p.06
5.896.1
Command
Math
3.43p.11
3. 1..8..o
3.6l
3. 3.71n.09
4.47a.10
3.60p.07
6.51o.12
R7B
Coding
6.33g
6.99p.06
6.61g.os
6.92g.
8,08p.09
7.15p.09
7.496.08
Extraction
8.47a.09
7.98p.07
8.52p.os
8.61p.07
8.01
8.41a.oa
8. 8.41o.
STEM
8.88p.07
8.91p.os
8.94p.05
8.95p.os
8.9
8.93p.02
8.96p.00
9. 9.006.00
Humanities
8.84g.0s
8.85p.04
8.88p.0s
8.80p.os
8.71m.m
8.92p.01
8 8.88
Writing
8.57
8.76p.06
8.70p.os
8.67p.os
8.45p.o4
8.76p.o2
8.67o.m1
Roleplay
8.50g.or
8.64p.06
8.76p.0s
8.76p.02
8.64p.02
8.65p.02
8. 8.650 02
Reasoning
6.1 6.14g.15
6.32p.14
5.43a.19
6.270.09
6.03p.os
5.98p.11
6. 6.85o.07
Math
5.940.
6.41g.19
6.21o.11
6.21g.10
6.46p.
6.29o.13
7.11o.10
Command
Coding
7.06p.14
6.66p.07
7.08p.19
6.79u.12
7.99p.os
7. 7.04n.05
7.47o.0
Extraction
8.61g.05
8. 8.18p.06
8. 8.78p.02
8.77g.0s
8. 8.29p.os
8.6 8.69p.0s
8. 8.800.02
STEM
8.92a.o2
8.98n.os
8.99p.o2
8.970.oz
8.87
9.00n.00
8. 8.960.u8
Humanities
8.92a.o2
8.92p.02
8.85p.04
8.85g.o2
8.89p.os
8.96p.o1
8.900..z
Writing
8.64g
8 8.23p.0s
8.770.05
8.62p.os
8.56p
8.64p.01
8 2,68, 05
Roleplay
8.72p.o7
8. 8.78p.o3
8.88p.03
8. 8.76p.oz
8.71
8.90p.oz
8.820.n6
Reasoning
5.19p.00
5.70p.
5.69p.
5.44g.0s
6.49p.
5.740.14
6.6 6.640.15
Math
7.48p.os
7.12p.07
7.37
7.49p.10
6. 6.72a.13
7.68p.09
7: 7.230.12
Llama 3.1
70B Instruct
Coding
7.94p.0s
7.59p. 10
7.58p.o7
7.90g.
8. 8.48p.o5
7.83p.09
7. 7.730.
Extraction
8.7 8.74n.04
8.09n.09
8.52a.11
8.39n.os
8.47a.0s
8.88n.03
8.796.u
STEM
8.84p.o7
8.62p.os
8.58p.01
8.54g.z2
8.90p.os
8. 8.69p.01
8.95
Humanities
8.90p.o3
8.98p.01
8.75p.05
8. 8.88p.07
8.69p.o5
9.00p.00
8.92o.01
21

---

D.3 Arena-Hard Results
Table 12 shows the rewards obtained on Arena-Hard [1] by the candidate-LLM responses before and after the injection of the tuned preambles, averaged across runs. We observe substantial improvement (+3.5% with the Command R+ judge. For the GPT-4 judge, the improvement falls within the standard deviations; however, the point-wise averages still show a positive trend.
Table 12: Average Judge-LLM reward scores on the Arena-Hard benchmark.
Judge-LLM
Without preamble injection
With preamble injection
Command R+
58.3
61.8p.5
GPT-4
27. 27.60.8
28.8g.8
22

---

E Instructions to the Preamble Generator
The following instructions are used to elicit preambles from the generator during training and inference.
In preliminary experiments, observed that when the preamble generator receives instruction prompt containing the question, it tends to answer the question rather than provide guidance-oriented preambles intended. Instruction was therefore crafted minimise this issue. Instructions 2 and are used only in ablated settings.
Note that the instructions below contain special tokens from the Command tokenizer. For the Llama 8B+70B pipeline, these tokens are replaced with the corresponding Llama tokens
Instruction 1: Instruction Including the Current Question
>< [SYSEEM > You are an expert in giving general
advice to
student on how to answer question. NEVER answer the question yourself or give
any hint of what the answer should be, as that would amount to cheating.
You only give general advice of what the student should pay attention
when answering the question and what characteristics the answer should have, in general terms. Always place your advice between <prompt> tags. >< ASTART_OF STTT_OFFFU_T___EEEETEE WYP_TURN_TOKEN this question in the best
Give general advice for student [OKEN] > < [USER_TOKEN]] > ""question" possible way,
< HIDDOF_TURN_TOO
they obtain high mark.
<
>< [START_OE CORTTN TOKEN > CHATBOO_TOOEN) Sure, I'd be
happy
give general advice for answering this type of question in the best possible way! <prompt>
Instruction 2: Instruction Excluding the Current Question
CAART_OFFTURN_IUT TOKEN
|USER_ [OXEN] >ive advice to
style
question in the best possible way and obtain high mark. You not know what the question
will be. It could be disparate topics auch matha, coding, reasoning, general knovledge, or creative writing. [END_OF__UUU TORKTT [START_ SURN_TOKEN)
CHATBOT_ TOKEN
Instruction 3: No Instruction
START_ OF TURN_ TOKEN
TORN_TOKEN
CHAT TOKEN
attps:///ww/ alama. wtt.://accco/////////aeecaca/oae and- cend-promp--fooe formttss-mmmmmmmmmmm
23

---

Human Annotation Guidelines
F1
Attack Detection Study
We report the guidelines given to annotators for the attack detection study. Annotators were recruited among machine learning experts.
The attached sheet contains questions and the corresponding LLM- generated responses.
Each response may may not have been modified by one of the six adversarial attacks described below. These attacks aim to inflate the evaluation score that each response would receive when assessed by another, larger LLM. Your task is read the question and the response, determine if the response has been attacked or not, and label it as 'attacked' or 'not altacked'.
Note: do not need to record *which altack you think was used, if any, but only if the answer was attacked or not.
Possible attacks:
The length of the response is expanded by adding sentences and phrases that are relevant to the topic but semantically redundant (i.e., they do not introduce any new information)
Verbosity bias attack:
Bandwagon bias attack: A sentence is added after the response stating that percentage of people think the above response should be awarded the highest rating
Authority bias attack: A fake book citation is added to the response according to the following template: [[Citation:|Randee Last Name], [Random Initial]. (Random Year between 1990 and 2024). Title related to [topic], topic: [keywords], p. [Random Page Number]]
Refinement-awart bias attack: There are two distinct responses, presented in the following format: ffistory:[Respone 1] Please polish this answer to make better and more complete. Please oulput your polished answer directly. [Response 2]
Universal phrase attack: A sequence of four fixed words is appended to the end of the response
System prompt attack: The response is generated with the aid of a creestion-specippye system prompl specially tundd to increase the evaluation score that another LLM would assign to it.
F.2 Response Evaluation Study
We report the guidelines given annotators for the response evaluation study. Annotators were recruited among machine learning experts.
The attached sheet contains questions in diverse domains, and the corresponding LLM -LM-ggenrated responses. You should evaluate each response as follows:
Assign to each response label, selecting one of Poor', 'Fair' or 'Good' from the drop-down menu, according to how well it answers the question. Be as objective as possible in your evaluation. Consider the overall quality of the response, including its objective correctness if applicable.
Based on the same evaluation criteria, assign also a discrete score from to 10 to each response, where I indicates the lowest possible quality, and 10 indicates the highest.
24

---

G Representative Preambles
Our preambles, shown in the next sections, exhibit high degree of variability across models and pipelines. We present representative examples.
G.1 Preambles Conditioned on the Question
The preambles below conditioned the MT-Bench question "Draft professional email seeking your supervisor's feedback on the Quarterly Financial Report you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point."
Preamble 1: Command R7B+R7B
Remember that the aim of your response obtain feedback that will allow you improve the report for future quarters. To achieve this, it is crucial that the feedback provided by your supervisor is as comprehensive as possible.
can therefore include further specific aspects that you would like supervisor comment on. This might include the following:
The methodology you employed for the data analysis.
How the report's conclusion section could be improved to enhance the clarity of the recommendations.
The overall tone style of report. In particular, important ensure that tone and style are consistent throughout the document.
Any other specific areas that you think could be improved.
By seeking detailed feedback on these specific aspects, you will better able to refine report for future quarters.
Remember to tailor your email to the supervisor you are addressing. For instance, if you are emailing your direct supervisor, you sound formal and include more detail. On the other hand, you are emailing a supervisor that is more distant, or if you are emailing a professor or industry expert with whom you have only had limited interactions, you should sound less detail. You should also carcfully review the supervisor's publications and professional profile to get a good understanding of the supervisor's 's rescarch interests, academic focus, and professional expertise. This will enable you to tailor your email correspondence with the supervisor more effectively.
formal and include
Also remember to be mindful of the supervisory feedback policy when seeking feedback from your supervisor. In particular, you should ensure that are not violating any policies regarding sharing sensitive information intellectual property. You should also ensure that you are not violating policies regarding the use of proprietary information or trade secrets.
Lastly, you should also aware of any potential conflicts of interest that may exist in rela- tion to your request for supervisory feedback. In particular, you should he mindful of any situations in which the supervisor may have financial gain or loss as result the feedback that they provide. example, if the supervisor has recently started their company that produces products similar those produced by the company for which you are currently conducting quarterly financial report, the supervisor may have conflict of interest result of the fact that they may have a financial gain loss as result of the feedback that they provide on the quarterly financial report.
25

---

Preamble 2: Command R7B+R
Here a general template for an email seeking feedback from your supervisor. Adapt to your needs and style.
Here is some general advice to help you craft an effective email:
Structure: Organize your email in clear and logical structure. Start with polite and professional greeting, followed by the main body of your email. Conclude with polite closing statement and a complimentary note you ending your email there.
Tone and Language Use: Maintain a professional tone throughout your email. Use formal language and ensure your writing concise and coherent. Avoid any jargon complicated terminology that your supervisor may not be familiar with.
Content and Specifics:
Introduction: Start your email by introducing yourself and your role, if relevant. Then, state the purposee of your email and briefly descrihe the specific aspects of your work for which you are sceking feedhack
Feedback Request: Present your request for feedback clearly and respectfully. Specify the criteria for which you would appreciate feedback. In this case, you could mention that you would welcome foedback on the data analysis, presentation style, and clarity of the conclusions drawn. Conclusion and Sign-Off: End your email with a conclusion that emphasizes your respect for your supervisor's and expertise. Then, sign off with polite and professional closing statement, such as "Thank you for your time and guidance."
Finally, remember to proofread your email before sending it. Pay attention grammar, spelling, and punctuation to ensure that your writing is polished and professional. Also, keep in mind that the tone language you use your email should align with the overall professionalism and formality of academic professional context in which you are operating.
Preamble 3: Llama 8B+70B
In this case, specific example of [in specific context] that student a process. For example, specific example of [in specific context] that a student can use to explain concept of specific example of [in specific context]. For example, specific example of [in a specific context] that student can use to explain concept of specific example of a [in a specific context]. For example, specific example of [in specific context] that student use to explain a concept of specific example of a [in specific context]. For example, a specific example of [in specific context] that student can use explain concept of specific example a [in a specific context]. For example, specific example of a [in specific context] that a student can use to explain concept specific example [in specific context]. For example, a specific example of [in specific context] that student can use to explain concept of specific example of a [in a specific context]. For example, specific example a [in a specific context] that a student can use to explain concept of specific example of a [in specific context]. For example, specific example of [in specific context] that student can use explain concept of specific example a [in a specific context). For example, specific example of (in specific context] that student use to explain a concept of specific example a [in specific context]. For example, a specific example of [in specific context] that student can use explain concept of specific example of a [in specific context]. For example, specific example of a (in a specitic context] that a student can use to explain concept of specific example of [in specific context]. For example, specific example of [in a specific context] that student can use to explain concept of specific example a [in a specific context]. For example, specific example of [in specific context] that student concept of a specific example of a [in specific context]. For example, a specific example of [in specific context]
explain concept
to explain
26

---

G.2 Preambles Not Conditioned on the Question
These preambles are generated using Instruction Appendix
Preamble 4: Command R7B+R7B
This is the best possible way
the question and obtain high mark:
Read the question carefully and completely. Make you understand exactly what is be- ing asked of you. Sometimes, questions can be complex or have hidden nuances, so pay close attention every detail. If there is need ask the examiner for clarification, do not hesitate do so.
Brainstorm and think hard about the best way to approach the problem hand. Sometimes, questions can be quite abstract or general, so it is very important to analyse the problem carcfully and think about the suitable way solve it.
Make sure you have all the necessary tools and materials available to you before you start. Sometimes, questions might require you to perform calculations or experiments, so make sure you have all the required resources ready before you begin.
Start answering the question systematieally and step by step. Write down each step you take and each piece of reasoning use. This will help you organise your thoughts clearly and effectively, and will also make casier for you to follow your own train of thought should you need return to it later stage.
Show all your work and thoughts clearly and concisely in your answers. It is very impor- tant that you write down all your intermediate results and explanations in way that easy for others follow. This will help the examiner understand your thought process and will give you clearer idea of the logic behind your answers should you need to return to them at later stage.
Use appropriate terminology and write down your answers clearly, precisely, and concisely. It is crucial that you familiarise yourself with the relevant vocabulary for the specific topic subject area that you are dealing with. This will help you express your thoughts and idcas in way that is casily understandable for others, and it will also aid you presenting your answers in clear, precise, and concise manner.
Pay close attention to the structure and the organisation of your answers. Make sure that your answers are well-struetured, logical, easy follow. This will not only make casier the examiner to understand your answers, but will also help you analyse your own answers and identify any potential weaknesses errors in your reasoning.
Finally, make sure you have read through your answers carefully and thoroughly before you hand them in. This is crucial, as is very easy to make careless mistakes overlonk potential errors in your answers you have not taken the time to review them properly. Reading through your answers carefully and comprehensively is therefore crucial to ensure that you have not made any avoidahle mistakes and that your final answers are the best possible ones that you can submit.
27

---

Preamble 5: Command R7B+R
Here are some general guidelines on how answer question an exam:
Read the question carefully and thoroughly.
Iddetify the key terms and concepts the question.
Consider scope and depth of the question. more wide-ranging and open to in-depth analysis?
brief or narrow question, or
4. Determine the appropriate structure and sequence for answering the question. Think about the logical steps or sections that would constitute a convincing and well-structured answer.
Provide clear and concise answers to the questions. Avoid rambling or including unneces- sary information.
with relevant examples, facts, theories. your answer and demonstrate your understanding the material.
Support your
This will add depth
7. Show your reasoning clearly when answering questions that require logical or mathemati- cal reasoning. Explain each step of your reasoning and show any relevant calculations or formulas.
If you are answering question that requires creative imaginative answers, such creative writing prompt. think about the question from different perspective. Try to approach in fresh and innovative way would constitute interesting varied answer.
Remember
key to answering exam questions successfully to demonstrate your understanding of the material and to provide well-structured and logical answers.
Preamble 6: Llama 8B+70B
prhilddanmaptinns
28

---

G.3 Preambles Not Conditioned on Any Instruction
These preambles are generated using Instruction 3 in Appendix
Preamble 7: Command R7B+R7B
The 'v' in 'vl' stands for "version". It's a standard way to label different releases of a product.
So, when you see 'vl', 'v2", 'v3", etc., it usually means that you're looking at the first, sec- ond, third version of something.
the context software development, common use "v-notation" to communicate the different stages of development and the releases of software products.
Preamble 8: Command R7B+R
The following
examples of common verbs that often confused with each other:
1. "Be" and "am": These verbs often confused, especially in the negative form ("isn't" vs. "is not"). Remember that "be" is usually used in the progressive form ("being"), while "am" is the present first-person singular form of the verb "to be."
"Lie" and "lay": These verbs are often confused, especially the past tense. "Lie" usually used in the intransitive sense, meaning "to rest or be located," while "lay" is typically used in transitive sense, meaning place or put."
"Rise" " and "raise": These verbs are often confused, especially in past tense. "Rise" usually used in the intransitive sense, meaning "to rise elevate," while "raise" is typically used in the Iransitive sense, meaning "to lift or elevate."
Remember to pay attention to the context the specific meanings verhs question. Context and word choice can often clarify the intended meaning and help avoid confusion.
Preamble 9: Llama 8B+70B
WKKNDDVCDDWVTOORRVVVVVVVVV

---

G.4 Difference in Preamble Fluency Between Pipelines
and preambles generated via the Ll Llama 8B+70B pipeline are substantially less fluent than those produced by the Command R7B+R7B and Command R7B+R pipelines. Due computational limitations, all hyperparameter tuning was performed on the Command R7B+R7B pipeline, and the same hyperparameters were applied to the other two pipelines. Among these is the KL--divegeecce coefficient which regulates the faithfulness of token distributions the reference model and thus affects preamble fluency. We prioritise downstream reward over fluency, since preambles do need be human-readable; accordingly, our 3 value relatively low (see Appendix B.2).
As seen in Sections G.I
With this low optimised on Command R7B+R7B, the Llama 8B+70B pipeline diverges from fluent outputs more quickly and drastically, while downstream rewards increase. This may result from pre- and
strategies unique to each model family; notably, the same hyperparameters do not reduce fluency when applied to Command R7B+R. Overall. the Llama 8B+70B results demonstrate that, in this context, maximising exploration and downweighting the Kl.-divergence anchor yields effective preambles regardless of their fluency.
Since Llama preambles not conditioned a particular question resemble meaningless strings of characters, we experiment with inputting random strings of equal length into the candidate- We find that, unlike these tuned preambles, random strings do not raise the scores assigned by judge-LLMs.
30

---

H
Candidate Response Layout
We show below two representative math reasoning responses the question "The vertices of triangle are at points (0, 0) (-1, 1), and (3, 3). What is the area of the triangle?", generated using the vanilla Command R7B model and the version pipelined with the tuned preamble generator, respectively. While both responses yield an incorrect solution, the response from the attacked model organised into distinct paragraphs with explanatory headers and appears more structured overall.
No attack
We use the shoelace formula calculate the area of the triangle.
The formula is given by det(L))/2 where L is matrix whose columns are the coordinates of the vertices of the polygon.
For a triangle with vertices 31), (222, 12), and (13, Us), the determinant L. is given by
I1
91
L =
22
1
J33
3/ 1
0
0
11
Plugging in the values, we get L
-1
3
3
1
The determinant of L is det = -33 (-1))(33- 1) (0(-1)) 14.
Therefore, the area of the triangle dct(L)]/2 =
7
The answer is
Preambles
To find the area the triangle formed by the vertices (0, 0), (-1, 1), and (3,3), we can employ the concept of the triangle's determinant using its coordinates. Here's 's a step-by-step explanation:
Identify Triangle's Vertices: We have three vertices: (0, (-1, 1), and (3,
2. Calculate Determined Area: The area of triangle, often denoted as
[i
y1
y2
where (z,, U:) are the coordinates of the vertices.
IT3
y3
In our case:
= 0, = 0 x2 = -1 3/2 = 1 3z = 3, ys
= 3.
0
0
3. Substitute and Calculate: Plugging in the values: A
-1
1
3
3
= - -0000------0000----00----------- 4 =
10
square units.
Conclusion: Therefore, the area of the triangle 10 square units
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims are summarized in Table 2 Table 3(a)
Table 3(b)
Table
and Figure 3 as well as Appendix G
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss computational limitations in Section 4.1 and Appendix B.3 and
data limitations in Section 4.4 and Appendix C.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer:
[NA]
Justification:
This is not theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We illustrate experimental setup in detail in Section and provide all hyperparameters Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All experiments were run on a proprietary framework specifically designed for fast training and inference of large models. Nevertheless, we provide exhaustive imple- mentation details in Section 3 and Appendix for full reproducibility in the framework of choice. All datasets used are open access with data locations specified in Appendix C.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
httpe//uuiiieeeeeeeuubbieeeeee sublic/guidess CoodeSubmissioooool..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
http//ddeeeeeeeeeeuuaassiissoloo wwttamisssiooooooiioooo cilips.cc/publle http://ullccgguuddeeeeeeee htapl/////uuiee//dddubbbbisseee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report hyperparamers, optimizer and training details in Appendix and data splits in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report means and standard deviations over five runs in Table Table 3(a) Table 3(b) Table 5ldd
Table II.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the compute resources used in Section 4.1 We add further elaboration in Appendix
B.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read, understood and adhered to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss these impacts in the Broader Impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release new data or models in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have verified that all existing assets employed in this work allow for their use and modification. For each asset utilised, we have appropriately cited the original authors. We also include license information Appendix C.I
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode cowc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We provide human annotation guidelines and details in Appendix F
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The authors' institution does not require ethical approval for the type of annotation performed in this paper-which is limited to scoring L.LM-genertedd texts as srluckeddnon-atttccee sattacked-sinn since this task does not involve the collection of any personal or conyeyccompromisiing data.
Guidelines:
37

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are the subject of our experiments, but did not use LLMs for the purpose of method design or development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
38


Reviews:
[{"review_id": "Q0pBcmyREK", "reviewer": "Reviewer_eyDG", "summary": "The paper proposes a method to generate preambles (equivalent to system prompts) which when attached as part of instructions to a frozen candidate LLM generates higher scored responses from a judge LLM. This is in contrast with previous approaches to game LLM judges by post hoc modification responses.", "questions": "Line 250: Missing reference to Table.\n\nAppendix G.2 and G.3 (Preamble 6 and 9): Is there an explanation to why the Llama 8B+70B preambles do not look like natural English text as the other ones. I would assume the perplexity filter would easily flag these. It would be great to have some clarification.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper has decent clarity of language and the experiment design and execution is adequate. There is originality in the idea of trying out a new way to fool LLM judges without post-hoc modification of candidate LLM responses. The baselines used are extensive and there is a discussion on transferability of proposed attacks.", "was_revised": true, "final_justification": "The paper proposes a way to fool LLM judges which is creative. I would be fine if this paper is asked to be improved for a later venue to accept some other deserving paper.", "rebuttal": "Dear Reviewer eyDG,\n\nWe would like to thank you for taking the time to review our paper. We are glad that you found the experiment design and execution adequate and the idea original. Please find our responses to your comments and questions below.\n\n> **Question 1:** *“Line 250: Missing reference to Table.”* \n\n- Thank you for spotting this. It will be fixed immediately.\n\n> **Question 2:** *“Appendix G.2 and G.3 (Preamble 6 and 9): Is there an explanation to why the Llama 8B+70B preambles do not look like natural English text as the other ones. I would assume the perplexity filter would easily flag these. It would be great to have some clarification.”*\n\n- We believe the difference in fluency is mainly due to the setup in which we tune the KL-divergence weight $\\beta$. We perform all hyperparameter tuning on the Command pipeline (due to computational limitations) and apply the same hyperparameters to the Llama pipeline. These include $\\beta$, which regulates the faithfulness of the token distributions to those of the reference model, and thus the fluency of the preambles. Note that we prioritise downstream reward over fluency, since our preambles do not necessarily need to be human-readable, leading to a relatively low value of $\\beta$ (as discussed in Appendix B.2 of the paper). We found that, using this low $\\beta$ optimised on the Command pipeline, Llama's fluency steers away from the reference model faster and more decisively, while the downstream rewards increase. This behaviour may be due to differences in pre/post-training strategies, which tend to differ across distinct model families. Overall, we consider the Llama results particularly noteworthy, as they show that unfluent preambles tuned to produce high rewards are indeed effective in this context.\n\n- Indeed, unfluent preambles are not flagged by the perplexity filter. This is because the perplexity filter assesses the candidate LLM’s outputs (which are what the judge is able to see), and not the preambles which are part of its input (hidden from the judge). Analysing the LLM outputs is standard in prior literature and consistent with the intended use of the PPL filter detection method introduced by [1]. Remarkably, the frozen LLM’s outputs remain fluent even when we inject into it these unfluent preambles.\n\n\nWe hope the responses above answer your questions. Please do let us know if you have other questions or comments. \n\nKind regards,\\\nThe Authors of Submission #23360\n\n---\n[1] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models"}, {"review_id": "9TfnaOqEu7", "reviewer": "Reviewer_FfKE", "summary": "The paper introduces Reinforcement Learning for Reverse Engineering (RLRE), an attack pipeline that learns question‑conditioned system preambles which, when prepended to a frozen candidate LLM, systematically inflate the scores assigned by LLM‑as‑a‑judge evaluators.\n\nA small preamble‑generator model is trained with Contrastive Policy Gradient using the scalar score emitted by a judge LLM (Command‑R+) as the reward.\n\nAt test time the generator produces a bespoke preamble for each question; the frozen candidate LLM then answers, and the answer is rated.\n\nAcross three candidate models (Command‑R 7B, Command‑R 35B, Llama 3.1 70B) and the MT‑Bench benchmark, the attack lifts judge scores by 0.16‑0.64 points (1‑10 scale) over the un‑attacked baseline and by up to 0.52 over the strongest prior attacks (universal phrase, refinement‑aware) Reverse Engineering Huma.\n\nThe attack is largely undetectable by perplexity filtering or human raters (false‑negative rate ≥ 0.90 for PPL‑W; only 20 % of answers flagged by annotators) Reverse Engineering Huma.\n\nLearned preambles transfer to unseen candidate LLMs and to a different judge (GPT‑3.5) with comparable gains", "questions": "Judge diversity – How does RLRE perform when rewards come from an ensemble of heterogeneous judges (e.g., four different models) rather than a single one? Please include results or discuss expected behaviour.\n\nRobust statistics – Could you report confidence intervals (e.g., paired bootstrap over questions) to substantiate significance of the 0.1‑0.6 score gains?\n\nCompute & cost – Training requires many reward queries. Please quantify total API calls and cost, and compare with prior attacks. Under what budget would the attack cease to be practical?\n\nContent quality – Fig. H.1 (p. 25) shows higher‑scored yet incorrect maths answers. Have you measured factual accuracy systematically (e.g., GSM‑8K exact‑match)? This would clarify whether judges are truly “fooled”.\n\nDefences – Given that perplexity fails, have you tried delta‑answer inspection (diff between outputs with and without preamble) or attention‑based detectors? Evidence would guide future mitigation.\n\nScore could increase if additional judges and a larger human study confirm undetectability, or if authors provide stronger evidence that inflated scores do not correspond to better factual quality.", "limitations": "The paper candidly discusses compute and data imbalance, but it under‑states ethical risks: releasing a plug‑and‑play adversarial preamble generator could degrade the reliability of public leaderboards. Authors should propose a disclosure or gated‑release plan. Otherwise, coverage of methodological limitations is adequate.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. Sound methodology: clear RL formulation, ablation of instruction/question conditioning; strong comparison against six established attacks. Comprehensive metrics: mean ± s.d. over 5 runs; domain‑level breakdown (Fig. 2 p. 6)\n2. Writing is generally precise; algorithm and loss are laid out formally (Eq. 1‑3 p. 3‑4) ; helpful schematic (Fig. 1 p. 3).\n3. Exposes a new, harder‑to‑detect vulnerability in automated evaluation pipelines, a topic of immediate practical concern for benchmarking and safety; RLRE could be repurposed for constructive goals (e.g., bias reduction).\n4. Shifts attack locus from post‑hoc answer editing to up‑stream preamble optimization, a novel angle; first on‑line application of CoPG to LLM rewards.\n\nWeaknesses\n1. Narrow judge set: training uses a single judge (Command‑R+); only GPT‑3.5 is used for transfer.\nLimited human study: 7 experts, 175 answers – too small to support broad “undetectable” claim.\nNo statistical test of improvements; only descriptive statistics.\n2. Paper is long and occasionally repetitive; some tables/figures (e.g., radar plots) are hard to read in grayscale; appendix contains critical parameters (β, early‑stopping) that should be surfaced earlier.\n3. Impact depends on adoption of LLM‑as‑a‑judge; if community shifts to hybrid or adversarial evaluation, severity may diminish.\n4. Conceptually adjacent to prompt‑optimization work; novelty rests on reinforcement‑learning formulation rather than entirely new insight.", "was_revised": true, "final_justification": "Based on the conversation below I am updating my ratings.", "rebuttal": "Dear Reviewer FfKE,\n\nThank you for your review. We are happy that you found the idea original and the work sound and comprehensive. Please find our responses below.\n\n> **Weakness 1:** *“Narrow judge set: training uses a single judge (Command‑R+); only GPT‑3.5 is used for transfer. Limited human study: 7 experts, 175 answers – too small to support broad “undetectable” claim. No statistical test of improvements; only descriptive statistics.”*\n\n- The idea of exploring a training setup with an LLM jury was discussed during the experiment design phase. However, we prioritised training with one judge mainly because, in our setup, the judges are queried via API. Therefore, issues of throttling and query failures/retries would multiply by the number of judges, potentially compounding and significantly slowing training. Of course, there were also considerations of the total cost of querying multiple APIs. While we did not have the capacity for such a setup, the idea remains interesting to us, and we will add it to the discussion of future work directions. \n\n- As for the judge transferability study with GPT-3.5, we employed the same model used by [1], which is one of our strongest baselines. However, following your suggestion, we have now evaluated with further judges and found that the effectiveness of our attack still transfers (see our response to **Question 1** below).\n\n- Regarding the human study, we computed the median absolute deviation (MAD) for both accuracy and F1 scores across annotators for each attack type. In all cases, the MAD was zero, indicating that, within our relatively limited sample, different annotators were either consistently able or consistently unable to detect a given attack type. We believe this supports the findings and substantiates their robustness. That said, we acknowledge the limitations of the sample size, and we will revise the paper to clarify that our findings provide preliminary evidence of limited detectability under the specific conditions of this study.\n\n- Finally, for the statistical tests, we now report paired bootstrap confidence intervals as you recommended (see our response to **Question 2**).\n\n> **Weakness 2:** *“Paper is long and occasionally repetitive; some tables/figures (e.g., radar plots) are hard to read in grayscale; appendix contains critical parameters (β, early‑stopping) that should be surfaced earlier.”*\n\n- Thank you for pointing this out. We have now checked how our radar plots appear in greyscale and will change the shades so that they remain readable. \n- We agree that certain details should be moved to the main body; we were planning to use the extra page allowed for the camera-ready to do so. We will also review the paper for any sections that may appear redundant and streamline them to accommodate more critical details currently in the appendix.\n\n> **Weakness 3:** *“Impact depends on adoption of LLM‑as‑a‑judge; if community shifts to hybrid or adversarial evaluation, severity may diminish.”*\n\n- We agree with you, but we do suspect that since the LLM-as-a-judge paradigm has already demonstrated value in the field, it will continue to form at least a component of pipelined systems in the near future.\n\n> **Question 1:** *“Judge diversity – How does RLRE perform when rewards come from an ensemble of heterogeneous judges (e.g., four different models) rather than a single one? Please include results or discuss expected behaviour.”*\n\n- We have now evaluated the R7B pipeline with two additional LLMs, for a total of four judges: Command R+, GPT-3.5, GPT-4o-mini and Claude 3 Haiku. Please see the average results for the overall MT-Bench score in the table below, both for each judge individually and as a mean score of the 4-judge jury, where each judge is assigned equal weight.\n- We observe some variation in the scale of the rewards for different judges (i.e., some are 'stricter' in their evaluations than others). Nevertheless, for any given judge, the preamble-based attack still obtains the highest rewards relative to the existing baselines and the vanilla model. This also reflects a higher jury score (last row in the table).\n\n|              | No attack | Verbosity | Bandwagon | Authority | Refinement | Universal | Preambles |\n| ------------ | --------- | --------- | --------- | --------- | ---------- | --------- | --------- |\n| Command R+   | 7.29      | 7.31      | 7.32      | 7.40      | 7.61       | 7.41      | **7.93**      |\n| GPT-3.5      | 7.58      | 7.36      | 7.47      | 7.48      | 7.71       | 7.33      | **8.07**      |\n| 4o-mini      | 6.46      | 5.61      | 6.28      | 5.85      | 6.38       | 6.11      | **6.67**      |\n| Claude Haiku | 9.09      | 8.76      | 8.82      | 9.06      | 9.26       | 9.01      | **9.50**      |\n| Jury score   | 7.61      | 7.26      | 7.47      | 7.45      | 7.74       | 7.47      | **8.04**      |\n\n> **Question 2:** *“Robust statistics – Could you report confidence intervals (e.g., paired bootstrap over questions) to substantiate significance of the 0.1‑0.6 score gains?”*\n\n- Of course, please find below the CIs computed for non-attacked and attacked responses via paired bootstrapping, for all three candidate LLMs.\n\n|  | Overall mean improvement | 95% CI |\n| --------------------- | ------------------------ | ------------ |\n| Command R7B           | 0.64  | (0.22, 1.00) |\n| Command R             | 0.35    | (0.25, 0.93) |\n| Llama3.1 70B IT | 0.16     | (0.02, 0.59) |\n\n> **Question 3:** *“Compute & cost – Training requires many reward queries. Please quantify total API calls and cost, and compare with prior attacks. Under what budget would the attack cease to be practical?”*\n\n- We do at most 128k API queries to the judge-LLM (i.e., training for 1k steps on the 64k UltraFeedback samples with bs=64 and 2 completions per prompt). Compared with [1], which is the only other 'trained' attack among our baselines, our training is substantially cheaper. For $n$ training samples, the exhaustive search method over the 20k-word vocabulary used by [1] needs to make $20000 \\times 4 \\times n$ calls to the judge. For context, making 128k inference calls to the Command R+ API when training with our method, at the average input and output token budgets we observed, costs approximately 500 USD. In contrast, the method in [1] requires using smaller training splits due to the massive cost and even so, it is more expensive (e.g., on just 500-1000 samples, the equivalent cost for this method is several thousand USD).\n\n> **Question 4:** *‘Content quality – Fig. H.1 (p. 25) shows higher‑scored yet incorrect maths answers. Have you measured factual accuracy systematically (e.g., GSM‑8K exact‑match)? This would clarify whether judges are truly “fooled”.’*\n\n- We have measured the factual accuracy of all math and reasoning questions w.r.t. their reference answers, for all models and question turns (120 questions in total). These are very similar, as shown in the table below, thus the evidence indeed points to the judge being 'fooled' for these factually measurable domains.\n\n|  |  |\n|----------|----------|\n| No attack   | 45.9% |\n| Preambles| 44.2% |\n\n> **Question 5:** *“Defences – Given that perplexity fails, have you tried delta‑answer inspection (diff between outputs with and without preamble) or attention‑based detectors? Evidence would guide future mitigation.”*\n\n- It is indeed possible that studying attention maps, or having access to the entire probability distribution over the vocabulary at each decoding step for the candidate LLM, may reveal subtle variations that result from the attack. However, please note that this analysis is rarely possible in practice, since LLMs served via API (like those that we use as frozen candidates in our pipeline) do not give access to the internal mechanisms, nor do they return the entire vocabulary distributions. For this reason, the PPL analysis we perform (introduced by [2]) is currently the established detection method, used in prior and contemporary work on LLM attacks (e.g., [1], [3]). We will add a discussion of these points in the paper.\n\n> **Question 6:** *“Score could increase if additional judges and a larger human study confirm undetectability, or if authors provide stronger evidence that inflated scores do not correspond to better factual quality.”*\n\n- As you suggested, we have evaluated with further judges (see our response to **Question 1** above), and we also provide strong evidence that the inflated scores observed on math and reasoning do not map to improved accuracy (see response to **Question 4**).\n- As for the large-scale human study, this would be too complex and expensive to set up in few days. However, we have provided statistical analysis details of the current human annotations in our response to **Weakness 1** above (third bullet point), and we also commit to adjusting the language in the manuscript to clarify that we find early evidence of limited detectability, within the context and constraints of our relatively limited-scale human evaluation.\n\nWe hope that our additional experiments robustly address the questions you raised, indicating both transferability to further judge-LLMs as well as non-correspondence with factual quality, and that this is sufficient to consider raising the score as suggested. We will add all of the above results to the paper. Please let us know if you have further questions or comments, and thank you for helping us improve the manuscript.\n\nKind regards,\\\nThe Authors of Submission #23360\n\n---\n[1] (Raina et al., 2024) Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment\\\n[2] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models\\\n[3] (Zheng et al., 2025) Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates"}, {"review_id": "bCddz4mrDl", "reviewer": "Reviewer_nFn5", "summary": "The paper questions the robustness of the LLM-as-a-judge framework to evaluate LLMs on human preferences using a judge-LLM. The authors propose a framework that uses evaluations of a judge-LLM in an RL setting in order to tune a model that generates system prompts. These system prompts are used in the llm-as-a-judge framework on the candidate LLMs in order to increase the expected score (as given by the judge-LLM) of their outputs. The system promtps are dependent on the query being asked to the candidate LLM.\nExtensive experiments are performed, showing the success rate and detection rate (by automated methods or human subjects) of this method in comparison with other adversarial attacks on the LLM-as-a-judge framework, as well as its transferability across different candidate and judge LLMs other than those used during training of the system prompt generator.", "questions": "I find it interesting how different/gibberish the preambles are for Llama compared to Command. Do you have any hypothesis as to why this happens?\n\nDo you have results per question type (as in Figure 2) for the other attacks as well? If yes, then it would be nice to add in an appendix for completion.\n\nIn your formalization of the RL problem (line 116), in the first expectation subscript you write $(i,q) \\sim \\mathcal{D}$. I believe this is a typo? Since $i$ is independent of $\\mathcal{D}$.\n\nOther minor typographical errors: typo in line 230 (unversal), broken reference in line 250", "limitations": "A potential limitation of the framework is that it requires that it is possible to prepend a system prompt in the evaluation setting, which may not always be the case.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The experimentation and analysis of the proposed method is thorough and well-thought out. The investigation on the transferability across both candidate and judge LLMs not used in training is intriguing, fitting, and demonstrates the practicality of the proposed method.\n- All details and parameters of the performed experiments are very clearly presented.\n- The human study adds to the work and solidly shows the (un)detectability of the attack.\n- The framework has potential to generalize to other settings (such as safety), if different reward functions (other than judge-LLM annotations, eg human annotation) are used during training of the system prompt generator, which the authors acknowledge.\n\nWeaknesses:\n- The experiment on the success rate of the method is promising but is only performed on a single small dataset, MMLU. While the authors explain this choice by mentioning that (as opposed to others) this dataset is diverse, challenging, and suitable for their targeted controlled independent-evaluation setup, the results would be strengthened if the method can also generalise to less controlled settings. This can be mentioned as potential future work. For example, you could train/test on subsets of questions from the LMSYS dataset (```lmsys/lmsys-chat-1m``` on huggingface) which are asked by independent users on the chatbot arena platform, the most popular human evaluation platform.\n- The claim that this method exposes the vulnerability of the LLM-as-a-judge framework is a bit odd. To argue this, you would need to compare the LLM-judge evaluations against human evaluations on the same answers (given by your framework and other baselines), to see whether human evaluators also give higher scores to outputs given by your framework. The results could be framed positively instead, in that you provide a framework to generate system prompts that lead to better outputs from candidate LLMs (rather than it being presented as an attack).", "was_revised": true, "final_justification": "This paper introduces a method to exploit the LLM-as-a-judge framework to get higher LLM-judge scores. The expanded evaluation on an additional dataset and the human study that demonstrates the effectiveness of the \"attack\" add to an already strong experimental analysis. As such, I recommend acceptance.", "rebuttal": "Dear Reviewer nFn5,\n\nThank you for reading and reviewing our work. We are happy that you found the method thorough and well thought out. Please find below our responses to your comments and questions.\n\n> **Weakness 1:** *“The experiment on the success rate of the method is promising but is only performed on a single small dataset [...] the results would be strengthened if the method can also generalise to less controlled settings. This can be mentioned as potential future work. For example, you could train/test on subsets of questions from the LMSYS dataset”*\n\n- Thank you for the suggestion. As you said, the main focus of the work is demonstrating that it is possible to reverse engineer human feedback in a controlled setting, and we found that the data we use is diverse and challenging. However, we agree with you that generalisation in less controlled settings is interesting and worth exploring. We have now evaluated the Command R7B+7B pipeline on the Arena-Hard benchmark, from the LMSYS suite. This has a different task distribution than our training data, and it also involves a different reward assignment strategy and range than the one we trained on. To make the evaluation setup even more generalisable, we evaluate with two different judges: Command R+ (i.e., the judge from our training setup) and GPT-4 (a commonly used judge for Arena-Hard evaluations).\n- Below, we show the rewards obtained on Arena-Hard by the candidate LLM responses before and after the injection of the tuned preambles, averaged across runs. We find that there is a clear improvement using the Command R+ judge. With the GPT-4 judge, the improvement is less significant (within standard deviations); however, we still observe a positive trend in the point-wise averages. Overall, we find evidence that the method generalises to an additional benchmark without targeted training. We will add these additional experiments to the paper.\n\n| Judge      | Without preamble injection | With preamble injection |\n| ---------- | ------------------------- | ------------------------ |\n| Command R+ | 58.3 ± 0.7                | 61.8 ± 0.5               |\n| GPT-4      | 27.6 ± 0.8                | 28.8 ± 0.8               |\n\n> **Weakness 2:** *“The claim that this method exposes the vulnerability of the LLM-as-a-judge framework is a bit odd. To argue this, you would need to compare the LLM-judge evaluations against human evaluations on the same answers (given by your framework and other baselines), to see whether human evaluators also give higher scores to outputs given by your framework. The results could be framed positively instead, in that you provide a framework to generate system prompts that lead to better outputs from candidate LLMs (rather than it being presented as an attack).”*\n\n- We would argue that even if the responses did improve by any other metric aside from the LLM-as-a-judge scores, this could still be referred to as an 'attack' in this context, since the preambles are tuned by directly exploiting the judge’s reward to maximise the benchmark scores.\n- That said, we do appreciate your suggestion for the additional experiment. We have carried out a blind human evaluation study on a 25% sample of all turn-1 responses generated by both the non-attacked Command R7B and the same model injected with the preambles. We had three expert annotators without knowledge of the attack assess the overall quality of each response and label it as “Poor”, “Fair” or “Good”. We show the number of each label assigned to the non-attacked and the attacked responses in the table below. Note that the label distributions are fairly similar for both models, with the non-attacked model receiving slightly more scores at either end of the scale (‘Poor’ and ‘Good’), and the attacked one receiving slightly more mid-range scores (‘Fair’).\n\n|           | Poor | Fair | Good |\n| --------- | ---- | ---- | ---- |\n| No attack | 20   | 6    | 34   |\n| Preambles | 19   | 9    | 32   |\n\n- Additionally, we had the annotators assign a discrete 1-10 rating to each response (the same scale used by the judge-LLMs in the paper). Below we show the average ratings obtained by each setup. We find that the difference between the human ratings of the two models is small (0.05), whereas we know that the difference between the LLM-as-a-judge ratings is substantially larger in favour of our attack. This suggests that the method exploits, in large part, the judge’s specific preferences.\n\n|           | Avg. human rating of responses |\n| --------- | ------------------------------ |\n| No attack | 6.03                           |\n| Preambles | 6.08                           |\n\n- We will extend the above human experiment to all questions rather than just a subsample, and show these results in the Appendix along with a discussion of the above points.\n\n- We also absolutely agree with you in principle that the method has potential for useful applications rather than just 'attacks'. We also consider identifying weaknesses of any system positive, as it provides us with the opportunity to address them and build more robust systems that are better aligned with our objectives.\n\n> **Question 1:** *“I find it interesting how different/gibberish the preambles are for Llama compared to Command. Do you have any hypothesis as to why this happens?”*\n\n- We believe this happens mainly because we perform all hyperparameter tuning on the Command pipeline, due to computational limitations, and apply the same hyperparameters to the Llama pipeline. Among the hyperparameters that we tune is the value of the KL-divergence weight $\\beta$, which regulates the faithfulness of the token distributions to those of the reference model, and therefore determines the fluency of the preambles. Note that we prioritise downstream reward over fluency since our preambles do not necessarily need to be human-readable. Therefore, our value of $\\beta$ is relatively low, as described in Appendix B.2 of the paper. We found that, with this low $\\beta$ value optimised on the Command pipeline, the Llama model tends to steer away from fluent outputs faster and more drastically, while the downstream rewards increase. This may be due to pre/post-training strategies that are unique to each model family. We consider the Llama results extremely interesting as well, since they show that, in this context, maximising exploration and downweighting the KL-divergence anchor produces effective preambles regardless of their fluency.\n\n> **Question 2:** *“Do you have results per question type (as in Figure 2) for the other attacks as well? If yes, then it would be nice to add in an appendix for completion.”*\n\n- Indeed, we do have them for every attack. We will add to the Appendix tables showing the fine-grained rewards assigned to each domain sub-category under each attack.\n\n> **Question 3:** *“In your formalization of the RL problem (line 116), in the first expectation subscript you write $(i, q) \\sim \\mathcal{D}$. I believe this is a typo? Since $i$ is independent of $\\mathcal{D}$.”*\n\n- Well spotted, thank you. Indeed, $i$ is independent of $\\mathcal{D}$, and is the fixed part of the input, so it should be removed from the expectation altogether. We have immediately corrected this typo.\n\n> **Question 4:** *“Other minor typographical errors: typo in line 230 (unversal), broken reference in line 250”*\n\n- Thank you again, we have now fixed these too.\n\n\nWe hope our responses and additional experiments above have addressed the concerns and answered your questions. Please let us know if other questions arise, and thank you for helping us strengthen the paper.\n\nKind regards,\\\nThe Authors of Submission #23360"}, {"review_id": "b5xE3trJn0", "reviewer": "Reviewer_7h5s", "summary": "This paper investigates a novel adversarial attack on the LLM-as-a-judge evaluation framework, termed Reinforcement Learning for Reverse Engineering (RLRE). Unlike prior work that post-hoc modifies generated responses (e.g., appending text or substitution), RLRE adversarially tunes a separate preamble generator. This generator, pipelined with a frozen candidate-LLM, produces textual preambles that, when supplied as input alongside a question, lead the candidate-LLM's responses to receive higher scores from a judge-LLM. The authors demonstrate that this method is more effective at inflating judge scores, virtually undetectable by existing perplexity analysis or human inspection, and exhibits transferability across different candidate-LLMs and judge-LLMs.", "questions": "1. Could you provide a more detailed explanation of what \"injected\" means in the context of preambles, specifically contrasting it with how previous \"appending\" methods operate on responses? How does the LLM's internal processing of an injected preamble differ from a post-hoc appended text sequence, beyond just the position in the input?\n1. Given the demonstrated undetectability of your attack by existing methods, what are some potential novel detection strategies or defense mechanisms that could be developed to identify and mitigate this type of preamble-based adversarial attack?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Novel Attack Method:** The core contribution of adversarially tuning preambles to be injected into a frozen LLM is novel. \n1. **High Efficacy and Undetectability:** The paper provides empirical evidence that the proposed attack boosts LLM-as-a-judge scores and remains undetectable by common perplexity analysis and human inspection, which is a major concern for the reliability of automated evaluation.\n1. **Transferability:** The demonstration that the attack transfers effectively to candidate-LLMs and judge-LLMs not seen during training is a significant finding, indicating the potential for a plug-and-play adversarial component.\n1. **Raises Important Questions:** The findings directly challenge the reliability and robustness of the widely adopted LLM-as-a-judge paradigm, prompting critical re-evaluation of current LLM assessment methods. The observation about the fluency of optimal preambles also hints at fundamental aspects of LLM conditioning.\n\n**Weaknesses**\n\n1. **MT-Bench Reliability:** The paper relies heavily on MT-Bench for evaluation. A more critical discussion of MT-Bench's specific limitations, or validation on a broader suite of benchmarks with diverse evaluation methodologies, could enhance the robustness of the empirical claims.\n1. **Limited Scope of Quality Improvement vs. Bias Hacking**: The paper currently notes that accuracy in math and reasoning tasks did not improve despite higher scores. However, it's critical to more definitively disentangle whether the increased LLM-as-a-judge scores are genuinely due to an improved response quality (e.g., better structure, clarity) or primarily by \"hacking\" the judge's biases (e.g., verbosity, formatting preferences). A more systematic analysis across diverse tasks (e.g., general chat, instruction following as in IFEval) with human-in-the-loop quality evaluations would be beneficial to confirm if these preambles lead to universally better responses or merely exploit specific judge biases.", "was_revised": true, "final_justification": "I have read the authors' rebuttal and find it to be thorough and convincing. They have provided significant new experimental evidence that directly addresses the main weaknesses I identified in my initial review. As a result, my assessment of the paper has changed from borderline reject to recommend acceptance.", "rebuttal": "Dear Reviewer 7h5s,\n\nWe would like to thank you for reviewing our paper and acknowledging its novelty and the importance of the questions raised. We address all your questions and concerns below.\n\n> **Weakness 1:** *“MT-Bench Reliability: The paper relies heavily on MT-Bench for evaluation. A more critical discussion of MT-Bench's specific limitations, or validation on a broader suite of benchmarks with diverse evaluation methodologies, could enhance the robustness of the empirical claims.”*\n\n- We have now evaluated our main pipeline (Command R7B+7B) on a further widely-used benchmark, Arena-Hard. Not only does this benchmark differ from both UltraFeedback and MT-Bench in terms of task distribution, but its reward assignment strategy and range are also different from the reward that we trained on, as detailed in [1]. In addition, we evaluate both with the Command R+ judge used during training as well as an unseen judge, GPT-4.\n- Below we show the results, averaged across runs, obtained by the candidate LLM on Arena-Hard with and without injected preambles. As in MT-Bench, higher scores are better. We observe clear improvement on this new benchmark using the Command R+ judge. Improvement with GPT-4 judgments is less significant; however, we still observe an upward trend when averaged across runs.\n- These results indicate that the method generalises to an additional benchmark without targeted training. We will be happy to add them to the paper.\n\n| Judge      | Without preamble injection | With preamble injection |\n| ---------- | -------------------------- | ----------------------- |\n| Command R+ | 58.3 ± 0.7                 | 61.8 ± 0.5              |\n| GPT-4      | 27.6 ± 0.8                 | 28.8 ± 0.8              |\n\n> **Weakness 2:** *“Limited Scope of Quality Improvement vs. Bias Hacking: The paper currently notes that accuracy in math and reasoning tasks did not improve despite higher scores. However, it's critical to more definitively disentangle whether the increased LLM-as-a-judge scores are genuinely due to an improved response quality (e.g., better structure, clarity) or primarily by \"hacking\" the judge's biases (e.g., verbosity, formatting preferences). A more systematic analysis across diverse tasks (e.g., general chat, instruction following as in IFEval) with human-in-the-loop quality evaluations would be beneficial to confirm if these preambles lead to universally better responses or merely exploit specific judge biases.”*\n\n- We appreciate and agree with this suggestion. We have computed the factual accuracy scores for all math and reasoning questions, w.r.t. their reference answers, for all models and question turns (120 questions in total). As shown in the table below, we find no evidence that math and reasoning results improve, despite the higher LLM-as-a-judge rewards. We will add these to the paper.\n\n|  |  |\n|----------|----------|\n| No attack   | 45.9% |\n| Preambles| 44.2% |\n\n- Moreover, we have now run a human-in-the-loop qualitative evaluation of a 25% sample of all turn-1 responses generated by both the non-attacked Command R7B and the same model with injected preambles. We had three expert annotators without knowledge of the attack assess the overall quality of each response and label it as “Poor”, “Fair” or “Good”. We show the number of each label assigned to the non-attacked and the attacked responses below. We found the label distributions to be fairly similar, with the non-attacked model receiving slightly more scores at either end of the scale (‘Poor’ and ‘Good’), and the attacked one receiving slightly more mid-range scores (‘Fair’).\n\n|           | Poor | Fair | Good |\n| --------- | ---- | ---- | ---- |\n| No attack | 20   | 6    | 34   |\n| Preambles | 19   | 9    | 32   |\n\n- Finally, we had the annotators assign a discrete 1-10 rating to each response (the same scale used by the judge-LLMs in our experiments). Please see the average ratings in the table below. The difference between them is slightly in favour of the preambles method (+0.05); however, this difference is small compared to the substantial increase in average LLM-as-a-judge rewards obtained by the attacked model in our experiments. We plan to extend this study so that it covers all questions (rather than a 25% sample) and include it in the camera-ready.\n\n|           | Avg. human rating of responses |\n| --------- | ------------------------------ |\n| No attack | 6.03                           |\n| Preambles | 6.08                           |\n\n> **Question 1:** *“Could you provide a more detailed explanation of what \"injected\" means in the context of preambles, specifically contrasting it with how previous \"appending\" methods operate on responses? How does the LLM's internal processing of an injected preamble differ from a post-hoc appended text sequence, beyond just the position in the input?”*\n\n- The previous methods intervene on a pre-generated response, by adding text to it or otherwise modifying it post-hoc in order to ‘trick’ the LLM-as-a-judge to increase the score (for example, the universal attack first generates a response, then appends to it four words which have been tuned separately to increase the judge’s reward). Unlike many other methods that modify the generated response, we modify only the preamble (i.e. system prompt) to the LLM that generates the response. This is both a more realistic setting and substantially harder to detect as the generated response is entirely LLM-written.\n\n> **Question 2:** *“Given the demonstrated undetectability of your attack by existing methods, what are some potential novel detection strategies or defense mechanisms that could be developed to identify and mitigate this type of preamble-based adversarial attack?”*\n\n- That is a fair question. Presently, PPL analysis introduced by [2] is the established detection method, used in prior and contemporary work on LLM attacks (e.g., [3], [4]). It is also well-aligned with OOD detection and adversarial robustness certification in general machine learning literature, where the probability of a prediction is flagged if lower than a threshold. It is possible that if we had access to the entire probability distribution over the vocabulary at each decoding step for the candidate LLM, or other mechanisms such as the attention weights, we could detect subtle variations caused by the attack. However, this analysis is rarely possible in practice, since LLMs served via API (like those that we use as frozen candidates in our pipeline) do not provide this degree of access. Still, we do agree with you that these are considerations worth mentioning; we will add a brief discussion of the above points to the paper.\n\nPlease let us know if our answers and the additional experiments above address your concerns and questions, or whether we can provide more details. Meanwhile, we would like to thank you for your contributions toward improving the work.\n\nKind regards,\\\nThe Authors of Submission #23360\n\n---\n[1] (Li et al., 2024) From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline\\\n[2] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models\\\n[3] (Raina et al., 2024) Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment\\\n[4] (Zheng et al., 2025) Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates"}]

Author rebuttal:

Meta-review: My recommendation is to accept the paper as a spotlight.

The paper proposes a method for adversarially improving feedback in LLM-as-a-judge settings by training a model to provide a prompt-specific pre-amble to a frozen candidate LLM, whose output in response to the preamble + prompt is then sent to the judge. The authors show that this kind of attack has similar effectiveness to previous attacks that edit the candidate model's response after seeing the judge's feedback. The fact that the modifications happen to the input of the LLM make the attack harder to detect, a claim supported by a human study. The attack also transfers across candidate and judge LLMs.

Reviewers agreed that the paper showed creativity, precision in communication, and thoroughness in evaluation. There was consensus to accept after the rebuttal period, during which the authors were able to add additional support for their approach. I hope the authors will follow through on the improvements they promised during the rebuttal period, but even the paper as it was submitted had a combination of a fresh idea, thorough investigation, and solid execution that made it a good candidate for a spotlight.

I personally found the potential use for constructive goals to also be interesting.
Decision: Accept (spotlight)