Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024)

Abstract: The automatic evaluation of natural language generation (NLG) systems presents a longlasting challenge. Recent studies have highlighted various neural metrics that align well with human evaluations. Yet, the robustness of these evaluators against adversarial perturbations remains largely under-explored due to the unique challenges in obtaining adversarial data for different NLG evaluation tasks. To address the problem, we introduce AdvEval, a novel black-box adversarial framework against NLG evaluators. AdvEval is specially tailored to generate data that yield strong disagreements between human and victim evaluators. Specifically, inspired by the recent success of large language models (LLMs) in text generation and evaluation, we adopt strong LLMs as both the data generator and gold evaluator. Adversarial data are automatically optimized with feedback from the gold and victim evaluator. We conduct experiments on 12 victim evaluators and 11 NLG datasets, spanning tasks including dialogue, summarization, and question evaluation. The results show that AdvEval can lead to significant performance degradation of various victim metrics, thereby validating its efficacy.1

heading: 1 Introduction
text: Recent advancements in generative language models, especially advanced large language models (LLMs), have led to their widespread application in creating human-like content across various domains. Concurrently, numerous automatic text evaluation metrics have been developed, aiming to identify unhelpful responses and understand the model’s capacity. These metrics have exhibited good correlations with human judgment across diverse natural language generation (NLG) tasks, e.g., dialogue generation (Zhang et al., 2021a), summarization (Zhong et al., 2022; Yuan et al., 2021),
∗ Corresponding author. 1Code is available at github.com/MatthewCYM/
AdvEval.
question generation (Mohammadshahi et al., 2023; Wang et al., 2022a). Despite their strong performance, the robustness of these metrics against adversarial attacks or misleading inputs remains critically under-explored. Therefore, in this paper, we aim to automatically identify "Achilles’ Heel", critical weaknesses, within well-established automatic evaluation metrics.
Adversarial attack techniques are widely adopted to assess the robustness of neural models. Yet, their application in NLP has predominantly focused on general classification tasks, e.g., sentiment classification (Jin et al., 2020; Li et al., 2019a). These attack techniques all build upon the label-preserving assumption that the gold label of input text will remain unchanged after adding bounded perturbation on the input text (Wang et al., 2022b). This assumption, however, does not hold on the NLG evaluation tasks, where even slight modifications can drastically alter, often negatively, a text’s perceived quality, which directly correlates with the regression label in the evaluation tasks. For example, changing a single word "movie" to "dish" in the dialogue response "That’s my favorite movie" will perfectly preserve the positive sentiment classification label but significantly alter the response relevance evaluation label, given that the dialogue context is about a movie. Consequently, these traditional attacking techniques can easily obtain low-quality adversarial texts with high metric scores while struggling to produce high-quality adversarial texts that score low. Existing studies on NLG evaluators’ robustness mainly rely on manual effort or heuristic perturbation rules to ensure the perturbed text accords with specified labels. These studies also fail to obtain high-quality adversarial texts with low metric scores and cannot scale up to generate large-scale diverse adversarial data (Khalid and Lee, 2022; Sai et al., 2021).
To overcome these limitations, we propose AdvEval, designed to obtain data that lead to two
1359
different types of disagreement between humans and victim evaluators: 1) text valued by humans but underrated by victim evaluators; 2) text underrated by humans but valued by victim evaluators. Inspired by recent works on using LLMs to iteratively improve their response based on current feedback (Madaan et al., 2023; Yang et al., 2023; Chen et al., 2023a), we adapt LLMs as the data generator to iteratively optimize the benign input text with the adversarial objective. As mentioned above, label preservation posts a unique challenge in attacking NLG tasks. Recognizing the recent trend of leveraging large language models (LLMs) as a substitute for human annotators to generate and annotate high-quality data (He et al., 2023; Ye et al., 2022a,b; Meng et al., 2022), we also utilize LLMs as the evaluator to give feedback during the optimization process, thereby imposing an implicit quality constraint on the generated text.
We evaluate AdvEval on 11 datasets covering three distinct NLG tasks against 12 automatic evaluation metrics that vary in their reliance on references, models, and LLMs. Experimental results demonstrate AdvEval’s ability to generate highquality adversarial data effectively. Through manual inspection, we also confirm the label preservation property of AdvEval. Further experiments on adversarial learning, ablation studies, and case studies convincingly validate the effectiveness and generalization ability of our proposed AdvEval framework.

heading: 2 Background

heading: 2.1 NLG Evaluator
text: The evaluation of AI-generated content is increasingly important recently in various domains (Su et al., 2020; Gao et al., 2019; Jiang et al., 2023). Automatic NLG evaluators can be broadly divided into reference-based and reference-free metrics. Traditional reference-based metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) are favored for their simplicity and versatility across NLG tasks. Yet, their inability to capture complex textual semantics has led to the development of newer reference-based metrics utilizing contextualized embeddings for enhanced semantic understanding, such as BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), and BLEURT (Sellam et al., 2020).
In open-ended tasks like dialogue generation, reference-based metrics often fall short in mir-
roring human judgment (Liu et al., 2016). This has led to the development of trained referencefree metrics such as RUBER (Tao et al., 2018), DynaEval (Zhang et al., 2021a), UniEval (Zhong et al., 2022), TRUE (Honovich et al., 2022a), and PoE (Zhang et al., 2023). These reference free metrics offer closer alignment with human evaluations but still don’t reach the criteria for strong automatic evaluators2 and remain task-specific.
The emergence of LLMs marks a significant step towards developing more general and effective NLG evaluators. Several recent works (Chen et al., 2023b; Liu et al., 2023a; Lin and Chen, 2023) examine the NLG evaluation capability of instruction-following LLMs via prompting and report strong correlations with human evaluation. Remarkably, GPT-4 (OpenAI, 2023) is found to exhibit human-level judgment while being more scalable and less expensive than crowd-sourced workers (Zheng et al., 2023; Gilardi et al., 2023; Chiang and Lee, 2023). Inspired by these findings, our AdvEval framework incorporates advanced proprietary LLMs like GPT-4 and PALM-2 to serve as proxies for human evaluators. Moreover, we conduct a comprehensive analysis of the robustness of metrics from the aforementioned types3 across various NLG tasks, including dialogue, summarization, and question generation.

heading: 2.2 Adversarial Attack
text: Adversarial attacks are common practice for robustness evaluation through creating adversarial examples to elicit incorrect predictions from neural networks (Papernot et al., 2016; Ebrahimi et al., 2018b; Li et al., 2019b; Wallace et al., 2019; Le et al., 2022; Cheng et al., 2020; Zhang et al., 2021b). These attacks manipulate benign inputs at different granularity levels. Mainstream perturbations include character-level (Belinkov and Bisk, 2018; Ebrahimi et al., 2018a), word-level (Ren et al., 2019; Jin et al., 2020; Li et al., 2021; Garg and Ramakrishnan, 2020), phrase-level (Lei et al., 2022; Chen et al., 2021a; Zheng et al., 2020), and sentencelevel (Wang et al., 2020; Lin et al., 2021; Xu et al., 2021). The fundamental assumption of these works is label preservation under limited minor perturbation. The assumption does hold for various classification tasks, e.g., sentiment and intent classifi-
2Defined as having a correlation over 0.8 with human evaluations.
3Including reference-based and reference-free, trained and untrained, as well as BERT-based and LLM-based metrics.

heading: Scorer
text: cation. Yet, these perturbations are highly likely to affect the text quality, which is critical in NLG evaluation tasks. This discrepancy renders traditional adversarial techniques unsuitable for attacking NLG evaluators.
While there are also adversarial works targeting NLG evaluators (Khalid and Lee, 2022; Sai et al., 2021), they bypass the label concern by using only heuristic perturbation rules. These rules include invariant rules, e.g., phrase expansion/contraction, and deterioration rules, e.g., dropping words and reordering sentences. Yet, these rules are relatively simple and follow a certain pattern, rendering them infeasible to scale up and create diverse, high-quality adversarial data. In contrast, AdvEval utilizes a powerful generative model to craft adversarial samples with automatic quality control, which can generate large-scale, diverse natural adversarial samples.

heading: 3 Methodology

heading: 3.1 Problem Formulation
text: Our study focuses on conditional generation tasks, where a single data point consists of input context C, output response R, and reference ground-truth response G. Given the context and response, the evaluator E will assign a score S, indicating the quality of the response w.r.t. context:
S = E(C,R,G). (1)
We aim to design a generalizable framework to generate adversarial samples for various NLG evaluation tasks. Given a victim evaluator Evictim, we target to obtain a perturbed response from initial response R0 that leads to the disagreement between
score Svictim from Evictim and human judgment Shuman. Specifically, the perturbed responses include two types: 1) R+ where Shuman is high and Svictim is low; 2) R− where Shuman is low and Svictim is high. For all Evictim, we assume blackbox access to all Evictim, which means only the evaluation scores are accessible. To tackle this problem, we employ advanced LLMs as both adversarial generator G and evaluator Egold. For these two models, we only have black-box access, where only output text is available.

heading: 3.2 Overview
text: Fig. 1 illustrates the overall framework of AdvEval. The generator G iteratively optimizes the adversarial response R. In each step, G generates new candidate responses with the task-specific prompt as the input. These new candidates are then scored by both Egold and Evictim, generating a feedback score Sfb. Afterwards, these candidate responses, along with their feedback scores, will be included in the generation prompt for the next iteration to guide G towards generating better adversarial responses. The optimization process ends if the score reaches a pre-defined threshold or the iteration count hits the maximum limit. Finally, the response with the highest score will be returned as the adversarial response. The following sections will elaborate on the task-specific prompt and scorer designs.

heading: 3.3 Generation Prompt
text: We design task-specific prompts to iteratively improve the generated adversarial samples. Fig. 1 shows a dialogue task example prompt. The taskspecific prompt includes below components.
Task instruction: We give instructions to the adversarial generator about the optimization goal ("generate dialogue response with higher score"), the output format for easier parsing ("begin with <RES> and end with <RES>"), and also NLG task definition ("dialogue utterance as the response to the given dialogue context").
Evaluation criteria: In addition, we find that the viable response space is considerably limited, particularly for more constrained tasks like summarization. During the optimization process, the generator is prone to generate bad response, which reduces the R+ probability. Therefore, when generating R+, we optionally append the evaluation criteria (e.g., "The summary must be factually consistent with the article." for factual consistent summarization) to the task instruction. This emphasis makes the generator better understand the desired output, and implicitly constrain the output space.
Optimization trajectory: Rather than direct instructions on how to update the responses, we add optimization trajectory in the generation prompt. The optimization trajectory includes a history of generated responses and their associated scores, organized in ascending score order. The score serves as the proxy of disagreement between human and victim evaluator (Sec. 3.4). We only keep top-10 responses with highest scores to reduce the context length and inference cost. We aims to use the optimization trajectory to guide G towards improved responses by combining existing good responses, or perturbing the responses along a potential desired direction.

heading: 3.4 Human Judgement Proxy
text: As mentioned previously, our optimization goal here is to maximize disagreement between Svictim and Shuman. The disagreement serves as both the criterion for successful attacks and crucial feedback to the adversarial generator. Yet, unlike previous
work on prompt optimization, where the goal function is simply accuracy and easily accessible, it’s costly and infeasible to ask human to participate in the optimization process, complicating the assignment of scores that accurately reflect human judgment. To address this problem, we propose to use a strong evaluator Egold to produce Sgold as a proxy of Shuman. Specifically, the Egold can be a single LLM-based evaluator or an ensemble of several powerful evaluators. The Egold is asked to rate the response on a 100-point scale using the manual/automatic prompt shown in Appx. B. Since we observe that LLM-based evaluators sometimes produce unstable scores, we utilize self-agreement to produce a consistent score. We first obtain 8 ratings from the evaluator through repeatedly sampling. Then, we take the arithmetic mean of these ratings as the final gold score Sgold. After getting the Sgold, it is combined with Svictim to get the final score Sfb as the feedback to generator:
Sfb = { α× Sgold − Svictim, if R+ Svictim − α× Sgold, if R− , (2)
where α is hyper-parameter balancing the influence of Sgold and Svictim. A higher α encourages the generator to exploit the response space adhere to judgement of gold evaluator, while a lower α allows the generator to more aggressively explore diverse responses that can lead to misjudgement of victim evaluator.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Due to space limitations, for more details about model implementation and hyper-parameters, please refer to Appx. A.
Datasets: We conduct our experiments on three popular NLG evaluation tasks: dialogue response,

heading: Model DailyDialog DREAM
text: summarization, and question evaluation. We assess proposed AdvEval with 100 samples from each dataset. For dialogue response, we utilize DailyDialog (Li et al., 2017), DREAM (Sun et al., 2019), PersonaChat (Zhang et al., 2018), and MuTual (Cui et al., 2020). For summarization, data come from CNN/DailyMail (See et al., 2017), XSum (Narayan et al., 2018), and DialogSum (Chen et al., 2021b). For question generation, we use SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017), HotpotQA (Yang et al., 2018) and Natural Questions (Kwiatkowski et al., 2019).
Evaluation dimensions: In our study, each NLG task focuses on a key evaluation dimension. For dialogue response, we emphasize relevance, which strongly correlates with human judgment. For the summarization task, we prioritize factual consistency, a vital aspect of this task. Lastly, for question evaluation, we assess overall quality, encompassing both the answer-ability of the questions and their relevance to the provided answers.
Adversarial generators & gold evaluators: For consistency and simplicity, we utilize the same LLM as gold evaluator and adversarial generator. For dialogue response generation, we use PALM-2. For summarization and question generation, PALM-2 experiences severe hallucination,
and fails to generate factual consistent summaries or answerable questions in our preliminary experiments. Therefore, we use GPT-4 as the R+ generator/evaluator, and PALM-2 as the R− generator/evaluator.
Victim evaluators: Our study involves a thorough examination of a diverse array of evaluators, which vary in mechanism and model size. Tab. 1 lists these evaluators.4
Metrics: Our primary metric for evaluating adversarial methods is the attack success rate (ASR). We generate two types of adversarial data, and a successful attack must satisfy specific criteria:
Sgold > τ1 ∧ Sgold − Svictim > τ2, if R+ Sgold < τ1 ∧ Svictim − Sgold > τ2, if R− , (3)
where τ1 and τ2 are pre-defined thresholds. The first part of each condition ensures the adversarial data has a confident label (positive/negative), while the second part ensures the incorrect prediction by victim evaluators. Note that our ASR criteria involve Egold, which is a neural metric. Therefore, on top of ASR, we also manually check the label validity of the adversarial data for trustworthy scoring. We report the rate of data with correct labels.
4Experiments on InstructGPT is presented in APPX. C.

heading: Model HotpotQA NaturalQuestion
text: Baselines: We compare AdvEval against two types of baselines. The first type of baselines are specially designed for NLG evaluators, and utilize sophisticated rule-based perturbations to ensure the label validity (Sai et al., 2021; Khalid and Lee, 2022). The perturbations include invariant perturbation (synonym adjective, expansion, contraction) for R+ generation, and deterioration perturbation (jumble, change name, negate previous utterance, change question to assertion, change question word) for R− generation. The first type of baselines serve as our major baseline for their high relevance. For the second type of baselines, we adapt existing attack methods originally designed for general classification tasks, including TextFooler (Jin et al., 2020) and BERT-Attack (Li et al., 2020), for targeted attack with target label 0 to generate R+ and target label 1 to generate R−. We sample a random response as the initial response during generating R− to align with the label preservation assumption in these methods. For fair comparison, we allocate the same query budget 300 to both AdvEval and the baseline methods.

heading: 4.2 Main Results
text: Dialogue response evaluation: The comparison of different attacking methods against dialogue evaluation metrics are shown in Tab. 2. Overall, we have the following key observations. First, all evaluated metrics, regardless of their working mechanisms, are vulnerable to adversarial perturbations. Especially for R− case, even simple heuristic rulebased perturbations can deceive some models effectively. For example, negating previous utterance
achieves 47.8% ASR on average. Second, despite the promising performance of rule-based methods on generating R−, their effectiveness in creating R+ is limited, with only ASR 6.9% with synonym replacement, and 6.6% with phrase extension. These methods maintain label accuracy but lack data diversity and are easily detectable. Third, compared to rule-based methods, optimization-based baselines are more effective for R+ generation, but fall short in generating R−. Additionally, it struggles to produce adversarial samples for referencebased methods, which often assign high scores to perturbed responses due to the high vocabulary overlap with the reference caused by local perturbations. Lastly, AdvEval significantly outperforms all baselines, achieving an ASR of 84.8% for R+ and 92.5% for R−. This underscores the efficiency of AdvEval in navigating both R+ and R− scenarios. In addition, we observe that it’s easier to craft R− than R+ for all attacking methods, likely due to limited good response space compared to bad response space.
Question evaluation: The results on question evaluation are shown in Tab. 3. Findings from the question evaluation largely mirror those from the dialogue evaluation tasks. AdvEval achieves average R+ success rate 60.6% and R− success rate 92.7%. However, we observe a lower R+ ASR compared to dialogue tasks. The lower R+ ASR might be attributed to the stringent constraint, i.e., questions to be answerable from the article and aligned with the provided answer. This leads to a limited range of appropriate responses. Despite this, AdvEval still outperforms baseline methods

heading: Model CNN/Dailymail
text: in average performance. Summarization evaluation: The results on summarization evaluation are shown in Tab. 4. AdvEval surpasses baseline methods with an 85% R+ ASR, and 58.8% R− ASR. In addition, these summarization metrics are more robust against R− perturbation compared to dialogue and question metrics. The highest rule-based average ASR is only 15.3%, in stark contrast to 60.2% for question metric, and 47.8% for dialogue metric.
Overall, our proposed AdvEval significantly outperforms existing adversarial techniques across all examined tasks. The superior performance of AdvEval can be attributed to two key advantages. Firstly, AdvEval benefits from a larger search space. While all baseline methods are restricted to adding only local perturbations to the benign input, which significantly limits their search space, AdvEval leverages the remarkable generative capabilities of LLMs to perform both local and global perturbations. This results in a substantially larger search space, facilitating the discovery of novel and effective responses. Secondly, AdvEval includes improved search constraints. Baseline methods typically rely on either rule-based or sentence embedding constraints to maintain label integrity. However, as discussed in our paper, these constraints
fall short in text evaluation scenarios due to their inability to capture the nuances of natural language. To overcome this limitation, AdvEval adopts an LLM evaluator as a human proxy, which ensures superior label integrity by more accurately assessing the quality of generated responses.

heading: 4.3 Label Validity
text: In AdvEval, gold evaluator Egold is crucial as it acts as a human proxy for assessing response quality and guiding the adversarial generator. To evaluate its reliability, we conduct a manual examination on 780 adversarial data points, spanning various evaluation tasks and generated through different attack methods, with 260 data points per method. Three annotators manually label each data point as ’good’ or ’not good’ based on context and scoring criteria, with majority voting determining the final label. Then, we convert the Sgold to binary highquality/low-quality label with threshold 50. Finally, We compute the accuracy score between Sgold binary labels and human annotated ones to measure the reliability of Egold on adversarial data. Due to space limitations, detailed annotation schema can be found in APPX. E.
As shown in Tab. 5, Egold achieves an exceptionally high average accuracy over 80%, aligning closely with the highest inter-annotator agreement rates of 77%. This accuracy indicates a high degree of reliability of our gold evaluator, affirming its utility as a viable proxy for human judgment in guiding the optimization process. Additionally, we claim that a key issue with earlier adversarial methods is their inability to consistently preserve the quality label of R+. This is supported by our manual labeling results that 65% of R+ samples from TextFooler are labelled as bad response by human annotators.

heading: 4.4 Adversarial Learning
text: To assess potential countermeasures against adversarial attacks, we experiment with the adversarial learning on PoE-Large. We generate 3,649 adversarial data points from mixed data source of
DREAM, DailyDialog, and MuTual with AdvEval, and also 3,634 adversarial data points with TextFooler. Then, we further finetune the model on these two types of adversarial data mixed with original vanilla data respectively. As shown in Tab. 6, TextFooler can be easily defended with naive adversarial training, while our proposed AdvEval is more robust against this defense technique. After adversarial training, the model can hardly achieve any improvement, which further validates the capacity of AdvEval. We hypothesize that AdvEval can craft more diverse and complex data points that challenge the model’s adaptability.

heading: 4.5 Case Study
text: Tab. 7 presents adversarial data from different methods. More samples can be found in Appx. D. These samples are sampled from DailyDialog and SQUAD. Compared to previous methods, AdvEval is able to generate more valuable hard adversarial data with high quality and diversity. Regarding R+ generation, BERT-Attack and TextFooler exhibit similar tendencies, notably their failure to preserve the positive label after perturbation due to the inadequacy of their perturbation techniques. For example, in the dialogue sample, A asks whether the dish is good, but the perturbed response answers that the flat is popular, which is not related to the dialogue context at all. Therefore, it should be assigned a negative label. On the contrary, AdvEval generate a good response "You’ll never eat better" that directly answer the question from A. In addition, the AdvEval adversarial data are notably more complex. For instance, one need to infer "Ming dynasty succeeded Yuan dynasty" from "Yuan dynasty preceded Ming dynasty" to check the question’s answer-ability. For R− generation, all methods can generate samples with correct label. Compared to random responses from TextFooler and BERT-Attack, AdvEval samples are also more challenging to validate. The case study convincingly demonstrates the limitations of current attack

heading: Dialog Response Evaluation

heading: Question Evaluation
text: approaches and highlights the effectiveness and necessity of our proposed AdvEval framework.

heading: 4.6 Ablation Study
text: Gold evaluator: To understand the impact of the gold evaluator in AdvEval, we conduct a study with an AdvEval variant that only uses the victim score for feedback. For R− generation, we use random response as the initial input to avoid direct termination with initial input. As shown in Tab. 8, excluding the gold evaluator results in a significant drop in performance. The generation process tends to end prematurely because the score threshold is quickly met without the guidance of the gold evaluator.
Evaluation criteria: We optionally include eval-
uation criteria into the generation prompt, aiming to generate responses following the criteria. As shown in Tab. 9, excluding criteria leads to lower ASR. The results validate the effect of criteria in the generation prompt for guiding the LLM to generate response more adhere to the criteria.

heading: 5 Conclusion and Future Works
text: In this paper, we systematically evaluate the robustness of NLG evaluators. We also propose AdvEval, a novel black-box attack framework that effectively craft high-quality diverse adversarial samples that lead to performance degradation of various NLG evaluators. Specifically, AdvEval generates adversarial samples using LLM generator with feedback from LLM evaluator. Extensive experimental demonstrate the superior effectiveness of AdvEval across variety of NLG tasks. Future works include the development of defense methods against AdvEval, and the extension of AdvEval to other NLP tasks and NLG metrics.

heading: Limitations
text: The paper proposes the first optimization-based framework to examine the adversarial robustness of NLG evaluators. Yet, there are several limitations. Firstly, the framework is only able to find absolutely bad text with score around 0 or absolutely good text with score around 100. Techniques that can generate medium-quality text with score around 50 remains unexplored. We leave it to future work. Secondly, our research primarily targets the robustness of general NLG evaluators rather than specific evaluation dimensions. The dimensions we selected for study are either overall quality
or dimension that is highly correlated to the overall quality across various tasks. We leave dimensionoriented attacks as an important direction for future research. Thirdly, adversarial attack on strong proprietary LLM evaluators, e.g., InstructGPT, demands ensemble of several powerful evaluators, which requires more computational resources. We leave a more computational efficient framework to future exploration.

heading: Ethics Statement
text: We propose an adversarial attack framework against NLG evaluators on various NLG tasks in this work. We aim to study the robustness of NLG evaluators and provide insight to inspire future works on robust NLG evaluators. Our proposed framework may be used to attack online evaluation services. However, we believe that exploring this vulnerability and robustness of evaluator is more important than the above risks. Research studying attacks on evaluator will motivate improvements to the system security to defend against the attacks.

heading: Acknowledgement
text: This work is partly supported by ASTOUND (101071191- HORIZON- EIC -2021PATHFINDERCHALLENGES-01) funded by the European Commission, BEWORD (PID2021-126061OB-C43) funded by MCIN/AEI/10.13039/501100011033 and, as appropriate, by “ERDF A way of making Europe”, by the “European Union”, and the Research Grants for Young Investigators from Universidad Politécnica de Madrid (GENIUS:APOYO-JOVENES-21TAXTYC-32K61X37) funded by Comunidad de Madrid, Shenzhen Key Laboratory of Cross-Modal Cognitive Computing funded by Shenzhen Science and Technology Innovation Program (2024) 02, Shenzhen, China.

heading: A Detailed Experiment Setup

heading: A.1 Evaluator & Generator Implementation
text: We select victim evaluators that are across diverse categories, and commonly employed in latest studies (Sai et al., 2022). For standard metrics like SacreBLEU, ROUGE, BLEURT, and BERTScore, we use the implementation from huggingface evaluate library5. For PoE, UniEval, BARTScore, TURE, RQUGE, we use the official public repositories. For open-source LLM-based evaluators like Baichuan and Vicuna, we follow the methodologies in prior studies (Gupta et al., 2022; Fu et al.,
5github.com/huggingface/evaluate
2023) to compute the evaluation score directly with output logits. We use latest version of these LLMs, i.e., Baichuan2 and Vicuna-V1.5. For InstructGPT, we use latest gpt-3.5-turbo-instruct-0914. Regarding the gold evaluators and adversarial generator, the PALM-2 version we use is text-bison-001. The GPT-4 version is gpt-4-1106-preview. For all evaluation score S, we normalize it to the same scale of 0 to 100.

heading: A.2 Hyper-parameters
text: Note that our objective is to get responses of distinctly high quality (with scores exceeding 70) and those of distinctly low quality (with scores below 30). Accordingly, we set τ1 as 70 for R+ and 30 for R−. τ2 is set at 40 across evaluations. These thresholds are chosen to ensure that adversarial samples meet two requirements: 1) distinctly representing either extremely positive or negative labels; 2) having a clear gap in scoring between the victim and gold evaluators. The second requirement implicitly ensure incorrect classification by Svictim. For simplicity, α is fixed at 1 across all tasks without further tuning. We observe that this simple hyper-parameter setting has demonstrated strong performance.

heading: B Evaluation Prompt
text: In this section, we list the prompt used as input to our gold evaluator (Fig. 2, 3, 4). To boost the accuracy of our evaluator, the prompt can be manually crafted or automatically crafted using existing prompt optimization methods (Yang et al., 2023; Liu et al., 2023b) to align with human judgement. In this work, the dialogue response evaluation prompt is optimized from publicly available annotated data (Zhao et al., 2020; Mehri and Eskenazi, 2020b; Huang et al., 2020). The initial instruction provided is very straightforward and consistent with our evaluation dimension mentioned in Sec. 4.1: “Given the above context and response, rate the relevance of the response to the context on a scale of 1 to 100, providing only the numerical score as your output.” The extra learned criteria can be considered as auxiliary information to support the evaluation of our targeted relevance dimension (Li et al., 2024). Similarly, We use automatically optimized summarization evaluation prompt from previous work (Liu et al., 2023b). For question evaluation, we use manually crafted prompt. Note that all gold evaluation prompts ex-

heading: Context: [dialogue_history]

heading: Response: [candidate_response]

heading: 1. Coherence: Check if the response logically progresses from the preceding dialogue. It should align with both the immediate previous statement and the broader context of the conversation.

heading: 2. Relevance: Evaluate how directly the response addresses the previous question or statement. It should either answer the question or build upon the preceding statement.

heading: 3. Dialogue advancement: Determine how the response contributes to maintaining the conversation flow and promotes further interaction.

heading: 4. Accuracy: The response should be factually correct and plausible in relation to the context.

heading: Article: [article]

heading: Summary: [candidate_summary]
text: plicitly specify the output format as a single score number only. We take that score number as the gold score.

heading: C Attacking Proprietary LLM
text: To study the possibility of using AdvEval against powerful proprietary LLMs, we further extend AdvEval to attack InstructGPT evaluator on 50 data points from DailyDialog dataset. Given the similar strong performance of available proprietary LLMs, we employ an ensemble of GPT-4 and PALM-2 as our gold evaluator. The AdvEval R+ and R−
ASR are 24% and 84% respectively. Compared to smaller victim models, InstructGPT shows superior robustness against R+ perturbations, but remains vulnerable to R− perturbations. The results suggest the potential of using our proposed AdvEval framework to study the robustness issue of cuttingedge NLG evaluators, potentially guiding future improvements in evaluator robustness.

heading: D Additional Case Study
text: In Tab. 10, we present example adversarial data point from DialogSum with different attack tech-

heading: Article: [article]

heading: Candidate question: [candidate question]

heading: Provided answer: [provided_answer]

heading: 1. If the candidate question cannot be answered using only the information from the article, then directly return a low score.

heading: 2. If the candidate question is answerable, then verify whether the provided answer adequately addresses the candidate question.

heading: 3. In cases where the provided answer fully and accurately addresses the candidate question, award a high score. If it does not, a low score should be given.

heading: 4. If the provided answer can only partially address the candidate question, then assign a score around 50.
text: niques. Similar to the findings on dialogue and question evaluation, previous attack methods struggles in the NLG evaluation context, especially for the R+ generation, which reinforces the necessity and impact of our approach for enhancing evaluator robustness.

heading: E Human Validation of Gold Evaluator Labels
text: To verify the validity of gold evaluators, we instruct PhD students who are proficient in English to annotate adversarial data produced by our AdvEval framework, as well as those from TextFooler and BERT-Attack baselines. Drawing upon manual validation scale from previous studies (Mehri and Eskenazi, 2020b; Yeh et al., 2021; Mehri and Eskenazi, 2020a), a total of 780 data points are evaluated by 9 student annotators (each data point is annotated by three students). Regarding the data to be annotated, we sample adversarial data produced by AdvEval, BERT-Attack, and TextFooler. The data sources include DailyDialog, MuTual, NewsQA, SQUAD, CNN/Dailymail, and DialogSum. The victim models include BLEURT, PoE-Large, and Vicuna-13B for dialogue response; ROUGE, and UniEval for summarization; BLEURT, RQUGE, and Baichuan-7B for question generation. For each combination, we sample 10 R+ and 10 R− (if reference-free).
Instead of using 100-scale annotations, We design the annotation task as a binary classification that determines whether the output text is "good" or "bad" conditioned on the corresponding contexts and the scoring criteria. The instruction manual is presented in Tab. 11. This binary classification design aligns with Eq. 3 and the hyper-parameter choices detailed in Sec. A.2. With the constraints from Sgold part of Eq. 3, we only get responses of distinctly high-quality or low-quality, which can
be validated with binary manual annotations. Regarding the margin part of Eq. 3, we find that the
Svictim is much more extreme than bounds constrained by Eq. 3 on our final AdvEval adversarial data, which have an average Svictim of 18.1 for R+, and 85.4 for R−. That extremely low/high value of Svictim ensures that Evictim makes wrong prediction on our AdvEval data as long as the binary label computed from Sgold is correct. Therefore, our human verification process is naturally a binary classification problem instead of a regression problem. The rationale for generating binary good/bad adversarial responses is further supported by various evaluators’ training datasets, which are often constructed as positive and negative instances (binary labels) (Yeh et al., 2021).
The inter-annotator agreements are found to be 0.77, 0.77, and 0.68 for the dialogue, summarization, and question generation tasks respectively. Since achieving perfect accuracy is exceptionally challenging due to the inherent subjectivity of the task (Amidei et al., 2018; Zheng et al., 2023; Dubois et al., 2023), this inter-annotator agreements suggest good quality of the annotations. Note that the highest inter-annotator agreement in our study is 77%, which also shows the challenging nature of achieving complete alignment even among knowledgeable human annotators. Critically, the agreement rate between Egold and human annotators closely mirrors this highest interannotator score, indicating the reliability of our gold evaluation as human proxy to guide the optimization process.

heading: Future Work
text: The results suggest the potential of using our proposed AdvEval framework to study the robustness issue of cuttingedge NLG evaluators, potentially guiding future improvements in evaluator robustness