SlowGuess/NAACL_2022_OCR huggingface dataset:
Baber Khalid
110 Frelinghuysen Rd
baber.khalid@rutgers.edu
Sungjin Lee
300 Pine Street
Seattle, WA 98103
sungjinl@amazon.com
There is an increasing trend in using neural methods for dialogue model evaluation. Lack of a framework to investigate these metrics can cause dialogue models to reflect their biases and cause unforeseen problems during interactions. In this work, we propose an adversarial test-suite which generates problematic variations of various dialogue aspects, e.g. logical entailment, using automatic heuristics. We show that dialogue metrics for both open-domain and task-oriented settings are biased in their assessments of different conversation behaviors and fail to properly penalize problematic conversations, by analyzing their assessments of these problematic examples. We conclude that variability in training methodologies and data-induced biases are some of the main causes of these problems. We also conduct an investigation into the metric behaviors using a black-box interpretability model which corroborates our findings and provides evidence that metrics pay attention to the problematic conversational constructs signaling a misunderstanding of different conversation semantics.
Automatic evaluation of natural language models in general and dialogue models in specific has been a focus of ongoing research. The gold standard for evaluation of dialogues is human judgement (Meena et al., 2014; Ultes et al., 2013; Jang et al., 2020; Shim et al., 2021; Khalid et al., 2020b; Panfili et al., 2021) but human judgements are hard to obtain. Other than human judgements, dialogue simulations are used to judge different aspects of a model behavior (Jung et al., 2009; Eckert et al., 1997; Cuayáhuitl et al., 2010; Khalid et al., 2020a; Kreyssig et al., 2018; Sun et al., 2021a). Neural models of dialogue rely on text similarity metrics like BLEU, ROUGE or METEOR (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005), however these do not correlate well with the human
A: Do you like singing?
B: Yes, I do.
A: Let's go to a ETV tonight.
B: But I can't sing it well.
A: It doesn't matter. No one will laugh at you.
B: OK, I'll go with you. When shall we go?
Actual Response: What about six?
Adversarial Response: OK. Let's make it.
Human Score: 0.05; Adversarial Score: 0.43
Table 1: This example presents a problematic judgement by the DialogRPT metric. This metric scores an irrelevant response higher than a relevant and natural response. This example highlights the volatile nature of neural evaluation metrics.
judgement (Lowe et al., 2017).
Recent research focuses on the use of neural networks to tackle this problem (Kreyssig et al., 2018; Sun et al., 2021a; Lowe et al., 2017; Jang et al., 2020; Shim et al., 2021; Mehri and Eskenazi, 2020a; Pang et al., 2020; Gao et al., 2020; Kachuee et al., 2021). However, neural methods are known to be 1) poor at dealing with out-of-distribution data, 2) very hard to explain and 3) hard to train because of data-availability issues. An example of poor behavior by a neural evaluation metric, DialogRPT (Gao et al., 2020), is shown in the table 1. As these metrics are used in conjunction with different training methods to improve the quality of intelligent conversation models (Kachuee et al., 2022; Park et al., 2021), these problems and biases can trickle down into the trained models. Therefore, it is important to formulate techniques which can fix the existing problems in the evaluation metrics.
In this work, we present an adversarial test-suite which uses automatic heuristics to generate adversarial examples targeting specific aspects of dialogues e.g. logical entailment or natural vocabulary. Performance analysis of metrics on these adversarial examples provides insights into their assessments of different problematic behaviors and lets us deduce problems they face while judging these behaviors. We also use a black-box interpretability
Human Language Technologies, pages 5871 - 5883
technique, a modification of Kernel SHAP from Lundberg and Lee (2017), to find important language features for the metrics to provide additional insights into the behavior of evaluation metrics. This test-suite is meant to provide a novel benchmark for community which can be used to analyze the performance of proposed metrics and can be improved with further research.
2.1 Language Model Evaluation Test-Suites
There have been several test-suites which compare the performance of the language models on contrasting examples. Both Gauthier et al. (2020); Warstadt et al. (2020) compute surprisal for sentence pairs, where one of the pair has a syntactical mistake, to test if a language model finds the syntactically wrong sentence more surprising. Beyer et al. (2021a); Pishdad et al. (2020) both rely on this concept of calculating surprisal but use pairs of coherent and in-coherent language uses. However, they don't release heuristics for automatic generation of adversarial cases and just focus on coherence-based manipulations while we go one step further to see the effect of other manipulations on various core aspects of dialogue in an automated way. Ribeiro et al. (2020) presents a tool which evaluates language models with their performance on pre-determined tests and their outcomes.
2.2 Adversarial Evaluation Techniques
Previous works have tried to use adversarial evaluations to judge the performance of dialogue models. Cheng et al. (2019) successfully trains a RL agent using adversarial rewards against a negotiation dialogue agent and reduces its effectiveness. Jia and Liang (2017) proposes an adversarial attack where adding extra sentences in the comprehension text reduces the performance of comprehension models significantly. The closest work to ours is Sai et al. (2019) which evaluates a neural metric (Lowe et al., 2017). It shows effectiveness of simple syntactical manipulations, like reversing a sentence, in fooling the metric. We, however, rely on attacking more complex semantics using simple heuristics, like breaking co-reference chains, to pinpoint metric performance fluctuations.
2.3 Interpretability Techniques
There have been several works proposing algorithms to explain the predictions of neural networks. A measure of co-operative game theory used to measure marginal contribution of players in a game is called Shapley value (Shapley, 1952). It has been a focus of attention for ML community to explain predictions of neural models. LIME (Ribeiro et al., 2016) and trumbelj and Kononenko (2013) present methods which rely on feature perturbations to generate explanations. Bach et al. (2015) presents a method which provides explanation in form of pixel-based heatmaps. Lundberg and Lee (2017) modifies the methods mentioned earlier to approximate Shapley values. Li et al. (2017a) presents the variations in model outputs by erasing several input features as a measure of importance. We use a modification of Kernel SHAP from (Li et al., 2017a) to approximate feature importance in this work.
3 Conversation Properties of Interest
Dialogue systems- specially those based on neural architectures, show poor performance in generating consistent responses by keeping track of information in a dialogue context and are prone to generating repetitive, unnatural and bland responses (Yang et al., 2021; Khandelwal et al., 2018; Gao et al., 2018). These problems also guide the research in the area of dialogue system evaluation with many metrics focusing on evaluating coherence and consistency of dialogue models (Lai and Tetreault, 2018; Beyer et al., 2021b; Gao et al., 2020; Pang et al., 2020; Sun et al., 2021b). These findings help us choose relevant dialogue attributes to manipulate so the insights from the performance of different metrics can be used to address different problems in dialogue modeling.
To generate adversarial examples we focus on the following aspects of conversations: 1) coherence 2) naturalness 3) interestingness. We manipulate each of these attributes by targeting specific aspects which contribute to these: Coherence: i) entailment ii) pronoun use iii) named-entity use iv) co-reference v) contradictions vi) speaker sensitivity Naturalness: vii) unnatural repetitions viii) vocabulary diversity ix) unnatural paraphrasing x) natural paraphrasing xi) entrainment Interestingness: xii) dullness. One example for manipulation of each of the major attributes is shown in the table 2 and other examples are presented in the appendix A.
Coherence	Naturalness	Interestingness
Contradiction	Unnatural Paraphrasing	Dullness
A: Could you explain what you saw?
B: I was in the bank at the time of the robbery.
A: What did you see?
B: I saw a man come in with a gun.
A: Did you see his face?
Actual: No . He had a mask on.	A: You saved my life yesterday, Rachel.
I can’t believe I forgot to bring my wallet when we went to lunch with those clients.
B: It was a good thing I had enough on me.
Actual: Let me buy you lunch today to pay you back.	A: What are you doing tonight?
B: I have to run to the grocery store.
A: Don’t you hate fighting the crowds on the weekends?
Actual: Yes, but I am out of food and milk.
Adversarial: allow me purchase you luncheon today to give you back.	Adversarial: I don’t have a good answer to that.
Table 2: This table showcases one adversarial example for each dialogue attribute we target: coherence, naturalness and interestingness.
4 Adversarial Techniques
We specifically focus on four conversational datasets Daily Dialogue (DConfig) (Li et al., 2017b), Persona Chat (PerCh) (Zhang et al., 2018), Reddit (Gao et al., 2020) and MultiWOz (Budzianowski et al., 2018) to test our heuristics. We test an implementation of our heuristics using these datasets and release it with this work<sup>1</sup>.
In accordance with the given datasets, we consider a conversation  $D$  as a series of utterances between two alternating speakers  $(x_{1},y_{1},x_{2},y_{2},\ldots ,x_{n},y_{n})$ . We formulate the adversarial conversation generation problem as follows: Given a conversation snippet  $C_i = (x_1,y_1,\dots,x_i,y_i,x_{i + 1})$ , from a randomly sampled conversation  $D = (x_{1},y_{1},\dots,x_{n},y_{n})$ , we generate an adversarial conversation snippet  $C_i' = (x_1,y_1,\dots,x_i,y_i,x_{i + 1}')$ . This helps us learn the impact of different responses to the same context on neural conversational metrics.  $x_{i + 1}^{\prime}$  is generated using adversarial heuristics which are explained as follows:
4.2 Coherence
The purpose of these attacks is to disturb the logical flow of human conversations and investigate how different evaluation metrics react to these disturbances.
4.2.1 Entailment
In this case, the adversarial response  $x_{i+1}'$  is either a randomly sampled utterance from a random conversation in the dataset or a random response from the following:  $(y_{i+1}, x_{i+2}, \ldots, x_n, y_n)$  from  $D$ . (See Appendix A for example.)
4.2.2 Pronoun Use
Given  $x_{i+1} = (w_1, w_2, \dots, w_k)$ , where  $w_j$  is an utterance token, we manipulate  $w_j \subset P$ , where  $P$  is the set of pronouns, in the following manner: 1) We convert gender specific pronouns to object pronouns e.g. he/she to it/they and vice versa. 2) We convert singular pronouns it/this/I to the plural ones and vice versa. 3) We convert 1st person pronouns like I/we/our to a random 2nd/3rd person pronoun or vice versa. (See Appendix A for example.)
4.2.3 Named Entities
We replace named entities in  $x_{i+1}$  with 1) another random entity of the same type e.g. person with person 2) a named entity of different type e.g. person with company to generate  $x_{i+1}'$ . (See Appendix A for example.)
4.2.4 Co-Reference
Human conversations have extensive use of references to different entities being discussed. We use a pre-trained reference resolution model to detect co-reference clusters in the conversation snippet and replace references in  $x_{i + 1}$  to generate  $x_{i + 1}'$ . This results in adversarial  $x_{i + 1}'$  which means the same thing as  $x_{i + 1}$  and should be assigned a similar score by an ideal metric. (See Appendix A for example.)
4.2.5 Speaker Sensitiveness
Given a conversation snippet  $C$  which ends at a response  $x_{i + 1}$  to a question-answer pair  $(x_i,y_i)$ , we augment the response  $x_{i + 1}$  by concatenating it to the answer  $y_{i}$  to the question  $x_{i}$  to generate the adversarial  $x_{i + 1}^{\prime} = [y_{i},x_{i + 1}]$ . This creates situations which contain unnatural redundancies because a speaker does not acknowledge the answer
$^{1}$ https://github.com/baber-sos/Explaining-Dialogue-Evaluation-Metrics-using-Adversarial-Behavioral-Analysis
to a given question. A simple and effective heuristic to detect question answer pairs is to find an utterance with the question-mark ?. (See Appendix A for example.)
4.2.6 Contradictions
Here we focus on generating adversarial  $x_{i+1}'$  which contradicts the contextual information in a given conversation  $C$ . We generate  $x_{i+1}'$  by augmenting  $x_{i+1}$  in the following manner: 1) by negating the verbs in  $x_{i+1}$  using simple heuristics, 2) by replacing named entities in  $x_{i+1}$  occurring at least twice in  $C$  with a random entity of same type or 3) by replacing references in  $x_{i+1}$  to different named entities with the random entities of same type. For the verb negation, we either add a not after an auxiliary or add did not and does not depending on plurality of the subject and tense of the sentence while changing the verb form if needed. (See the table 2 for example.)
4.3 Naturalness
Humans prefer some vocabulary or style choices over others and a good evaluation metric should respect this behavior. We disturb these natural choices in the following manner:
4.3.1 Unnatural Repetitions
We augment  $x_{i+1}$  to generate  $x_{i+1}'$  by randomly sampling and repeating  $k$  times 1) non-stop English words in  $x_{i+1}$  or 2) multi-word noun phrases to generate unnatural repetitions in the adversarial response. (See Appendix A for example.)
4.3.2 Vocabulary Diversity
We replace randomly sampled words  $w_{j}$ , which have a synonym in the conversation  $C$ , from  $x_{i+1}$  with one of their synonyms in  $C$ . This results in  $x_{i+1}'$  with vocabulary choices which may not be as natural. An ideal metric would assign these examples a score no better than their human generated counter-parts. As a simple heuristic, we consider two words  $(w_{j}, w_{k})$  as synonyms if they are in synset of each other as specified in the Word-net(Miller, 1995). (See Appendix A for example.)
4.3.3 Unnatural Paraphrasing
We generate unnatural paraphrase  $x_{i+1}'$  by replacing randomly sampled non-stop English words from  $x_{i+1}$  with their synonyms from Wordnet. For synonym sampling, we rank the Wordnet synonyms according to word2vec embedding (Mikolov et al.,
2013) similarity and sample from the top  $k$  synonyms. To make paraphrasing unnatural,  $1/4^{th}$  of words are replaced with synonyms which are least likely to be used by humans while keeping it grammatically correct. (See the table 2 for example.)
4.3.4 Natural Paraphrasing
We generate natural paraphrases using a T5 model fine-tuned on the PAWS (Zhang et al., 2019) dataset. T5 model makes minor structural changes to  $x_{i+1}$  like removing punctuation or re-organizing utterances while using same vocabulary. An ideal metric should assign both  $x_{i+1}$  and  $x_{i+1}'$  similar scores. (See Appendix A for example.)
4.3.5 Entrainment
To disturb entrainment, we replace non-stop English words  $(w_{j} \in x_{i+1})$  used by both parties involved in the conversation with a synonym sampled from the top  $k$  synonyms in the Wordnet. (See Appendix A for example.)
4.4 Interestingness
Interestingness is a subjective measure but humans prefer interesting conversations over the dull ones. We consider those responses interesting which progress the conversation in a natural way than those which do not add anything meaningful to it.
4.4.1 Dullness
To generate a dull  $x_{i+1}'$  we either 1) replace  $x_{i+1}$  with an utterance from a set of generic responses, 2) replace an answer in a QA pair with a generic answer or 3) replace  $x_{i+1}$  with one of following:  $(x_1, \dots, x_i)$  which results in the repetition of a speaker contribution. (See the table 2 for example.)
5 Experiment Results and Analysis
We use three different evaluation metrics DialogRPT (Gao et al., 2020), Pang-Evaluation (Pang et al., 2020) and User Satisfaction (Sun et al., 2021b) to test our adversarial test-suite. The first two metrics we test are for open-domain conversations while the last one is trained to judge task-oriented conversations. We focus on both open-domain dialogue and task-oriented dialogue metrics to highlight the diversity of our test-suite and show how neural metrics in both of these settings contain fundamental problems. Both DialogRPT and User Satisfaction metrics use human feedback
Metrics	Datasets	Entailment	Pronoun	NER	Co-reference	Speaker Sensitive-ness	Contradic-tion	Dullness
DialogRPT	DDial	0.3	0.48	0.6	0.56	0.98	0.78	0.48
Reddit	0.13	0.66	0.64	0.32	0.91	0.72	0.53
PerCh	0.4	0.55	0.74	0.45	0.89	0.87	0.47
Pang-C	DDial	0.46	0.08	0.49	0.03	0.01	0.26	0.78
Reddit	0.5	0.49	0.41	0.58	0.04	0.47	0.9
PerCh	0.47	0.2	0.53	0.74	0.00	0.51	0.86
Pang-L	DDial	0.45	0.75	0.72	0.23	0.74	0.32	0.67
Reddit	0.49	0.63	0.57	0.35	0.59	0.29	0.81
PerCh	0.54	0.76	0.73	0.35	0.69	0.33	0.81
UWBERT	0.98	1.00	1.00	0.01	1.00	0.97	0.98
WHiGRU	MultiWOz	0.93	0.98	0.94	0.07	0.96	0.96	0.95
WBERT	0.98	0.97	0.97	0.01	0.96	0.98	0.94
Table 3: Error rates of different evaluation metrics on different datasets for coherence and interestingness attacks.
Metrics	Datasets	Unnatural Repetitions	Vocabulary Diversity	Unnatural Paraphrases	Natural Paraphrases	Entrainment
DialogRPT	DDial	0.86	0.14	0.42	0.44	0.62
Reddit	0.90	0.08	0.42	0.35	0.59
PerCh	0.91	0.08	0.53	0.41	0.67
Pang-C	DDial	0.02	0.00	0.00	0.87	0.46
Reddit	0.05	0.02	0.15	0.33	0.59
PerCh	0.01	0.06	0.00	0.89	0.23
Pang-F	DDial	0.99	0.88	1.00	0.97	1.00
Reddit	0.80	0.64	0.88	0.44	0.93
PerCh	0.87	0.83	0.98	0.87	0.94
UWBERT	MultiWOz	1	0.01	0.98	0.04	1
WHiGRU	0.96	0.02	0.98	0.1	0.95
WBERT	0.99	0	0.98	0.03	1
Table 4: Error rates of different evaluation metrics on different datasets for naturalness attacks.
during the training while components of Pang-Evaluation rely on pre-trained neural backbones. We postulate that this variability in training methods might help provide specific insights for different methods.
From our analysis of metric behavior on adversarial conversations, we have the following key takeaways: 1) neural metrics are not suited to judge overall quality of the conversations; 2) they are unable to correctly understand conversation semantics; and 3) they are prone to data and training-induced biases.
5.1 Noise in Adversarial Heuristics
Proposed heuristics are expected to fail in some cases while generating adversarial conversations. To estimate this failure, we manually analyze the 250 generated conversations across all attacks and find that heuristics work successfully  $84\%$  of the time. This signifies that heuristics are succeeding most of the time but requires further investigation for a more detailed estimate of their noise.
5.2 Analysis Methodology
An ideal metric would grade adversarial conversations worse than the original conversations in most cases, as most of the attacks are focused on generating bad behaviors. However, natural paraphrasing and attacks on reference use are expected to be graded similarly to the original conversations as they generate similar conversations as the original ones. Similarly, conversations generated by vocabulary diversity and entertainment attacks are expected to be scored at most as well as the original conversations as both of these manipulate vocabulary choices and sometimes these manipulations result in natural outcomes. To capture this variability, we define error rate as the proportion of times a conversation metric does not conform to the expected behavior.
We sample 100 adversarial conversations for each attribute per dataset and analyze the metric performance on those. We compute the proportion of times the metrics rate original conversations higher, equal and lower than the adversarial ones and use these statistics to compute the error rates. To make sure, we don't categorize insignificant
changes as greater or lesser we compute a minimum score threshold by computing the minimum change from the mean of metrics scores assigned to human conversations required to get a p-value less than  $0.05$  ( $p < 0.05$ ) in a t-test. We use pre-trained NER model in the spacy python package (Honnibal and Montani, 2017) for named-entity detection and reference resolution models in (Clark and Manning, 2016a,b) to detect co-reference clusters.
5.3 DialogRPT
This is an evaluation metric which was trained on Reddit threads using human feedback in the form of up/down votes, number of replies, and the depth of a conversation thread. It also has a component focused on separating human from random utterances. DialogRPT requires significant improvements when dealing with the most adversarial cases as shown by error rates calculated in tables 3 and 4. It performs the best on the entailment task on Reddit data because it was trained to pick relative human responses from random ones. Additionally, it performs better on Reddit data on average than other datasets which provides evidence for better performance on in-distribution data.
It is evident that DialogRPT favors repetitions when we analyze adversarial cases which are generated using repetitions e.g. speaker sensitive cases in table-3 and unnatural repetitions cases in table-4. DialogRPT has a  $93\%$  error rate out of which  $17\%$  of the adversarial conversations are rated similar to the original for the speaker sensitivity. This proportion of similar ratings increases to  $57\%$  in case of unnatural repetitions. The fact that conversations in speaker sensitive case are rated higher more often provides evidence for a bias towards rating conversations with similar responses to context.
The performance of DialogRPT on attacks other than entailment is not good with error rates greater than  $40\%$ , which provides evidence that DialogRPT does not have a correct understanding of different dialogue aspects especially contradictions and individual speaker contributions.
5.4 Pang-Evaluation Metric
Pang-Evaluation presents four evaluation metrics which try to evaluate specific properties of dialogues. 3 of these depend on a pre-trained neural backbone: i) context-coherence (Pang-C) ii) fluency (Pang-F) iii) logical consistency (Pang-L). Pang-C and Pang-F rely on a GPT2 (Radford et al., 2018) backbone fine-tuned on Daily Dia
logue dataset while the Pang-L relies on a Roberta model (Liu et al., 2019) fine-tuned on MNLI task (Williams et al., 2018) to detect contradictions.