Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences
Authors: Eleftheria Briakou
Navita Goyal
Marine Carpuat
Venue: EMNLP 2023 Main (2023)
Keywords: explainability
human-centered evaluation
machine translation evaluation
cross-lingual semantics
contrastive highlights
TL;DR: We introduce an approach to explain the predictions of NLP models that compare two texts by showing what differences between the inputs led to a prediction. We show that it helps people detect meaning differences in human and machine translations.

Abstract: Explainable NLP techniques primarily explain by answering "Which tokens in the input are responsible for this prediction?". We argue that for NLP models that make predictions by comparing two input texts, it is more useful to explain by answering "What differences between the two inputs explain this prediction?". We introduce a technique to generate contrastive phrasal highlights that explain the predictions of a  semantic divergence model via phrase alignment guided erasure. We show that the resulting highlights match human rationales of cross-lingual semantic differences better than popular post-hoc saliency techniques and that they successfully help people detect fine-grained meaning differences in human translations and critical machine translation errors.

Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences
Eleftheria Briakou'; Navita Goyal', Marine Carpuat "Google,* 2 University of Maryland ebriakou@google. com, {navita, marine)@cs. .nd. edu
Abstract
EL To μērpa nou epapuóorerpey owy Eddáöa 6éxōrpeay moldd deruck ayóλvz amd tov Egvo vuro now strpe try Ei- Adña yua tov nepropronó xápa.
Explainable NLP techniques primarily explain by answering "Which tokens in the input are re- sponsible for this prediction?". We argue that for NLP models that make predictions by com- paring two input texts, it is more useful to ex- plain by answering "What differences between the two imputs explain this prediction?". We introduce technique to generate contrastive phrasal highlights that explain the predictions of semantic divergence model via phrase- alignment-guider erasure. We show that the resulting highlights match human rationales of cross-lingus semantic differences better than popular post-hoc saliency techniques and that they successfully help people detect fine- grained meaning differences in human transla- tions and critical machine translation errors.
Tnp stánkaamys vrş navōnjlaa ow
GLOSS The measures put place Greece received many positive commentt from the foreign press for having slowed the spread of the disease in the country.
EN The measures put in place in Greece are among the most proactive and strictest in Europe and have been credited internationally for having slowed the spread of the disease.
The color-coded contrastive phrasal highlights explain meaning differences hetween Greek sentence (EL) and its English translation (EN) from Wikipedia.
Figure I:
In this work, we introduce a post-hoc explain- ability technique to answer "What differences be- tween two inputs explain a prediction" "? for NLP regression models that produce a score encoding a relationship between two text inputs. We apply it to the task of explaining meaning divergences be- tween a text in one language and its translation. We design a parrae-alignmentt to explain the predictions of a semantic divergence model and highlight minimal phrase pairs in the two inputs that result in meaning differences.
1 Introduction
A common strategy to explain the predictions of NLP models to highlight salient tokens in their inputs (Li et al., 2016; Ribeiro et al., 2016; Lund- berg and Lee, 2017). However, this is suboptimal for the many NLP tasks that require comparing and contrasting two or more pieces of text to predict a class or similarity score, such as natural language inference (Bowman al., 2015), semantic textual similarity (Agirre et al., 2012), or evaluation and quality estimation of text generation (Bojar al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020; Freitag et al., 2021b, 2022). At the same time, a long line of research in social sciences shows that human explanations are contrastive (Miller, 2019), i.e., humans do not explain why an event happened in vacuum but instead compare what happened contrast case. For instance, to explain how a Greek sentence and translation in English dif- fer, it is more intuitive to show how they differ (as presented in Figure 1) than to highlight all salient Greek and English tokens without specifying how they relate to each other.
erasure strategy
We extensively evaluate our approach with both proxy and human-centered evaluations. Compar- ing against human explanations for meaning differ- ences in English-French human translations, our approach matches human rationales better than popular post-hoc methods that highlight salient tokens independently (Ribeiro et al., 2016; Lund- berg and Lee, 2017). Acknowledging the limita- tions of such proxy evaluations (Buçinca et al., 2020), we conduct two IRB- -aprrveee human stud- ies to assess the usefulness of our contrastive phrasal highlights when crowd-soureing challeng- ing bilingual annotation tasks. We find that our approach helps annotators reliably discover fine- grained meaning differences in English-French and English-Spanish human translations and detect crit- ical errors in Portuguese- to- to-English machine trans-
Work done while at the University of Maryland.

---

lation. We make our code and data publicly avail- able: https://github. com/ wtt.eoooommaaaaaeeee Elbria/Ex Setorri//xx-eDDiiie SemDiv.
other family of approaches extracts highlights con- trastively: Jacovi and Goldberg (2021) extend erasure-based approaches to identify which input features lead to the actual prediction vs. a con- trastive prediction, while Yin and Neubig (2022) extend saliency methods to explain the generation of contrast cases for language generation tasks. Our work explains the predictions of models that com- pare two inputs, where identifying salient phrase pairs that differ across them provides natural mechanism for contrastive explanations.
2 Background
We focus on local explanations which explain why a specific prediction was made, often to help hu- mans rely predictions appropriately in machine- in-the-loop use cases (Doshi- Velez and Kim, 2017).
Highlight Explanations Local explanations of- ten consist of highlighting input features deemed important to a prediction. In glass-box settings, prior works seek to quantify feature saliency with model-internal signals such as gradients (Arras et al., ;0 Smilkov et al., 2017), contextual de- compositions (Murdoch et al., 20, attention mechanisms (Choi et al., 2016). However, these highlights' ability to serve as explanations has been questioned (Jain and Wallace, 2019; Wang al., 2020; Moradi et al., 2021), leading to work revisit- ing technical solutions and evaluations under which attention could be seen as an explanation (Wiegr- effe and Pinter, 2019; Moradi et al., 2019; Tutek and Snajder, 2020). Another family of approaches relies on crasure strategics to explain the predic- tions of black box models. Li et al. (2016) quantify token's 's importance the difference in model confidence once it is erased from the input, with several following works exploring different token erasure schemes (Feng et al., 2018; Kim et al., 2020). LIME (Ribeiro et al., 2016) approximates the model's local decision boundary with linear models by erasing multiple tokens from a model's input, while SHAP (Lundberg and Lee, 2017) computes Shapley values by estimating the marginal contribu- tion of each token across all possible erasures. We build on this line of work by designing an erasure strategy that identifies salient contrastive phrasal pairs rather than independent salient tokens.
Evaluation
Most current work in NLP adopts prexy evaluations that compare automatic expla- nations with human explanations of the gold la- bel, with numerous datasets encoding explanations using highlights or free-form text (Wiegreffe and Marasovic, 2021). Others rely on simplified human tasks, such as simulatability, evaluating whether explanations help people predict a model's pre- diction more accurately (Hase and Bansal, 2020; Nguyen, 2018). Despite the attractiveness of the above evaluations, evidence growing that they do not reliably indicate how useful explanations are in practice (Buçinca et al., 2020). Boyd-Graber et al. (2022) call for opplication-rrounded evaluations to directly assess whether explanations help peo- plc complete an actual task, suggested by work in HCI (Suresh al., 2021: Liao et al., 2020). We heed that call by complementing a proxy evaluation with two application-grounaee user studies.
Explaining Translation Differences Detecting translation differences is a core task in multilin- gual NLP, e.g., to predict machine translation errors (Rei et al., 2020), to study how humans translate (Zhai et al., 2020), or to understand how multi- lingual resources such as Wikipedia differ across languages (Miquel-Ribe and Laniado, 2018; Jiang et al., 2017). Automatic approaches often score the degree of (dis)similarity between an input and its translation to quantify machine translation qual- ity (Zhang et al., 2019; Sellam et al., 2020; Rei et al., 2020) or to detect meaning divergences in human translations (Vyas et al., 2018; Briakou and Carpuat, 2020; Wein and Schneider, 2021). However, sentence-level scores can be hard in- terpret and do not pinpoint how translations differ. This has been addressed by tagging text with hu- man translation processes (Zhai et al., 2018), with dedicated, supervised models or with word-level quality estimation (QE) tasks (Specia et al., 2018,
Contrastive Explanations Motivated by social science research suggesting that humans explain with respect to an (implicit) contrast case (Miller, 2019), recent work has sought to generate the lat- ter by producing counterfactual explanations: min- imally edited versions of the input that change the model prediction. Performing such edits usu- ally requires training dedicated editors (Ross et al.. 2021). prompting language models (Paranjape al., 2021), and accessing knowledge bases (Chen et al., 2021), among others (Li et al., 2020; Chem- mengath et al., 2022). Closer to our work, an-

---

2020, 2021), often addressed by applying post-hoc explanation methods sentence-level predictors (Treviso et al., 2021; Rei et al., 2022). We augment this work by contributing unsupervised contrastive explanations of translation differences and testing whether they help annotate them more reliably.
rive word alignments based on multilingual word embeddings (Jalili Sabet al., 2020) and then ex- tract all phrase pairs consistent with the word align- ments. A phrase pair (P1,P2) is consistent with the word alignment if p1 and p2 are each contiguous sequences of words in each language, where all alignment links that start within p1 map to tokens within P2 and vice versa. Unaligned words (i.e.. words aligned to null token) can be included in either the source (p1) or the target (p2) phrases. As result, the extracted phrase pairs comprise not only equivalent but also include related but diver- gent phrase pairs ( ("national anthem" and "chant patriolique") and unaligned phrase ("served for many years"), as seen in Figure 2.
3 Explaining Divergences with Contrastive Phrasal Highlights
We introduce highlighting approach to explain the predictions of model that compares and contrasts two text inputs. We hypothesize that phrase pairs provide more appropriate cognitive chunks (Doshi- Velez and Kim, 2017) than tokens basic units of explanations. For instance, instead of highlighting tokens that differ in the English- Freschh pair Fig- ure 2, is more natural to explain that the English text refers to the Maple Leaf Forever as a national anthem as opposed to a chant patriotique (patriotic song) in French and that the English phrase "for many years" " has equivalent in French.
3.2 Explaining by Minimal Contrast
Given the sentence pair S and its aligned phrase table P. we first extract set of contrast cases P consisting of all the phrasal pairs that, once crased, make the two inputs more equivalent, as measured by an increase in score larger than margin e:
Building on token-level crasure-based explana- tion methods (Li et al., 2016), we design phrase- alignment- dieemment-guided erasure strategy to explain the predictions of model (Briakou and Carpuat, 2020) that ranks bilingual sentence pairs based on the granularity of their semantic similarity (assuming S is an equivalent pair and $ pair containing se- mantic divergences, the model R()) ranks them such that R(S) > R(S)). As shown in Figure given two input texts, we first extract set of candi- date counterfactual inputs by masking a subset of phrase pairs ($3.1). Then, we explain the model's prediction by selecting the phrase pair whose era- sure maximally increases the similarity score be- tween the two inputs ($3.2).
P = P-fDER
{EE,, s.t. (DEL(((:)) > R S +e
Since explanations should be minimal (Hilton, 2017: Lipton, 1990), covering only the most rele- vant causes to reduce cognitive load, we select as contrastive phrasal highlights the phrase pairs that correspond to minimal edits with a large impact on predictions, as follows:
arg max R DEL (DDEEE(S (DELL(::P) BR Ss }
3.1 Guiddd Phrasal Erasure
e T if R (DELL 2
We propose an erasure approach that takes into ac- count the input's structure. Given a sentence pair S. we start by extracting set of candidate counterfac- tual instances by deleting a single phrase pair from
BR(SS
otherwise
where the first term ((DEI.((:)) encourages the extraction of a contrastive phrasal highlight p corresponding to a high score under the model (i.e., deleting this phrase pair yields contrast case), while the second term, (S,(s))) corresponds to brevity reward that encourages extraction of shorter highlights. [S] is computed by adding the length of each of the two sentences, and DEL. is a function that erases a phrase pair p from S.
Given that erasing all possible phrase pairs for each sentence is computationally prohibitive, we restrict our search to deleting phrases that belong to an aligned phrase table, P.
Our phrase pair candidates for erasure de- rived from classical statistical machine translation techniques developed for extracting phrase transla- tion dictionaries from parallel corpora (Och et al., 1999; Koehn et al., 2007). Given two texts, we de-

---

Leat Forever Canadian national onthem
The Mapill lect Forewer opu est un chant crotiolcque pro canaden
Glsaz This Mapple Lecf Forever is Conadion potiolic
INPUT
STEP T: AUGGNMMENT GUDED BRASURE
STEP 2: EXPLAIN BY MINIAAL CONTRAST
Roscot For
The Mapie localf Foreval wived mony Canadir national ariherm The Maple Leail Forevel qui est un chont patrioluue pro conodinn
(R
agav
The Mapie Leaf Forever served many @i Caradon potrodannoooonn nollond aitthar Mopee Leaf Forever qui est chant pottolque pro conoden
Ps
The Magie Leaf Forever sevved lor mony yecas ds Canador pational ontthem
Mapie served locmann Canacior onthomm Forever
conaden
Loon contes
Siwelyy leverd
The Waple loaf Porever served for many years Conaciion Forever qui caradien
OUTPUT
Conodion
The Maple Loof Forovor qui est cnant patfiliquu canacien
Figure 2: Our approach takes input sentence pair (S) and extracts of perturbed inputs by erasing phrasal pairs guided by word alignments (Step 1). Then it explains the prediction of a regressor R(S) by highlighting the phrasal pair p that, once deleted, maximizes the model's 's prediction (DEL(SS pJ) multiplied by brevity reward BR(SSp) p) that encourages the extraction of short phrasal pairs (Step 2).
The above approach identifies single phrase pair that explains the divergence prediction for the orig- inal sentence S. To extract multiple explanations, we repeat this process by iteratively erasing the extracted contrastive phrasal highlight from the current input sentence S' = DEL. p: and repeat the steps in $3.1 and $3.2. This iterative process ends when, for a given input, none of the extracted counterfactual instances yield more equivalent pair, i.e., P Ø, or we reach an equivalent input under the divergent ranker, i.e., R(S')= > 0.
methods, LIME and SHAP, that seek to explain the predictions of the explanandum.
Reference Highlights We evaluate our approach using the REFRESD dataset, which is manually annotated with divergences of fine-grained and course-grained granularity, along with rationales that justify the sentence label. We compare our contrastive phrasal highlights with the human ra- tionales on a subset of 418 challenging instances annotated as having "Some Meaning Differences" at the sentence level, where we expect the more subtle differences in meaning to be found.
4 Proxy Evaluation
In this section, we describe our proxy evaluation based on human-pre poooided highlights. We acknowl- edge that proxy evaluations encounter validity is- sues (Boyd-Grabee et al., 2022) and use them pri- marily to guide system development and validate against standard highlighting methods.
Evaluation Metries Following prior proxy evalu- ations of explanations (Ross et al., 2021; De Young et al., 2020), we compute: 1. Agreement with Hu- man Rationales: Precision, Recall, and F-1 scores computed against human rationales; 2. Minimal- ity: the length of the contrastive phrasal highlights, measured the number of highlighted tokens.
4.1 Experimental Setup
Explanandum We scek to explain the prediction of divergence ranking model that is trained as recommended in prior work (see Appendix A.l).
4.2 Results
As seen in Table I, explaining divergence predic- tions by extracting contrastive phrasal highlights significantly outperforms both L.ME and SHAP.
Explainers We contrast our contrastive phrasal highlights against a RANDOM baseline that high- lights tokens in each sentence at random with 0.5 probability, and two standard post-hoc explanation
Those standard highlighting baselines even under-
'https: /t:::://iiii wtw.ttthb..cooomm wwwp//bbrra/xlinn-eeeeeee tree tite/mastert ReterrRRE /REFreeD

---

ENGLISH
FRENCH
PR.
RE
F1
DEL.
PR.
RE.
DEL.
ORACLE
100
100
100
39%
100
100
100
43%
RANDOM
39
43
50%
49
50%
LIME
45
37
41
30%
44
37
40
34%
SHAP
53
34
41
25%
50
32
39
26%
OURS (- BR)
52
76
62
54%
56
74
64
55%
OURS
58
61
62
37%
60
55
64
37%
Table 1:
Proxy evaluation results with respect to human rationales on REFRESD.
The older peornaionn turtines genense kilonorts, and the modern tunhines instilleed penerate up to megrwatts depending on specific furbine and manifacturer. totiines mcieme géndration pérärrm kiloslis, alons que los Guliernes mudernss uni und puisannce pousant aller jusqp' mépooctts. (giozs) The rubrter of the old gonenatonn getenate Klowatts, while tie modere wind purbtee genotate up regemaatty of powerr
gonension ponente kilmons, unhines gésératinn
uchines instillss genaatte up megrossts, ileperding on specific tarhinz kilowatts que dolicence muderass use puissancs pouvani ailer jusqo' mággwarss
LINE
lilowans, modera tubiaes whites ginération génerem:
éobennes modernes megaraass
iubaae
jusqu' idgowatts.
OURS
The odde generation turbines generate lillwats,, and the mudeea tuhiiess instaalld gemorate megasaats, dependllng on the specillie tox bine and manufacturer Les uubiinss ancienue génération gênerem: des kilowats, alon que les dolienes modernes joni une pulssance pouvam aller jusqu' mégawaats.
Table 2: Examples of divergence explanations (HUMAN corresponds REFRESD rationales).
perform the RANDOM baseline. 2 A closer look at the outputs (Table 2) indicates that both LIME and SHAP suffer from two major issues: sparsity and accuracy. We attribute those issues to the fact that both approaches operate in token space, ignoring the interdependent relationships between tokens across the two languages. By explicitly modeling such relationships, our approach produces high- lights that match the ones in REFRESD better.
ation, proxy evaluations can be misleading. Given that prior work evaluates highlight bageeeght-based based explana- tions of (machine) translation errors based solety on automatic evaluations, we chose to initiate a human-centered discussion that evaluates the use- fulness of such highlights directly with in de-thhh exploration of multiple small-scale application- grounded evaluations as detailed below.
Application Evaluation I: Annotation of Semantie Divergences
We measure the impact of the brevity reward by ablation (i.e., OURS (- BR)). This achieves the highest recall the expense of precision: dropping the brevity reward produces fewer and longer high- lights per instance. It highlights more than 50% each sentence pair on average, while humans only highlight about 40%. The brevity reward thus helps match reference highlights better, support- ing the benefits of producing minimal explanations.
As shown by prior work, annotating fine-grained divergences is challenging task and requires ded- icated annotation protocols based human tionales (Briakou and Carpuat, 2020) or abstract meaning representation frameworks (Wein et al., 2022) to achieve moderate agreement. As result, such annotations are usually hard to collect with crowd workers since they require explicit annotator training. In this section, we ask whether contrastive phrasal highlights help growd-worken annotate di- vergences more reliably, which could case the need for complex and expensive annotation protocols. Concretely,
sum, these results are promising indicators that contrastive phrasal highlights could be helpful when detecting semantic divergences. However, highlighted by recent work on explanation evalu-
the following hypothesis: Con- trastive phrasal highlights improve the annotation of fine tree-grained semantic divergences in terms of accuracy and agreement.
"Since LIME and SHAP are feature attribution methods that assign a continuous score to each word, we checked that chang- ing the threshold from the default = does not improve the results, as hoth methods suffer from a precision-recall tradcoff.

---

Without Highlights
With Highlights
Without Highlights
With Highlights
100
100
Group
Majority
Group
Majority
90
90
90
86
86
87
83
82
84
80
80
80
80
80
78
76
78
7.
7.3
74
70
68
70
67
70
60
58
Re
Pr
F1
Re
Pr
F1
Re
Pr
F1
Re
Pr
F1
(a) English- French
(b) EnnliihhSpanish
Figure 3: Annotation- Lunotation-grounded evaluation comparing with- vs. without-higg highhights annotation conditions. Con-
trastive phrasal highlights significantly (p 0.1) improve the annotation of fine-grained semantic divergences.
5.1 Experimental Setup
lations and 10 syntactically divergent paraphrases) and 15 fine-grained divergences resembling trans- lation processes.
Explanandum We seek to explain the predic- tions of Divergent MBERT. We train separate mod- els for English-Frenn Freschh and English-Spanine follow- ing the approach detailed in Appendix A.
5.2 Study Design
We ran controlled evaluation across the two lan- guage pairs (English-Freen and English-Spanish). We describe the task and study design below.
Study Data To provide controlled yet realistic test samples, we construct a dataset of fine-grained divergences that mimic translation processes used by human translators (Zhai al., 2019). We start with English-French and English-Spanist human translations from the multi-paralles FLORES benchmark (Goyal al., 2022). We randomly select 50 seed translations among those identified as semantically equivalent pairs by Divergent MBERT. Then, we introduce fine-grained meaning differences by editing the English references of 10 of those samples. We perform edits motivated by translation processes, such modulation, explicitation, and reduction (Zhai et al., 2019), by interaction with ChatGPT.3 For instance, introduce a generalization as follows:
Task Description An annotation instance is sentence pair in English and, e.g.. French. Bilin- gual speakers are asked read each sentence pair closely, determine whether the "sentence pair contains any difference in meaning", ", and fi- nally characterize the difference as "Added" vs. "Changed", "Addedd" refers to added information, while "Changed" refers to information that is present in both languages but does not precisely match-mirrorring prior annotation divergence pro- tocols (Briakou and Carpuat, 2020). We include screenshot of the task description presented annotators Figures 5 and 7 of Appendix A.4.
Conditions We study two conditions: shown highlights
Despite leaving the show in 1993 he kept [the title of executive producer] + [a senior role] and continued to receive tens of millions of dollars every season in royalties.
in which participants (v /HGHHLIHHTS ) and a second which they are not ( X HIGHLIGHTs/ )). The only information avail- able to participants about the highlights is that they "are Al-generated and indicative of meaning differ- ences". We emphasize that highlights are not pre- sented explanations of divergence predictions, as we want annotators to choose how and whether they use them their assessments based on their intuitions.
To create difficult examples of equivalent transla- tions, we also paraphrase the English references of 10 seed examples with ChatGPT introduce syn- tactic changes. We validated all examples, ensuring that edits actually match the intended divergences in the first case (Figures 6 and 7) and that syntactic changes do not alter meaning in the second (Fig- ure 8). The final dataset consists of 35 semantically equivalent translations (split into 25 original trans-
Procedures We conduct between-subjects study where participants are randomly assigned to a condition. Participants are first presented with a tutorial that explains the task and relevant ter-
Details of the ChatGPT prompts are in Appendix A.3.

---

minology. Each participant is presented with 25 instances from one of the two studied conditions. Each batch of annotated instances contained 30% of semantically divergent edits, 20% of syntacti- cally divergent edits, and 50% original semanti- cally equivalent pairs. Instances within each batch are randomized. We include two attention checks where participants are asked to indicate their an- swers to the previous question. After completing the task, participants were asked complete brief survey assessing their perception of the gen- erated highlights (if assigned to this condition) and finally were asked to provide their free-form feed- back on the study along with demographic infor- mation, such as gender and age. The average time of the study was 20 minutes. In sum, we collect 3 annotations per instance and annotate a total of 100 instances (50 per condition) for each of the two language pairs studied.
5.4 Study Results
Reliability of Annotations shown in Figure 3, the bilingual group that annotated fine-grained meaning differences in the presence of contrastive explanations / HIGHLIGHTS, is significantly (p 0.1) more accurate across Precision, Re- call, and Fl scores and both language pairs, com- pared to X HIGHLIGHTS Furthermore, those im- provements carry over when aggregating annota- tion results by majority voting annotations across instances. As a result, this leads us to accept the hypothesis that contrastive phrasal highlights im- prove the annotation of fine-grained divergences by bilingual speakers. Finally, detecting divergences with highlights additionally improves the reliabil- ity of annotations are measured by Cohen's Kappa inter-annotator agreement statisties:
EN-FR
EN-ES
HIGHLIGHHS
0.51 (moderate)
0.33 (fair)
Participants We recruit 12 participants per lan- guage pair using Prolific." Each participant is re- stricted to taking the study only once. None of them failed both attention checks; hence we did not exclude any of the collected annotations from our analysis. All participants identified as bilingual speakers in the languages involved in each study. Participants are compensated at 15 USD per hour.
/ HIGHLIGHTS
0.66 (substantial)
0.52 (moderate)
Subjective Measures & User Feedback Over- all, bilingual speakers presented with contrastive phrasal highlights agreed they were useful in help- ing them spot fine-grained divergences- avverag self-reported usefulness of 3.8 for EN-FR and 1.2 for EN-ES. Finally, although contrastive phrasal highlights were useful, annotators also note that they cannot entirely rely on them. For instance, some of the participant's 's feedback is "The high- lights are useful but not 100% reliable, that is be- cause found other added/changed words that Al did not highlighi" and "In most cases, the words highlighted by the Al have helped to detect possible differences".
5.3 Measures
Our main evaluation measures are Precision, Re- call, and Fl computed by assuming that semanti- cally edited instances correspond to divergences, while the rest are treated as semantically equivalent pairs. We report those accuracy statistics both at the group level and also by majority voting anno- tations. Furthermore, we summarize the responses provided as free-form feedback and report subjec- tive measures that reflect participants' perceived understanding of the explanations, i.e., "the high- lights are useful in detecting meaning differences", if provided. The latter measures are collected on a 5-point Likert scale. We report statistical signif- icance based on bootstrap resampling, We draw samples randomly from the collected annotations with replacement. The sample size is the same as the one of the collected annotations, while the num- ber of resamples is set to 1,000. For the application- grounded evaluation I, which contains total of 150 annotations per language, the significance level is p= 0.1.
Application Evaluation II: Critical Error Detection In this section, we examine the potential of using contrastive phrasal highlights to assist bilingual hu- mans in detecting critical errors in machine trans- lation outputs. Recent work on quality estimation for machine translation proposes an error-based evaluation framework where bilingual professional annotators are asked to: highlight translation errors; rate their severity as major minor; and finally, indicate their type (c.g., mistranslation vs. termi- nology, etc.) (Freitag et al., 2021a). Drawing on those intuitions, we aim to study whether a sim- plified evaluation framework that uses contrastive

---

phrasal highlights yields reliable annotations of critical errors with bilingual crowd workers. This is a hard task as these errors are rare in high-quality systems and might be missed when parsing transla- tions quickly. In what follows, we test the hypothe- sis: Contrastive phrasal highlights help bilingual crowd-workers detect critical (accuracy) errors in machine translation outputs.
Without Highlights
With Highlights
100
Negation
WMT
9091
All
90
R6
82
78
76
72
70
65
63
68
65
67
60
58
50
Re
Pr
F1
Re
Pr
F1
Re
Pr
F1
6.1 Experimental Setup
Figure 4:
evaluation results com-
Explanandum We explain the predictions of Di- vergent MBERT trained for English-Porttguese fol- lowing the process detail in Appendix A.I. We emphasize that the explanandum is sensitive de- tecting any meaning differences and not only criti- cal errors, and therefore, we expect the contrastive phrasal highlights to surface both minor and major accuracy errors.
paring X HIGHLIGHTS vs. HIGHLIGHTS annota- tion conditions. Contrastive phrasal highlights signifi- cantly improve the Recall and Fl scores when detecting negation errors (p 0.05) and errors in the ALL set. Precision improvements across sets and any improve- ments on the WMT set are not significant (p > 0.05).
as being minor or major. Following Freitag et al. a)) minor errors are defined as errors that do not lead to loss of meaning and would not con- fuse mislead the reader, while major ones may confuse or mislead the reader due significant changes in meaning. We ask participants to factor out fluency issues in their assessment as much as possible. We include a screenshot of the task de- scription presented to annotators in Figures and 10 of Appendix A.4.
Study Data For the purposes of the study, use the synthetic Portuguese-Englihh dataset from the Critical Error Detection task of WMT (Freitag et al., 2021b). The dataset consists of pairs sampled from the OPUS parallel corpus (Tiedemann, 2012) treated as "not containing critical error". Then, the WMT organizers artificially corrupted 10% of those pairs to introduce critical errors that reflect hallucinated content, untranslated content, or mistranslated texts. We will refer to those errors as WMT for the rest of the paper. Additionally, make the task hard for humans, we a) filter out WMT critical errors that concern deviation in numbers, time, units, or dates and b) artificially introduce negation errors: we edit the English text locally by changing one or two words in such way that the meaning of the sentence is entirely flipped (e.g., the word benefits in English reference is replaced by the word harms). In total, the study dataset consists of 30 translations that do not contain a critical error (also detected as equivalent under the explanandum), 10 translations reflecting WMT errors, and 10 transla- tions reflecting local negation errors.
Conditions & Evaluation Constructs We use Precision, Recall, and F1 our main evaluation measures and follow the same conditions and sub- jective measures described in $.5. The above scores are calculated only against major errors, as those are the ones for which we have gold labels. We report statistical significance based on bootstrap re- sampling as outlined in $5.3. For this study, which contains a total of 250 annotations, the significance level is p 0.05.
Procedures Participants We collect assess- ments per instance and annotate a total of 100 in- stances (50 per condition). Each participant is pre- sented with 25 instances, among which 20% corre- spond to WMT critical errors, 20% to negation er- rors, and the rest to original OPUS parallel texts. We recruit a total of 40 participants, all of whom self- identified proficient in English and Portuguese. All other participant recruitment details and study procedures are the same as in $5.
6.2 Study Design
We test our hypothesis with a user study of criti- cal eтror detection in English translations of Por- tuguese texts, as described below.
Task Description An annotation instance is an excerpt in Portuguese and its (machine) translation in English. Bilingual speakers are asked to read the two excerpts, determine whether the translation contains an accuracy error, and then rate its severity
6.3 Study Results
Main Findings As shown in Figure 4, the user group presented with contrastive phrasal highlights

---

Narar
MIN
closer look at the users' feedback sheds some light on our approach's 's current strengths and limitations. Although highlights were in principle useful and a feature to have especially when proofreading a great amount of translated texts", some annotators spotted "a high percentage of false positives" " (see Figures 10i and 10j in Appendix A.4) and raised concerns that "relying solely them might make them less aware of errors hidden in non- non-highlighted texts" (see Figure 10b in Appendix A.4).
0
B
O
O
F
D
O
D
9
F
O
B
B
O
D
O
7 Conclusion
R
O
O
B
We introduce an approach to extracting contrastive phrasal highlights that explain NLP models that take as input two texts. Unlike existing techniques that highlight input tokens independently, we con- sider the input's 's structure and explicitly model the relationships between contrasting inputs.
B
F
Table 3: Critical Error Detection user study results. Each row represents an annotator, and each cell is an an- notation, where & indicates no accuracy error detected, indicates minor error, and a critical error.
We study the effectiveness of contrastive phrasal highlights by explaining the predictions of diver- gence ranker that compares and contrasts meaning differences in bilingual texts. Our proxy evalua- tion confirms our approach outperforms standard explainability approaches, that highlight tokens, by matching human-provided rationales of diver- gences in English-French Wikipedia texts. Finally, through series of suman-centered evaluations, we explore the usefulness of contrastive phrasal high- lights in opplication-grounde contexts for three language pairs. Our results suggest that contrastive phrasal highlights assist bilingual speakers in de- tecting fine-grained meaning differences in (hu- man) translated texts and critical errors due to local mistranslations in -trasil--t texts.
exhibits a higher recall in detecting critical er- rors compared to the one that does not access them. Crucially, we report significant improve- ments (p = 0.05) in detecting negation errors, which are harder to spot. A closer look at the dif- ferences between the two error classes reveals that detecting negation errors is more challenging as annotators have to pay close attention to the com- pared texts and the relationships between them to spot local mistranslations that cause a major shift in meaning. On the other hand, WMT errors are more easily spotted as they mostly represent detached hallucination phenomena (i.e., an entire phrase is added to the English translation), which we hypoth- esize can be implied by length differences of the compared texts. Our full annotation results, pre- sented in Table 3, further validate our hypothesis: WMT errors are not genuinely missed out but rather rated as minor by some annotators. On the other hand, usually, negation errors are not only misper- ceived as minor but are entirely overlooked. We include examples of annotations in Figure 10.
Our findings create opportunities for designing better machine-in-the-logg pipelines identify crit- ical machine translation errors grounded in high- stake settings, study translation data at scale, and fa- cilitate the creation of multilingual content through crowd-based efforts.
Acknowledgements
We thank Sweta Agrawal, Hal Daumé III, Luke Zettlemoyer, Philip Resnik, Leo Zhicheng Liu, the anonymous reviewers, and the CLIIP lab at UMD for helpful comments. This work was funded in part by the U.S. Army Grant No. 1 and by the National Science Foundation under Awards No. 1750695 and 2147292. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors.
Subjective Measures & User Feedback Over- all, bilingual speakers found the highlights help- ful, with an average self-reported usefulness of 3.9. Additionally, they reported they would like to use them as tool to assist them in detecting critical errors, with an average score of 3.8. We include de- tails on the distribution of ratings in Table 4. Last, a

---

Limitations
References
Our work contributed evidence that contrastive phrasal highlights can provide a framework that assists humans in detecting meaning differences of a specific nature (i.e., fine-grained translation processes or local critical errors). However, de- tecting translation differences in the wild requires covering the entire distribution of meaning differ- ences (Freitag et al., 2021a), which we leave for future work.
Eneko Agire, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agiree 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference Lexical and Compu- tational Semantics Volume Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semuntic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
Moreover, although our method builds on un- supervised modules (i.e., Divergent MBERT and SimAlign) that do not rely on supervised data and can, in principle, be applied to any language pair, we have only evaluated our approach in high- resource settings where we expect both the ex- planandum and the word alignment models to be reasonably accurate. Therefore, further work should be conducted to explore how our findings generalize to other settings, such as in low-resource regimes where we might expect the alignment to be of poorer quality, introducing errors that may impact humans' perceived understanding of the contrastive highlights differently.
Leila Arras, F. Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. "what is rele- vant in text document?": An interpretable machine learning approach. PLoS ONE, 12.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Dis- course, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Ondiej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Pro- ceedings of the Second Conference on Machine Trans- lation, pages 489-513, Copenhagen, Denmark. Asso- ciation for Computational Linguistics.
Additionally, future work should explore how the alignment module should be operationalized for other NLP tasks that take as input two texts, po- tentially exploring monolingual aligners (Lan et al., 2021) or additional structured information such as abstract meaning representations (Banarescu et al., 2013).
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Finally, the significance of our results could be strengthened by increasing the sample sizes of fu- ture user studies. We view our current findings as solid starting points for an in-depth exploration of the usefulness of highlights for human-centered applications that future studies can build upon and extend with broad explorations.
Jordan Boyd-Graber Samuel Carton, Shi Feng. Q. Vera Liao, Tania Lombrozo, Alison Smith-Renner, and Chenhao Tan. 2022. Human-centeree evaluation of explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies: Tutorial Abstracts, pages 26-32, Seattle, United States. Association for Computational Lin- guistics.
Ethics Statement
Both studies are approved by the University of Maryland Institutional Review Board (IRB num- ber 2018158 - 1). As discussed in the paper, all participants involved in our studies gave their con- sent and were compensated at rate of 15 USD per hour. We collected minimal demographic informa- tion, and participants could opt out of answering the demographic questions. We did not collect any further potentially identifiable information. Partici- pation in our study was completely voluntary, and participants had the right to withdraw at any time.
Eleftheria Briakou and Marine Carpuat. 2020. De- tecting Fine-Grained Cross-Lingual Semantic Diver- genees without Supervision by Learning to Rank. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1563-1580, Online, Association for Computa- tional Linguistics.
Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena I. Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. Proceedings of the 25th International Conference on Intelligent User Interfaces.

---

Saneem Chemmengath, Amar Prakash Azad, Ronny Luss, and Amit Dhurandhar. 2022. Let the CAT out of the bag: Contrastive attributed explanations for text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7190-7206, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
using BLEU neural metrics are better and more robust. In Proceedings of the Seventh Conference Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondiej Bojar. 202 Results of the WMT21 metries shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceed- ings of the Sixth Conference Machine Translation, pages 733-774, Online. Association for Computa- tional Linguisties.
Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang. Haiqing Chen, and Yin Zhang. 2021. KACE: Generating knowledge aware contrastive ex- planations for natural language inference. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguisties and the 1 1ith Inter- national Joint Conference on Natural Language Pro- cessing (Volume I: Long Papers), pages 2516-2527, Online. Association for Computational Linguistics.
Naman Goyal, Cynthia Gao. Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- evaluation benchmark for Low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
E. Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua A. Kulas, Andy Schuetz, and Walter F. Stew- art. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mecha- nism. In NIPS
Peter Hase and Mohit Bansal. 2020. Evaluating explain- able AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties. pages 5540-5552, Online, Association for Computational Linguisties.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Denis Hilton. 2017. Social attribution and explana- tion. Oxford Handbook of Causal Reasoning, page 645-676.
Alon Jacovi and Yoav Goldberg. 2021. Aligning faithful interpretations with their social attribution. Transac- tions of the Association for Computational Linguis- tics, 9:294-310.
Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Fiic Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online. Association for Computational Linguistics.
Sarthak Jain and Byron C. Wallace. 2019, Attention is not explanation. In North American Chapter of the Association for Computational Linguistics.
Finale Doshi-Velez and Been Kim. 2017. Towards a
Masoud Jalili Sabet, Philipp Dufter, François Yvon. and Hinrich Schütze. 2020. SimAlign: High qual- ity word alignments without parallel training data using static and contextualized embeddings. In Find- ings of the Association for Compudational Linguistics: EMNLP 2020, pages 1627-1643, Online. Association for Computational Linguistics.
rigorous science of interpretable machine learning. arXiv preprint / 702.08608.
Shi Feng, Fric Wallace, Alvin Grissom II, Mohit Tyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719 -378., Brussels, Belgium. Association for Computational Linguistics.
Yuncheng Jiang, Wen Bai, Xiaopei Zhang, and Jiao- jiao Hu. 2017. Vikipedia-basee information content and semantic similarity computation. Information Processing & Management, 53(1):448222
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- tics, 9: 160-1474
Siwon Kim, Jihun Yi, Funji Kim, and Sungroh Yoon. 2020. Interpretation of NLP models through input marginalization. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (FMNLP), pages 3154-3167, Online. As- sociation for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Ballison-Burch, Marcello Federico, Nicola Bertoldi Brooke Cowan, Wade Shen, Christine Moran,

---

Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Anmual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
metries shared task. In Proceedings the Fifth Con- ference on Machine Translation, pages 688-725, On- line. Association for Computational Linguistics.
Tim Miller. 2019, Explanation in artificial intelligence: Insights from the social sc iences. Artificial Intelli- gence, 267:1-38.
Mare Miquel-Ribe Ribbé and David Laniado. 2018. Wikipedia culture gap: Quantifying content imbalances across 40 language editions. Frontiers in Physics, 6.
Wuwei Lan, Chao Jiang, and Wei Xu. 2021. Neural semi-Marke Mani-Markov CRF for monolingual word alignment. In Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 1 Ithh International Joint Conference on Natural Lan- guage Processing (Volume I: Long Pupers), pages 6815-6828, Online. Association for Computational Linguisties.
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. 2019. Interrogating the explanatory power atten- tion in neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Trans- lation, pages 221 1-20,, Hong Kong. Association for Computational Linguistics.
Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Threekkdd. 2020. petristicalll-iffrmeee
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. 2021. Measuring and improving faithfulness of at- tention in neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2791-2802, Online, Association for Computational Linguisties.
transformations (LIT): A method for automatically generating contrast sets. In Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 126-135, Online. Association for Com- putational Linguisties.
W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decompo- sition to extract interactions from Istms. Cite 18xiv:1801 .05453Comment: Under review at ICLR 2018.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Un- derstanding neural networks through representation erasure. ArXiv, 1b1.0002.
Qingzi Vera Liao, Dan Gruen, and Sarah Miller. 2020. Questioning the ai: Informing design practices for explainable ai user experiences. Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems.
Dong Nguyen. 2018. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1069-1078, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Peter Lipton. 1990. Contrastive explanation. Royal Institute of Philosophy Supplements, 27:247-266.
Scott M Lundberg and Su-In Lee. 2017. unified approach to interpreting model predictions. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical ma- chine translation. In 1999 Joint SIGDAT Conference on Empirical Methods Natural Language Process- ing and Very Large Corpora.
Qingsong Ma. Ondfej Bojar, and Yvette Graham. 2018. Results of the WMT18 metries shared task: Both characters and embeddings achieve good perfor- mance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels. Association for Com- putational Linguistics.
Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi. and Luke Zettlemoyer. 2021. Prompting contrastive explana- tions for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4179-4192, Online. Association for Computational Linguisties.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019, Results of the WMT19 metrics shared task: Segment-level and strong MT sys- tems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation ( (Volume 2: Shared Task Papers, Day 1), pages 62-90, Flo- rence, Italy. Association for Computational Linguis- tics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondiej Bojar. 2020. Results of the WMT20

---

Alves, Luisa Coheur, Alon Lavie, and André F. Martins. 2022. CometKiwi: IST iST-ubbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Marcos Treviso, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2021. IST-unbabel 2021 sub- mission for the explainable quality estimation shared task. In Proceedings of the 2nd Workshop on Evalu- ation and Comparison of NLP Systems, pages 133- 145, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 11, page 1135-1144, New York, NY, USA. Association for Computing Machinery.
Martin Tutek and Jan Snajder. 2020. Staying true to your word: (how) can attention become explanation? In Proceedings of the 5th Workshop Represen- ration Learning for NLP, pages 131-142, Online. Association for Computational Linguisties.
Alexis Ross, Ana Marasović, and Matthew Peters. 2021. Explaining NLP models via minimal contrastive edit- ing (MiCE). In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 3840-3852, Online. Association for Computational Linguisties.
Yogarshi Vyas, Xing Niu, and Marine Carpuat. 2018. Identifying semantic divergences in parallel text with- out annotations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computaaionaal Linguistics: Human Lan- guage Technologies, Volume I (Long Papprrs), pages 1005,1515,M0 New Orleans, Louisiana. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. Smooth- grad: removing noise by adding noise. ArXiv, abs/1700..0885.
Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh. 2020. Gradient-based analysis of NI.P mod- els is manipulable. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 247-258, Online, Association for Computational Lin- guistics.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Er- ick Fonseca. Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the WMT 2020 shared task on quality estimation. In Proceed- ings of the Fifth Conference on Machine Translation, pages 743-764, Online. Association for Computa- tional Linguistics.
Shira Wein, Lucia Donatelli. Ethan Ricker, Calvin En- gstrom, Alex Nelson, Leonie Harter, and Nathan Schneider. 2022. Spanish Abstract Mcaning Repre- sentation: Annotation of a general corpus. In North- ern European Journal of I Language Technology, Vol- ume 8, Copenhagen, Denmark. Northern European Association of Language Technology.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishray Chaudhary, and André F. T. Martins. 2021. Findings of the WMT 2021 shared task on quality estimation. In Proceed- ings of the Sixth Conference on Machine Translation, pages 684-725, Online. Association for Computa- tional Linguisties.
Shira Wein and Nathan Schneider, 2021. Classifying divergences in cross-lingual AMR pairs. In Proceed- ings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 56-65, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón F. Astudillo, and André F. T. Martins. 2018. Findings of the WMT 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers. pages 689-709, Belgium, Brussels. Association for Com- putational Linguistics.
Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume I. Curran.
Harini Suresh, Steven R. Gomez, Kevin K. Nam, and Arvind Satyanarayan. 2021. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the 202/ CHI Conference Human Factors Computing Systems, CHI '21 New York, NY, USA. Association for Computing Machinery.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) pages 11 11-20, Hong Kong. China. Association for Computational Linguistics.

---

Kayo Yin and Graham Neubig. 2022. Interpreting lan- guage models with contrastive explanations. Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 184-198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
A.3 ChatGPT Details
Starting from a given English sentence, we asked ChatGPT to edit the sentence in way that intro- duces a small meaning difference. Below, we in- clude the different prompts we used:
Yumingg Zhai, Gabriel Mouuz and Anne Vilnat. 2020. Detecting non-literal translations by fine-tuning cross-lingual pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5944-5956, Barcelona, Spain (Online). International Committee on Computational Linguistics.
"Can you edit small phrase in sentence X to introduce small meaning difference?"
"Can you delete/add phrase in sentence X introduce small difference in meaning?
"Can you edit small phrase that makes sen- tence X more general/explicit."
Yuming Zhai, Aurélien Max, and Anne Vilnat. 2018. Construction of multilingual corpus annotated with translation relations. In Proceedings of the First Workshop Linguistic Resources for Natural Lan- guage Processing, pages 102-111, Santa Fe, New Mexico, USA. Association for Computational Lin- guistics.
We then reviewed the edits introduced by ChatGPTto make sure it indeed introduced small meaning differences. we are not satisfied by a current edit, we ask again and potentially specify what parts of the sentence we would like to be edited.
Yuming Zhai, Pooyan Safari, Gabriel Illouz, A. Al- lauzen, and Anne Vilnat. 2019, Towards recognizing phrase translation processes: Experiments on english- french. POLIBITS, 60:27-33.
A.4 User Study Interfaces
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.
Below we include screenshots of our user study interfaces along with annotation examples for each of our evaluation tasks.
A Appendices
A.1 Explanandum Training Details
The divergence ranking model is trained using the public implementation of Divergent MBERT (Bri- akou and Carpuat, 2020). Synthetic divergences are generated starting from the 5,000 top-scoring WikiMatrix sentences based on LASER score (i.e.. seed equivalents). We fine-tune the "BERT-Base Multilingual Cased" model (Devlin et al., 2019) and set the margin equal to as per the original implementation.
A.2 Subjective Measures
Table Bilingual speakers' agreement ratings with the statement "Highlights were useful in helping detect meaning differences in the compared texts" (left) and "I would like to use highlights help me detect critical emors" (right).

---

Instructions collspse)
In this study, you will be presented with 25 exceerps in Frggish and their respective (humany transiations in French. incicate whether not the excerpts contain some meaning diferencess using system predicts spans indicative meaninng
The goal this
that best describes type
meaning differonces (when they okett). will
highlighted excerpts. highiihted excerpts across the two sentencees
When do two excerpts contain some meaning differences? The excerpts contain some meaning differences, when they convey mostly the same information, except
hoth sides; some Finglish have more general speetti translation in French)) You will be further asked to indicate which of the tollowing cases best describes the reason the excerpts contain
details nuances (6.g. soma information addad andior missing
same meaning difforences:
Added (in Freech): The French excerpt containe piece of information that does not exist the Englisn excerpt.
information Changed: A piece of information that exiets in both the French and the English excerpts does not have the exact same mearing.
English): English excerpt
Frenchh excerpt.
Figure 5: Instructions for application application-grouneed evaluation I: Annotation of Semantic Divergences.
(a)
(b)
pnayy
(c)
(d)
(h)
Figure 6: Annotations of fine-grained semantic divergences (English-Spanise on the left and English-Frend French on the righ) reflective of explicitation and reduction translation processes. Contrastive highlights subsume added content that present one language but missing from the other.

---

(a)
(b)
(c)
(d)
(c)
(h)
(m)
(n)
Figure 7: Annotations of fine- -reeggr semantic divergences Senglish-Spanise on the left and English- English-French on the right) reflective of modulation translation processes. Contrastive highlights frequently subsume content that does not convey the exact same meaning across languages.

---

(b)
(c)
(d)
(c)
(h)
Figure 8: Annotations of semantically equivalent pairs (English-Spaan Spanish on the left and English- Frenchh on the right) that reflect syntactic divergences. Contrastive highlights surface false positive segments.

---

Task Instructions (Click to collepe))
this survey, you will be presented with lexts in Portuguree and the respective tranelations in English generaled by A) system.
The goal of this task is to indicate whather or not transiation on represents aocurataly the mearing of itss source text and indicate the severity of the spotted enrorisi (.a., major minor), whan they exist.
meening of the compared texts (ie.. factor fluency your judgmeets).. You also be shown highighted excerpps. The highlighted hed excerpts are extracted using an Al system that predicts spans indiowitve meening diffrencee across the two sentences.
You should try make decision based
What are the error severity levels?
A. Minor: : Emars that don't lead to loss of meaning and would not contuse or misicad the reader but would be noticed, wou decrease stylistic qualit, clarity, would make content less appealing.
Major: Emora that may contuse or mialead the raader duae significant change in meaning.
Figure 9: Instructions for application evaluation II: Critical Error Detection.
(a) Major error (negation)
(b) Major error (negation)
(c) Major error (negation)
(d) Major error (hallucination)
(c) Major error (hallucination)
(i) Major error (hallucination)
(g) Minor error
(h) Minor error
ii No error
(i) No error
Figure 10: Annotations of accuracy errors in Portuguese translations of English texts. Contrastive highlights frequently surface meaning differences across the compared texts.


Reviews:
[{"review_id": "OkvpI81ElM", "reviewer": "Reviewer_uUif", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback.\n\n&#8594; **Reproducibility concerns:**\nUpon publication, we will release any code, data artifacts, and interfaces produced in this paper.\n\n&#8594; **Sample size of application-grounded evaluations:** \nGiven that prior work evaluates highlight-based explanations of (machine) translation errors based *solely on automatic evaluations*, we chose to initiate a human-centered discussion that evaluates the usefulness of such highlights directly with an in-depth exploration of multiple application-grounded evaluations. We chose a sample size of 50 instances for each of our three studies, following the experimental settings of prior NLP work on human-centered evaluation of explanations (for other tasks), which evaluate 20-100 instances (Schuff et al., EMNLP 2020) (González et al., Findings 2021)  (Xie et al., Findings 2022). \n\nWe decided to split our budget, i.e., $500 in total, across different conditions to study how highlights can assist humans in detecting meaning differences in both human and machine translations across three language pairs. Besides the small scale of our studies, we report significant improvements and consistent patterns that indicate our contrastive phrasal highlights assist humans in detecting meaning differences across all three conditions studied. \n\nFinally, we agree with the reviewer that increasing the sample sizes of our studies would strengthen the significance of our results. We view our current findings as solid starting points for an in-depth exploration of the usefulness of highlights for human-centered applications that future studies can build upon and also extend with broad explorations.\n\n&#8594; **Possible terminology bias:** \nWe agree with the reviewer that the wording around highlights presented to the users—in our case, “highlights are AI-generated and indicative of meaning differences”—could potentially bias the users’ responses. This is, however, true of any possible wording, and therefore, answering the question requires a study that explores alternative prompts and a per-user analysis. \n\nAlthough those are not questions we have asked in this paper, we can reasonably draw some inferences by looking at users’ agreement with correct AI and false AI predictions in our application-grounded evaluation I. As detailed in the paper, for this evaluation, we deliberately introduce semantically equivalent paraphrases that result in phrasal highlights that do not reflect meaning differences. Such cases allow us to draw some conclusions on whether bilingual works over-rely on highlights, i.e.,  do they blindly tag everything highlighted as having a meaning difference? Below are some additional statistics that suggest there are no (at least major) biases introduced by our terminology: \n\n* Agreement with correct AI: 90% (French) and 87% (Spanish)\n* Agreement with false AI:     33% (French) and 48% (Spanish)\n\n&#8594; **Response to stated hypotheses:**\nWe accept both stated hypotheses as described in lines 439-440 and 534-536 for application-grounded evaluations I and II, respectively. \n\n&#8594; **Connection between different application-grounded evaluations:**\nBoth of the application-grounded evaluations revolve around the annotation of meaning differences in translation pairs and show how the *same phrasal highlights* can assist humans in detecting both nuanced differences in human translation and critical errors in machine translations. \n\nOur first study establishes the correctness of our contrastive phrasal highlights more directly: bilingual speakers are asked to identify *any* small meaning differences in human translations. Our second study, building on the first one, investigates a broader scenario: bilingual speakers are asked to not just identify meaning differences in (machine) translations but also mark the ones that correspond to critical translation errors. The latter study reflects a broader application as our highlights are not tailored to a specific severity; therefore, both minor and major meaning differences can be highlighted.\n\nOur findings suggest that the usefulness of our phrasal contrastive highlights can be generalized to different use cases, as they help humans tailor their annotations to errors of different severity. This promising finding opens the room for broader explorations of how the *same highlights* can help humans detect different types of errors that might be specific to a language pair, a domain, or an application based on the use case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-grounded and includes a thorough overview of highlight and contrastive explanations, the evaluation of such explanations, and prior work performed on explaining differences in translated texts.\n\nThe contrastive phrasal highlighting also presents a novel contribution, also in the way that it prioritizes brevity. This process is well-explained and illustrated.\n\nThe design of the human-centered studies is transparent, and the results are evaluated for significance. These results include a qualitative analysis, examples of which are provided in the paper.\n\nRegarding the studies, the main findings of each study is discussed, and supporting arguments from free-form feedback is provided. ", "reasons_to_reject": "The main weakness of the paper originates from the data used in the application-grounded evaluations, its small-scale, and the sample size of the study. The first study contains 60 translation pairs, while the second includes 50. Regarding sample size, the first study includes 24 (12 per language pair) participants, weakening the generalizability of the reported statistical significance.\n\nIn addition, the prompts possibly could bias user responses, such as that the highlights “are AI-generated and indicative of meaning differences”. Such terminology could induce bias based on the user’s opinion on AI-generated content.\n\nFollowing the presentation of the results, the discussion does not extend beyond the presentation of the results, making it unclear what conclusions the authors draw from the interpretation of their results, in response to the stated hypotheses. A connection between the results of the separate studies would strengthen the analysis – for example, how can the results of second study, positing that bilingual crowd-workers are assisted by contrastive phrasal highlighted in detecting some errors, be connected to the results that the same highlighting assists in detecting semantic differences?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper “Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences” focuses on the task of improving explainability in the task of detecting errors and mistranslations, particularly from a human-centered perspective. Centering around the importance of phrasal pairs rather than tokens, the authors introduce a new method to produce highlights to explain differences between pairs of translated text. The authors validate this method against existing methods through a series of designed evaluations, which suggest that the introduced method strengthens the ability for humans to detect fine-grained differences in translated texts.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 102: seems like a word is missing here\n\nLine 105: typo – “has seeks”\n"}, "extra_text": {}}, {"review_id": "GAdMoOoLcw", "reviewer": "Reviewer_w8E1", "summary": "", "questions": "A. For the study in section 6, what exactly were the annotation possibilities per sentence? No error, minor error, major error? It seems from the text that you focused on major errors. It's thus unclear how exactly you define precision/recall/F1 for Figure 4. Please clarify this.\n\nB. Please provide full details on what statistical tests you used in sections 5 and 6. It also seems that you didn't do any significance testing for section 4; are you able to provide this?\n\nC. Figure 3 (and on line 433) you claim statistical significance but say p=0.1. Given the magnitude of the difference, I assume that this is a typo (and you meant, e.g., p=0.01), as you later  in Figure 4 say that a value of p>0.05 is *not* significant.\n\nD. Lines 215–216: it's not clear what is meant by \"unaligned words... can be included in either phrase\". Does this mean that, if unaligned words are adjacent to an identified phrase, they are added to the phrase? What do \"can\" and \"either\" mean here?\n\nE. Line 238: what is |S|? Is it |S| = max(|S_1|, |S_2|) where S = (S_1, S_2)?\n\nF. Line 238: why is it meaningful to look for R(...) >= 0 if R is a ranker trained for pairwise ranking (as indicated on lines 190–191)? Should this perhaps be R(DEL[S;p]) - R(S) >= 0?\n\nG. Line 238: the answer to the previous question might help resolve this, but as-is I'm not clear on why you increase the score in the 'otherwise' case.\n\nH. Table 2: \"Del.\" isn't explained. Judging from the text just below it this is the percentage of the words in the sentence that are highlighted. Is that correct?\n\nI. Line 293: the random baseline isn't defined. Assuming I'm correct about \"Del.\", it looks like this just randomly highlights tokens with 50% probability. Is that correct?\n\nJ. Why no F1 scores in Table 2? (Assuming I'm correct about it, )\"Del.\" is IMO less useful, and the brief discussion of it in the text just below Table 2 suffices.\n\nK. Do you have data on time-to-completion for the tasks that your participants performed? That's another measure that could help support the hypothesis of the utility of contrastive phrasal highlights. For example, participants in the with-highlights condition might have been able to complete the task faster than those in the without-highlights condition, on top of the improvements in F1.\n\nL. Line 547: you hypothesize that the WMT errors are more easily spotted. This seems intuitively plausible, but it seems that your data in Figure 4 contradict this: the without-highlights precision, recall, and F1 are very similar in the WMT and Negation cases. Is that assessment accurate?\n\nM. Footnote 7: what do you mean by \"corroborate\" here?\n\nN. Since you have 12 participants, please specify how you handle ties (if there are any) in computing the majority stats for Figure 3.\n\nO. In section 5.3, it seems that you have multiple measures that reflect participants' \"perceived understanding of the explanation\", yet you give only one example. Were there others? How did the participants respond?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and the many useful suggestions and corrections. We will incorporate all the suggestions in our revised version.\n\nWe start by addressing the primary concerns about reproducibility raised by the reviewer and then proceed to address their questions.\n\n&#8594; **Reproducibility concerns:** \nUpon publication, we will release any code, data artifacts, and interfaces produced in this paper. Additionally, we will expand on the prompts used for the ChatGPT experiments (detailed below). \n\n&#8594; **ChatGPT details:**\nStarting from a given English sentence, we asked ChatGPT to edit the sentence in a way that introduces a small meaning difference. Below, we include the different prompts we used:\n* “Can you edit a small phrase in sentence X to introduce a small meaning difference?”\n* “Can you delete/add a phrase in sentence X to introduce a small difference in meaning?”\n* “Can you edit a small phrase that makes sentence X more general/explicit?”\n\nWe then reviewed the edits introduced by ChatGPT to make sure it indeed introduced small meaning differences. If we are not satisfied by a current edit we ask again and potentially specify what parts of the sentence we would like to be edited.\n\n&#8594; **Question A:** That’s correct. As shown in Figure 10, bilingual speakers are asked to first indicate whether the translation contains an adequacy error and then specify its severity, i.e., major or minor. The severity is, by default, set to N/A in case the translation contains no adequacy error. Precision/recall/F1 scores are calculated only against major errors, as those are the ones we have gold labels for. That said, if a translation contains a major error (per our gold labels) but is annotated as minor or as N/A by humans, it counts as a false negative. \n\n&#8594; **Question B:** We report statistical significance based on bootstrap resampling. We draw samples randomly from the collected annotations with replacement. The sample size is the same as the one of the collected annotations, while the number of resamples is set to 1000. For the first study (application-grounded evaluation I), which contains a total of 150 annotations per language, the significance level is p=0.1, and for the second study (application-grounded evaluation II), which contains a total of 250  annotations, the significance level is p=0.05.\n\nWe did not include significance tests for our proxy evaluations as the Recall improvements we report over prior work are huge, i.e., 37-39 points, with similar Precision. However, we can confirm that all Recall improvements are significant, with a p-value of 0.001. Precision improvements are not statistically significant as expected, given the small difference from prior work.\n\n&#8594; **Question C:** Given the smaller sample size of the application-grounded evaluation I, the calculated significance yields a p-value of 0.1, as detailed in our response to B.\n\n&#8594; **Question D:**  The sentence means that “unaligned words can be included in either the source ($p_1$) or the target ($p_2$) phrases”. We will revise the wording to address the confusion.\n\n&#8594; **Question E:** $|S| = |S_1| + |S_2|$. Thanks for pointing this out; we will add this definition to the revised version.\n\n&#8594; **Question F:** Our search is implemented in two steps as defined in the equations of Lines 230 and 237. At first, we are extracting a set of candidate counterfactual instances based on $\\tilde{P}$. This set consists of all phrases, the deletion of which increases the ranking difference $R(\\mathrm{DEL}[S;p]) - R(S) \\geq 0$, as you correctly outline in your question.\n\nThen, in the second step (Line 237), we search within $\\tilde{P}$ for the counterfactual that not only increases the score $R(\\mathrm{DEL}[S;p])$ but also does so in a way that deletes the shortest possible phrase. Therefore, Line 238, which you indicate in your question, does not really implement the search but instead defines the brevity reward that encourages the selection of short phrases when searching within $\\tilde{P}$. \n\nThis reward is operationalized as an exponential function ($e^{|x|}$)  used when ranking the candidate counterfactuals in $\\tilde{P}$. To achieve this, we switch the sign of the exponent in the brevity function in order to consistently reward shorter phrases when multiplying with $R(\\mathrm{DEL}[S;p])$ (which can have either positive or negative values).\n\n&#8594; **Question G:** See our response in F.\n\n&#8594; **Question H:** Yes, that’s correct. We will clarify this in our revisions.\n\n&#8594; **Question I:** Yes, that’s correct. We will clarify this in our revisions.\n\n&#8594; **Question J:** We will add F1 scores in our revised version. Our approach surpasses prior work based on F1, too. For reference, here are the scores for the English and French sides separated by dashes: \n\n* Random: 43 - 45  \n* LIME: 41 - 40\n* SHAP: 41 - 39\n* Ours:  62 - 64\n\n&#8594; **Question K:** Yes, we have collected time-to-completion data; however, there are no significant differences between the completion times of the different user groups. One reason we have not seen differences in that matter might be that the users who see highlights spend some time familiarizing themselves with what highlights represent. However, it would be interesting to explore whether those results change once users become familiar with the highlights or receive more training as to what they are, which we leave for future work.\n\n&#8594; **Question L:** Thanks for pointing this out! You are correct that this hypothesis is not supported by Figure 4, as those scores ignore altogether adequacy errors annotated as minor. Our hypothesis comes from our observation that bilingual speakers tend to spot WMT errors as adequacy errors but rate their severity as minor. This led us to conclude that they are not being genuinely missed out. On the other hand, Negation errors are not only misperceived as minor but are entirely overlooked. To clarify this, we will include the full annotation results (including minor and major errors) in the revised version.\n\n&#8594; **Question M:** We will rephrase it to “emphasize”.\n\n&#8594; **Question N:** Each instance is annotated by 3 participants. That way, majority voting has no ties (see lines 401-405). \n\n&#8594; **Question O:** For  Section 5.3, we only collect feedback based on the statement, “Highlights were useful in helping me detect meaning differences in the compared texts.\". For Section 5.4, we additionally ask: “I would like to use the highlights to help me detect critical errors\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and easy to follow. Seriously! I have technical questions, sure, but most papers I review aren't as well-organized or as cohesive and coherent as this paper. For example, the hypotheses for the two application-grounded studies are clearly stated up front to help clarify what is being explored in the relevant section.\n* The method presented is (mostly) easy to understand.\n* The method presented is conceivably applicable to arbitrary models (the authors don't mention this, and I think it'd be worth mentioning as a strength in the paper!), and the use of pre-existing phrasal alignment methods means that the authors' method will benefit from advancements in phrasal aligners.\n* The method presented improves on prior work and the human evaluations suggest that the contrastive phrasal highlights are legitimately useful.\n* The authors tested their method with multiple language pairs.", "reasons_to_reject": "Nothing I'm writing here is a reason to reject, really (though I have some questions in the \"Questions For The Authors\" section that may impact my sense of this). But there are some minor weaknesses:\n\n* The method presented relies on external phrasal alignment models (this is also a strength as noted above!)\n* Some details are unclear (see \"Questions For The Authors\")\n* While the authors argue strongly (and, I think, correctly) that phrasal highlights are more useful than simpler token-based highlights, the human evaluations do not directly test this. It's conceivable that, for example, annotators would find the simpler token-based highlights just as useful as the contrastive phrasal ones. While I think this is unlikely, it undermines the overall argument somewhat. To be clear, I'm not especially bothered by this as the very fact that the authors included application-grounded evaluations is a useful contribution (and we can make some reasonable inferences based on the results from the proxy evaluations).\n* There's no indication that the authors will release their code or the data presented to annotators for the studies in sections 5 and 6 (including annotations from the participants). I would highly encourage this, especially since it can help clear up details that might be unclear from the paper's main text (or even in appendices). Similarly, including full details of the ChatGPT prompts in section 5.1 (beyond just footnote 5) would help reproducibility.", "questions_to_authors": "A. For the study in section 6, what exactly were the annotation possibilities per sentence? No error, minor error, major error? It seems from the text that you focused on major errors. It's thus unclear how exactly you define precision/recall/F1 for Figure 4. Please clarify this.\n\nB. Please provide full details on what statistical tests you used in sections 5 and 6. It also seems that you didn't do any significance testing for section 4; are you able to provide this?\n\nC. Figure 3 (and on line 433) you claim statistical significance but say p=0.1. Given the magnitude of the difference, I assume that this is a typo (and you meant, e.g., p=0.01), as you later  in Figure 4 say that a value of p>0.05 is *not* significant.\n\nD. Lines 215–216: it's not clear what is meant by \"unaligned words... can be included in either phrase\". Does this mean that, if unaligned words are adjacent to an identified phrase, they are added to the phrase? What do \"can\" and \"either\" mean here?\n\nE. Line 238: what is |S|? Is it |S| = max(|S_1|, |S_2|) where S = (S_1, S_2)?\n\nF. Line 238: why is it meaningful to look for R(...) >= 0 if R is a ranker trained for pairwise ranking (as indicated on lines 190–191)? Should this perhaps be R(DEL[S;p]) - R(S) >= 0?\n\nG. Line 238: the answer to the previous question might help resolve this, but as-is I'm not clear on why you increase the score in the 'otherwise' case.\n\nH. Table 2: \"Del.\" isn't explained. Judging from the text just below it this is the percentage of the words in the sentence that are highlighted. Is that correct?\n\nI. Line 293: the random baseline isn't defined. Assuming I'm correct about \"Del.\", it looks like this just randomly highlights tokens with 50% probability. Is that correct?\n\nJ. Why no F1 scores in Table 2? (Assuming I'm correct about it, )\"Del.\" is IMO less useful, and the brief discussion of it in the text just below Table 2 suffices.\n\nK. Do you have data on time-to-completion for the tasks that your participants performed? That's another measure that could help support the hypothesis of the utility of contrastive phrasal highlights. For example, participants in the with-highlights condition might have been able to complete the task faster than those in the without-highlights condition, on top of the improvements in F1.\n\nL. Line 547: you hypothesize that the WMT errors are more easily spotted. This seems intuitively plausible, but it seems that your data in Figure 4 contradict this: the without-highlights precision, recall, and F1 are very similar in the WMT and Negation cases. Is that assessment accurate?\n\nM. Footnote 7: what do you mean by \"corroborate\" here?\n\nN. Since you have 12 participants, please specify how you handle ties (if there are any) in computing the majority stats for Figure 3.\n\nO. In section 5.3, it seems that you have multiple measures that reflect participants' \"perceived understanding of the explanation\", yet you give only one example. Were there others? How did the participants respond?", "paper_topic_and_main_contributions": "In this paper, the authors consider the problem of identifying differences in translated sentences. These differences may be produced by errors in statistical MT systems, by crowdsourced (or other non-professional) translators, etc. In particular, the work in this paper focuses on identifying the parts of the sentences in a translation pair that correspond to the semantic differences, while leaving semantically-equivalent parts alone.\n\nThe authors' method for this relies on pre-existing phrasal alignment packages to produce what they call *contrastive phrasal highlights*, where the focus is on finding relevant phrases rather than individual tokens. The method is relatively straightforward, which I see as a strength. The authors compare their method against previous methods by evaluating against human judgements, finding that contrastive phrasal highlighting produces results that match human judgements much more closely.\n\nThe authors then follow this up with two application-grounded studies involving bilingual crowdworkers. In the first study, which includes both English-French and English-Spanish translation pairs, participants are asked to determine whether there is a difference in meaning between the sentences in a translation pair. Here, participants who are given contrastive phrasal highlights do significantly better than those who receive no highlights, as measured against ground truth, and self-report that the highlights are useful. IAA is also higher, as measured by Cohen's kappa. In the second study, participants are asked to detect translation errors and rate their severity. Here, too, participants who are given contrastive phrasal highlights do better, and again self-report the usefulness of the highlights.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It may seem like I have a lot here, but I promise, I only provide this kind of feedback in my reviews when the writing is above a certain level, which you have very clearly surpassed! And yes, some of the comments here will be nitpicky...\n\n* The colours do not come across well when printed in black & white, which is difficult to adjust for, but I do strongly recommend that you verify that your colour choices are appropriate for those with colour-deficient vision (colorbrewer2.org is a good place to start).\n* There's no mention of translation as the primary task in the abstract; instead, more broad, general language is used, which gives the reader (or at least me) the impression that the work in this paper will be applicable to a wider range of tasks.\n* Line 037: \"show how they differ\" comes across as circular here, or at least vague: \"...to explain how [they] differ, it is more intuitive to show how they differ...\". What exactly does \"show\" mean, and how is it different from what follows (i.e., \"highlight all salient Greek and English tokens without specifying how they relate to each other\" qualifies as \"showing\" in my mind... or at least as an attempt to show...). Obviously this becomes clear through the rest of the paper, but when I was reading this passage I had no clue how to interpret it. Ironically, what's missing here is a *contrasting* example. Figure 1 shows contrastive phrasal highlights, but the alternative is vague and unspecified. An accompanying example for comparison (or rather, contrast) would be greatly helpful here. This comes up again on lines 176–177: what does \"highlighting tokens\" mean here? What would it look like as an example against the aligned phrase examples that follow?\n* Line 043: when I read \"the two inputs\" on this line, I thought, 'Which two inputs?' The easiest fix would be to remove the \"the\".\n* Paragraph starting on line 183: this is an important paragraph, as it defines the type of model that your method will be working with, and thus scoping the applicability of your work. It might be worthwhile to make this importance clear by putting this paragraph in its own subsection.\n* Figure 2: It would help to visually cue the reader that larger values are on top in step 2, e.g. via arrowheads on the lines\n* Figure 2: the aspect described in the first paragraph of section 3.2 is not made clear in the figure\n* Line 470: \"mistranslation error\" is very vague. Wouldn't all errors in a translation be mistranslation errors? Examples would help here, but at the least use a more specific error type.\n* Line 518/footnote 10: again, an example that contrasts a minor error vs. a major one would help here.\n* Abstract: 2x occurrences of \"NLP\" should be in small caps to be consistent with the small-caps typesetting of \"NLP\" and (most) other abbreviations in the main text.\n* Lines 010–011: \"phrase alignment guided\" -> \"phrase-alignment-guided\" to match line 048\n* Line 078: \"model internal\" -> \"model-internal\"\n* Line 080: \"the ability of these highlights\" -> \"these highlights' ability\" (avoids a garden path)\n* Line 105: \"seeks\" -> \"sought\"\n* Line 148: \"differs\" -> \"differ\"\n* Line 190: Use `\\cdot` instead of period\n* Line 191: First closing parenthesis looks like it's in text mode when it should be in math mode\n* Figure 2 caption: \"Explain by\" -> \"Explaining with\", to match your paper title. (I actually think you can get rid of this so that the caption just starts with \"Our approach...\")\n* Figure 2: English sentence in p_1: \"served as\" -> \"served\"\n* Line 200: \"sentence-pair\" -> \"sentence pair\"\n* Line 204: \"prohibited\" -> \"prohibitive\"\n* Line 284: \";2\" -> \"; 2\"\n* Line 268: I assume this should be Table 2, not Table 4? (It'd be ideal to refer to the tables in numerical order; I think what is currently Table 2 could be moved over to page 4 without bumping its reference to page 5; that way the first table that you refer to is called Table 1, etc.)\n* Line 291: I'd suggest removing the words from this line, as there's no need to repeat this information (it was just explained in the preceding two paragraphs)\n* Line 297: \"the token space\" -> \"token space\"\n* It's unclear whether you're using small caps for abbreviations only or you're also using it for system names (e.g., \"Random\", \"Ours\", etc.) and even those are inconsistent (line 303 has \"Ours\" typeset regularly, but it is in small caps in Tables 1 and 2; likewise, \"Random\" is in small caps on line 293 but is typeset regularly in Table 2). Personally I prefer to leave small caps for the abbreviations (including, e.g., BR in line 303 and Table 2), but whatever you do, consistency would be nice :)\n* Line 303: use minus instead of dash (like in Table 2).\n* Line 309: \"help\" -> \"helps\"\n* Line 314: The sentence beginning \"However, as highlighted...\" is an unnecessary repetition of the same point made in the first paragraph of section 4—it doesn't really say anything new. I think you can delete this sentence and instead add a sentence or two of detail to the first paragraph of section 4, specifically about how exactly the proxy evaluations can be misleading (with examples as needed). If you do this, I think it'd make sense to move the last sentence of sentence 4 to be the last sentence of the first paragraph of sentence 4 and rewrite accordingly (e.g., \"We shall turn to application-grounded evaluations in subsequent sections.\").\n* A quick googling suggests that the name of the model is \"Divergent mBERT\", not \"divergentmBERT\" as you use in several places in your paper.\n* Line 355: \"chatGPT\" -> \"ChatGPT\"\n* Line 355: \"hard\" -> \"difficult\" (if that is the intended meaning, anyway)\n* Line 358: \"All examples are validated by the authors ensuring that\" -> \"We validated all examples, ensuring that...\"\n* Line 363: \"semantic\" -> \"semantically\"\n* Figure 3 caption: \"with vs. without highlights\" -> \"with- vs. without-highlights\"\n* Figures 3 and 4: to be consistent, it'd be ideal to use the checkmark/x based labels for the conditions.\n* Lines 368–369: \"...across two language pairs, i.e., English-French and English-Spanish\" -> \"...across the two language pairs (English-French and English-Spanish)\"\n* Line 377: \"conditions where\" -> \"conditions: one in which\"\n* Line 379: \"in the other\" -> \"a second in which\"\n* Footnote 6: \"conveys\" -> \"refers to\"\n* Footnote 6: \"...\"Changed\" information...\" -> \"...\"Changed\" refers to information...\"\n* Footnote 6: Since Figures 5 and 7 are so far away, it might be preferable to refer to them as \"Figures 5 and 7 in Appendix A.3\"\n* Line 433: \"accurate, across\" -> \"accurate across\"\n* Line 434: \"and language-pairs\" -> \"and both language pairs\".\n* Line 435: remove \"the group that does not access them—\" (the following X HIGHLIGHTS label is sufficiently clear!)\n* Line 444: I appreciate the alliteration, but \"Cohen's cappa\" -> \"Cohen's kappa\". Also, I've always seen kappa values reported as decimals, so the values should be 0.51, 0.33, 0.66, and 0.52.\n* Line 469: \"(i.e., major vs. minor)\" -> \"(as major or minor)\"\n* Line 499: \"we artificially\" -> \"artificially\"\n* Line 542: \"the former\" -> \"negation errors\" (there is no proper referent for \"former\" here)\n* Footnote 10: \"Figures 10 and 9\" -> \"Figures 9 and 10\" (the cleveref package can help for stuff like this)\n* Line 559: \"user's\" -> \"users'\"\n* Lines 565 and 568: \"example(s)\" should be \"Figure(s)\". The subcaption package might help here. As I suggested for Footnote 6, since the figures are so far away, it'd help to indicate that the figures are in Appendix A.3.\n* Lines 589–590: \"in (human) translated\" -> \"in human-translated\"\n* Table 3: reinforce that the left results are for the task given in section 5 and the right results are for the task given in section 6\n* Lines 610–611: \"can be, in principle, port to any language pairs\" -> \"can, in principle, be applied to any language pair\"\n* Lines 618–619: \"quality introducing\" -> \"quality, introducing\"\n* Line 633: \"15 USD\" -> \"15 USD/h\" or \"15 USD per hour\"\n* I'm not sure how this might have happened, but your opening and closing double quote glyphs are mismatched everywhere except on lines 1055–1056."}}, {"review_id": "43DBYxFWuH", "reviewer": "Reviewer_WUTA", "summary": "", "questions": "1. Are the highlights always continuous spans? How do handle the scenarios where the alignments are not at the phrase level and are non-continuous spans? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and for providing the missing references. In the revised version of our paper, we will incorporate a discussion of those works into our Background section.\n\n&#8594; **Applicability to low-resource languages:** We agree with the reviewer that the application of our work, as well as the application of any quality estimation model, is limited in low-resource scenarios. This is something we additionally acknowledge and discuss in our Limitations section. However, despite those limitations, we also argue there is a greater necessity for the implementation of our approach in high-resource settings where machine translation is reasonably fluent and local critical errors are more likely to be missed by humans. On the other hand, low-resource machine translation typically suffers more from fluency issues that, to some extent, are also indicative of other adequacy issues. Thus, it remains unclear whether highlights-based explanations alone are useful signals in low-resource settings to start with or whether other types of explanations are needed.\n\n&#8594; **Reliance on explainer:** We agree with the reviewer that by problem construction, the success of our approach, i.e., the quality of the produced highlight-based explanations, is conditioned on the success of the underlying explainer. However, this is an issue with *any* explainability technique, including the prior work we compare our approach with. Additionally, although this is not an application we explored in the current paper, our explanations could potentially be used to expose issues with the explainer and further improve it.\n\n&#8594; **Q1. Handling non-contiguous spans:** This is correct; our highlights are always continuous spans as we use Simalign as our aligner, which does not handle alignments between non-continuous spans. However, in principle, such cases could be addressed within our pipeline if we replace Simalign with a non-contiguous alignment-based model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Unlike existing techniques for MT explainability that highlight tokens independently, the proposed contrastive phrasal-pair erasure takes into account the input's structure and explicitly model the relationship b/w contrasting inputs. The results suggest that contrastive phrasal highlights assist bilingual speakers in detecting fine-grained differences in translated texts and other critical mistakes due to mistranslations. This creates opportunities for designing better machine-in-the-loop pipelines to detect MT errors.  \n\nThe paper is well written and easy to follow. The experiments are thorough with ample analysis.", "reasons_to_reject": "There are no major reasons to reject. However, based on the prerequisite of constructing a phrase table for the technique to work, the use of this approach in low-resourced scenarios might be limited.\n\nThe approach relies on divergent sentences for fine-grained meaning differences. This likely makes the approach sensitive to the quality of this candidate as paraphrase models are imperfect, especially when the input is domain specific.", "questions_to_authors": "1. Are the highlights always continuous spans? How do handle the scenarios where the alignments are not at the phrase level and are non-continuous spans? ", "paper_topic_and_main_contributions": "Taking a departure from the conventional token-relevance based approaches for model explanability, this paper proposes to focus on the differences between two input sentences to explain an output translation. The paper introduces phrase alignment guided erasure to generate contrastive phrasal highlights to explain predictions of semantic divergence model. This augments post-hoc explanation methods by contributing unsupervised contrastive explanations of translation differences.\n\nThe basic design follows a phrase-alignment guided occlusion strategy that ranks bilingual sentence pairs based on the degree of their semantic similarity. Given two input text, a set of candidate counterfactual inputs is extracted by masking a subset of phrase pairs using a statistical MT based phrase table. Then, the model's prediction is explained by selecting the phrase pair whose omission maximally increases the similarity between the two inputs. To guide this, two measures are proposed - 1. a penalty term that encourages the extraction of a high contrast phrase pair under the model, and 2. a brevity reward that encourages shorter phrases.\n\nFinally, instead of relying fully on proxy evaluation, application-grounded evaluation measures are proposed. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Arras et al., 2017, “What is relevant in a text document?”: An interpretable machine learning approach\n\n[2] Moradi et al., 2021, Measuring and Improving Faithfulness of Attention in Neural Machine Translation, EACL 21\n\n[3] Murdoch et al., 2018, Beyond word importance: Contextual decomposition to extract interactions from LSTMs, ICLR 2018\n\n[4] Moradi et al., 2019, Interrogating the explanatory power of attention in neural machine translation"}}]

Author rebuttal:

Meta-review: The paper focus on how to better assist humans to identify translation differences, proposes a new approach, and performs detailed analyses and comparisons between different approaches. 

All reviewers agree that this paper is sound and exciting. The paper is well-written and easy to follow. 

But in the final version, the authors should include discussions on the scale of the data used in the application-grounded evaluations, as pointed out by reviewer  uUif.
Decision: Accept-Main