Towards General Error Diagnosis via Behavioral Testing in Machine Translation
Authors: Junjie Wu
Lemao Liu
Dit-Yan Yeung
Venue: EMNLP 2023 Findings (2023)
Keywords: behavioral testing
machine translation
TL;DR: This paper presents a novel bilingual translation pair generation based behavioral testing approach that could provide comprehensive and faithful behavioral testing results.

Abstract: Behavioral testing offers a crucial means of diagnosing linguistic errors and assessing capabilities of NLP models. However, applying behavioral testing to machine translation (MT) systems is challenging as it generally requires human efforts to craft references for evaluating the translation quality of such systems on newly generated test cases. Existing works in behavioral testing of MT systems circumvent this by evaluating translation quality without references, but this restricts diagnosis to specific types of errors, such as incorrect translation of single numeric or currency words. In order to diagnose general errors, this paper proposes a new Bilingual Translation Pair Generation based Behavior Testing (BTPGBT) framework for conducting behavioral testing of MT systems. The core idea of BTPGBT is to employ a novel bilingual translation pair generation (BTPG) approach that automates the construction of high-quality test cases and their pseudoreferences. Experimental results on various MT systems demonstrate that BTPGBT could provide comprehensive and accurate behavioral testing results for general error diagnosis, which further leads to several insightful findings. Our code and data are available at https: //github.com/wujunjie1998/BTPGBT.

Towards General Error Diagnosis via Behavioral Testing in Machine Translation
Junjie Wu', Lemao Liu?, Dit- Yan Yeung'
'The Hong Kong University of Science and Technology, Hong Kong SAR, China
TTencett AI Lab
junjie. wuêconnect. wsw. hk, redmondllu@ttencett com, dyyeung@cse. ust. hk
Abstract
Capability: NER
x: In order to tell new storiss to the marker, Zuckerberg sold the story about "Meu-anbwese"
Behavioral testing offers a crucial means of diagnosing linguistic errors and assessing ca- pabilities of NLP models. However, applying behavioral testing to machine translation (MT) systems is challenging human efforts to craft references for evaluating the translation quality of such systems on newly generated test cases. Existing works in behav- ioral testing of MT systems circumvent this by evaluating translation quality without ref- erences, but this restricts diagnosis to specific types of errors, such as incorrect translation of single numeric eurrency words. In order to diagnose general errors, this paper proposes a new Bilingual Translation Pair Generation based Behavior Testing (BTPGBT) framework for conducting behavioral testing of MT sys- tems. The core idea of BTPGET is to employ a novel bilingual translation pair generation (BTPG) approach that automates the construc- tion of high-quality test cases and their pseudo- references. Experimental results various systems demonstrate that BTPGBT could provide comprehensive and accurate behay- ioral testing results for general еттоr diagnosis, which further leads to several insightful find- ings. Our code and data are available at https: //gi thub. com/ homm//ujunjiaa iie 998/BTPGBT.
H ALE/DREUTT "
y: #7 Sa
H DD T
T 09
BIPG (x,r)
2: In order to tell new stories the market, Zuckerberg told the story about "omask""
generally requires
F: 1) ta #i #USE
"cmuske"
BTPG Generntes (x',
x1 ordar tell new stories to the market, Zackerberg told the story sbout
r: #
HU T
"TFE""
y aimiiiitt to HU "Metaverse" BO
(COEETy.r) CONETly'r') >
An example of behavioral testing the DeepL translator on the capability of translating named entities (NER). the original input and r is its reference. The masked counterparts of æ and and f respectively, with the to-be-masked segments highlighted boldface. generated test case, x', re- places the named entity "Meta-universee in a with "the Metaverse", and its crafted pscudo-reference is denoted as r', DeepL's outputs for z and x' are denoted as y and y'. DeepL does not pass the behavioral testing on x'' since it outputs an erroneous y that directly copies "Metaverse" instead of translating it """""""" which identified by the substantial difference in the COMET scores between y and y²
Figure 1:
denoted
testing probes various system capabilities for error diagnosis by inspecting input-output behaviors test cases with targeted edits, irrespective of knowl- edge about the system's 's internal structure (Beizer, 1995). Therefore, it is suitable for providing fine- grained evaluations of different capabilities and diagnosing errors for MT systems.
1 Introduction
In recent years, machine translation (MT) systems have achieved significant advancements in trans- lation performance. Yet current MT' systems are still brittle and could generate erroneous transla- tions that lead to severe commercial losses which makes error diagnosis of MT systems a crucial task to be solved. Behavioral testing, which has been widely applied in software engineering re- search to test complex systems, is desired di- rection tackle this issue. Generally, behavioral
There are systematic studies of behavioral test- ing on classification tasks such sentiment anal- ysis (Ribeiro et al., 2020), hate speech detcc- tion (Röttger et al., 2021) and clinical outcome prediction (Van Aken et al., 2022). However, im- plementing behavioral testing in MT tasks poses significant challenges. The main reason is that it is usually difficult to automatically obtain the ref-
hhttps://wwar http://wwww.addroooollooooooo comm

---

erence translation for test case modified from source sentence, making it difficuul to automatically judge the quality of the translation of the test case. To avoid this challenge, existing work attempts to judge the translation quality of such test case without its reference translation, yet they can only diagnose translation errors related to specific types of edits that target certain capabilities. For exam- ple, He al. (2020); Sun et al. (2020); Gupta et al. (2020); Ji al. (2021) only diagnose translation errors on test cases with the editing of a single noun or adjective word, and Hc et al. (2021); Wang et al. (2021); Raunak et al. (2022) can only diagnose in- correct translation of noun phrases, quantities or currency units that is related to the edits on them.
ioral testing to diagnose general translation errors targeting different capabilities. Extensive experi- ments on six MT systems using BTPGBT demon- strate that our method excels in evaluating multiple MT system capabilities with high error detection precision, outperforming prior works. Furthermore, our in-depth analysis of the testing results uncovers several insightful findings. The main contributions of this paper are summarized below:
We design a new framework that can automat- ically conduct general behavioral testing to di- agnose MT systems. Specifically, we propose a novel bilingual translation pair generation method to generate high-qualing test cases and corresponding pseudo-references for behav- ioral testing.
To address these challenges, this paper presents Bilingual Translation Pair Generation based Behavior Testing (BTPGBT), novel framework for behavioral testing in MT systems. The core idea our framework is the novel Bilingual Translation Pair Generation (BTPG) approach that can auto- matically generate high-quality test cases and their pseudo-references from standard MT test sets, 2 which allows the diagnosis of general translation errors in test cases targeting various MT system capabilities. As shown in Figure 1, BTPG takes source sentence and its reference as an input trans- lation pair, then masks specific aligned segments "MMet--nnivese" and "プ # in Figure 1) in both sentences targeting the capability that needs to be tested (NER in Figure 1). Only masking aligned segments enables BTPG to make the best use of the structure information of unmasked positions in the original source and reference during generation, which enhances its generation quality. BTPG then generates a new bilingual translation pair as the test case and its reeude-referrece by using Chat- 3 a large language model proposed by Ope- nAI that has shown impressive performances on various NLP tasks, to infill the masked segments in both sentences at once. In this way, the test case and its rreeeneeeeeerence could have similar quality human constructed ones, illustrated in $4.2. With the high-qualt high-quality test cases and their pscudo- references crafted by BTPG, we can conduct behav-
2. Through extensive experiments on six MT sys- tems, we demonstrate our proposed method's effectiveness, leading to several insightful findings.
2 Revisiting Behavioral Testing in MT
2.1 Principled Definition
Behavioral testing (Beizer, 1995), which studies the capabilities of a system through examining its input-output behaviors on targeted test cases without knowing its internal information, has been demonstrated to be effective in NLP classification tasks (Ribeiro al.. 2020: Röttger al., 2021; Van Aken et al., 2022). Following these works, the principled definition of behavioral testing MT can be extended as: given an input sentence x and its reference r. Let x' be a test case edited from æ for testing a specific capability of an MT system M (e.g., x' in Figure 1 targeting the NER capability), and r' be the reference translation of x' We say M passes the behavioral testing on x' if it translates both æ and x' in high-quality according to their references r and r'. The idea behind this definition straightforward: if M has the specific capability to handle the targeted edits in x'. it should translate a' in the same quality as æ. Specifically, we first calculate the absolute dif- ference of the quality between y and i.,, M'' translations of and æ'. as follows:
with references, but this is not a very strong requirement since there are many off shelf test datasets and particular this standard setting behavioral testing. For instance, the pioneered re- search about behavioral testing (Ribeiro et al., 2020) and many follow-up works (Röuger al., 2021; Van Aken et al.. 2022) use test dataset with ground-truth labels. Phttps: / //chttcctttoot openai. openai.com/
"Our framework indeed involves
Diff y (Q) Qual(y/)l (1)
Here we can apply various quality measuring strate- gies as Qual() to evaluate y and y while it would

---

be more authentic if we use and r' for evaluation. M passes the behavioral test on the test case x' if it meets the following criteria:
the behavioral testing if its translations of these sentences have similar syntactic structures, based on an assumption that the translations of similar sentences should be analogous. The reason why they only modify a single noun or adjective is to avoid largely shifting the structure of æ, yet still limits the types of capability they can test as well as translation errors they can find. Further, even small modifications in æ could dramatically change its semantic meaning, making the assumption of these methods not stable and thus largely biases their corresponding evaluation results.
Qual(y) 2 a Diff(y, 3/) 3
(2)
where a and 3 are thresholds ranging in [0, 1]. In practice, a should be high and 3 should be low to ensure the lesting effectiveness. Interpreting the above criteria is intuitive: if (y) is higher conclude that y is high-quality translation of E. On this basis, if Diff(y. y. v²) is lower than B, can conclude that y' keeps similar and high translation quality with y and thus M passes the behavioral testing. Otherwise, we say M does not pass the test and y is a diagnosed erroneous translation. As an example, the Deepl. translator in Figure fails on the behavioral testing on w' since the erroneous translation s breaks Eq. 2, which is caused by the incorrect translation of "Metaverse" in y/. We use Pass Rate, i.e., the percentage of test cases that an MT system passes, to quantify the behavioral testing results, where higher score means less translation errors.
than a,
Other Error Diagnosing Methods. To provide fine-grained evaluation of MT systems, various evaluation methods except behavioral testing have been proposed, such as evaluating robustness adversarial perturbations (Zhang et al., 2021; Lai et al., 2022), ambiguous words (Emelin et al., 2020) and diverse biases (Saunders and Bymne 2020; Wang et al., 2022a). However, they all focus on a specific type of phenomenon in MT, thus lack the generality in diagnosing translation errors.
3 BTPGBT Framework
To solve issues in previous works, we propose novel BTPGBT framework to conduct behav- ioral testing for general translation error diagnosis. Given source sentence and its reference the core idea of BTPGBT is to automatically edit one or more segments in æ and construct various test cases x' and their pseudo-references r' target- ing specific capabilities of MT systems. Then we can diagnose these systems following the steps de- scribed in $2. In the following sections, we first describe how we implement the above core idea, then illustrate the capabilities BTPGBT tests well how to generate test cases targeting these capabilities for behavioral testing.
2.2 Related Work: Challenges and Existing Solutions
However, it is challenging to craft a high-quality reference for each test case since it requires much human efforts. Hence, previous works propose var- ious approaches without the references of test cases to diagnose translation errors on testing results.
Existing Solutions for MT Behavior Testing. (Wang et al., 2021; He et al., 2021; Raunak et al.. 2022) construct test case x' by modifying spe- cial components like numbers, physical units and noun phrases in æ, since the reference translations of these components can be obtained ahead from external dictionaries. They define M passes the behavioral testing on x' if the reference transla- tions of such components appear in y Although these behavioral testing methods hold a high pre- cision in diagnosing translation errors, the errors found by them are limited to such specific com- ponents. In other words, the recall of translation errors identified by these approaches is low. On the other hand, (He al.. 2020; Sun et al., 2020; Gupta et al., 2020; Ji al., 2021) first edit a single noun or adjective in x to create series of similar sentences as test cases, then denote M as passing
3.1 BTPG Method
However, implementing the core idea of BTPGBT is challenging due to two reasons. First, an appro- priate strategy to decide which segment in æ and r can be modified is needed, since we do not want the modification to largely shift the meaning of non- edited parts in æ and and hampers the generation quality. Second, how edit the selected segments to craft fluent ø' and r' is non-trivial problem. while similar works like text infilling (Zhu et al., 2019; Donahue et al., 2020; Xiao et al., 2022) can- not be directly applied since we need to edit both x

---

and at once. To tackle these two issues and con- struct qualified x' and r' effectively, we present the bilingual translation pair generation (BTPG) approach and detail it in the following.
Jiao et al., 2023; Hendy et al., 2023: Peng et al., 2023), inspiring us to apply ChatGPT generate x' and r'. However, ChatGPT is still struggling with generating unbiased translations without hal- lucinations (Hendy et al., 2023; Guerreiro et al., 2023) when handling machine translation related tasks. To tackle this issue and make the best use of the power of ChatGPT, we propose to incorpo- rate the masked translation pair (æ,, f) that contain semantic information of (x,r) for constructing new bilingual translation pair, Given a masked pair (ă, f), we instruct ChatGPT through its API to first fill in the masked posi- tion in &, then fill in the masked positions in f based on the filled ê. We then guide ChatGPT to paraphrase the filled sentences to ensure fluency, After all, obtained new bilingual translation pair (æ', r') where x' is the test case and r' is its pscudo-reference. Through repeating the above steps, we can craft various test cases and their pseudo-references from (x, r).
Determining Modification Positions. The first step of BTPG to determine the segments that will be edited in the source sentence æ, where seg- ments here consist of words and phrases. Inspired by Knight (2000), we require that only consecu- tive segment in æ that is solely aligned with another consecutive segment in r can be edited, which avoids largely shifting the structures and meanings of non-edited segments. Take Figure 1 as an ex- ample. The segment "In order tell" in cannot be edited since it is aligned with a non-consecutive segment "> Jt Hhl y. Specifically, we ap- ply the Mask-Align tool proposed by Chen et al. (2021a) to obtain a word-to-word alignment be- tween x and r for extracting two types of segments in that can be edited:
build prompts
word in æ that only aligns with word or consecutive phrase in
3.2 Constructing Test Cases Targeting Various Capabilities
consecutive phrase in that only aligns with word or a consecutive phrase r.
With BTPG, we can make edits on any segments in that is aligned with another segment in to build test cases and their pseudo-references for behav- ioral testing. This generality enables BTPGBT diagnose translation errors towards various capabil- ities of MT systems. In the following, we describe the nine types of MT capabilities we diagnose with BTPGBT as well as how to use BTPG to craft targeted test cases and their pseudo-referencer
where phrases in æ and r are identified by Stanza 4, If a longer segment overlaps with another shorter segment, we will only keep the longer segment since it is usually more complete. Finally, we se- lect one or several segments from the extracted segments for further editing.
Generating Bilingual Translation Pairs. Next, we illustrate how to craft a new bilingual transla- tion pair (æ', r'). Specifically, we first mask the selected segments in æ and its aligned segments in r to construct masked translation pair (á, f), then construct a new bilingual translation pair (a', r') via filling in the masked positions in (.. f). How- ever, existing text infilling approaches are con- ducted solely on source sentences (Zhu et al., 2019; Donahue et al., 2020) or target sentences (Chen et al., 2021b; b; Wang et al., 2022b; Xiao et al., 2022), and thus could not be adapted our task since need to fill in & and f at once.
First, we consider a reliable MT system should appropriately understand the meanings and func- tions of segments that include words with different parts of speech. Therefore, we construct test cases with edits targeting seven types of parts of speech (POS). Next, it is crucial for MT systems to under- stand the tenses of different verbs in sentence for correctly translating this sentence, thus we create test cases that edit the tense of a verb in a sentence to diagnose this capability (Tense). Finally, named entities that usually convey important information of sentence should be translated properly. To this end, we build test cases with named entities- based edits investigate MT systems' capability to properly translate named entities (NER). Except for modifying segments to other segments in the same category, we are also interested in MT sys- tems' capability to handle uncertain modifications
Recently, the large language model ChatGPT designed by OpenAl has shown impressive perfor- mances on various NLP tasks including text gen- eration and machine translation (Qin et al., 2023;
"https://stanforddllp tit waab.iiosstta//////

---

Capability
Description/Test Test Case Generation Instruction
Example
POS
Whether an MT system can understand the fune-
Ori:
(Noun/ Verb/Adverb/
Aooun/Adjective/
tion and meaning of segment in source sentence
printed out the page and took it to my local shop.
that contains a word with certain POS tag. Test cases are created by replacing such segment in the source sentence with another segment that has a word with the same POS tag.
Test Case: (Noun)
Preposition/
printed out the page and took it to my local book- store.
Others)
Tense
Whether an MT system is able to identify and un- derstand the tense of a verb in source sentence and output translation with the correct tense. Test cases are created by changing the tense of non- past perfect tense verb in the source sentence to the past perfect tense.
Ori:
Tracking number will be provided after dispatching the parcels.
Test Case:
Tracking number had been issued after dispatching the parcels.
NER
Whether an MT system can translate named en- tities (NEs) correctly. Test cases are crafted by changing named entily in source sentence to another named entity in the same NE category.
Ori:
This Arab state plans to boost trade with Russia
Test Case:
This Arab state plans to boost trade with Turkey.
General
Whether an MT system can accurately understand and translate certain segments in a source sentence. Test cases are created by randomly modifying some segments in the source sentence, while limit the total number of words in the modified segments to be less than 0.2 x len(
Ori:
The stage when Chinese companies were burning
money overseas a large scale has passed.
Test Case:
The stage when people were sending money overseas on a large scale has passed.
Table Descriptions of different capabilities that BTPGBT evaluates for error diagnosis and instructions on how to edit corresponding test cases. Modified positions in each example sentence are in italic.
in the source sentence. Therefore, we craft test cases that do not limit the type and number of edits to examine this specific capability (General).
matic and human evaluations to measure the quality of the BTPG method. Next, we apply BTPGBT to diagnose six MT systems to show the effectiveness of our proposed behavioral testing framework, and perform in-depth analysis on the evaluation results to obtain some insightful findings.
Detailed descriptions of each capability and the corresponding test cases generation instructions are shown in Table 1. For each capability, the selected segments that will be masked in (ă, f) should also meet the condition shown in Table Then we can craft test cases x' and their pseudo-reference r' with BTPG for behavioral testing.
4.1 Settings
We conduct our experiments using data extracted from the test sets of the WMT21/22 En-Zh/Zh-Bii news translation task to build dataset named WMT for our experiments, which are not included in the training data of our evaluated research MT systems. As for detailed settings of the BTPGBT framework, we apply the reference-based COMET- 22 metric 1wwr22-COMET-da as the quality mea- surement Qual() and ,3 in Eq. 2 are set to 0.8 and 0.05, respectively 6
3.3 Judging Behavioral Testing Results
With test cases and their pseudo-referenee in hand, we judge an MT system M's performance on test case x'' following the principled definition in $2.1 using its pseudo-refcrence r'. To further enhance the reliability of Eq.2 when judging behavioral test- ing results, for each crafted test case w', we keep it only if Diff(r, r') < B, using the reference-free version of the widely used metric COMET (wmr20- COMET-qe-da) (Rei et al., 2022) as the quality measurement Qual(). This filtering step avoids the situation that Diff(g, y²) < 3 is caused by the dif- ference between their corresponding references.
4.2 Quality of BTPG
Since we aim at using BTPG to craft high- bilingual translation pairs as behavioral testing cases and their pseudo-references, we are inter- ested in its generation quality. To this end, we randomly sample 200 source sentences and their
4 Experiments
hittps://wwe. wwwtmt.trg//mtttt.t org/wmt21//
https:
//www.stet wwwtsttttttrrrre org/wmt22/
We conduct experiments on the English-Chinese Chinese translation task. First, we implement both auto-
"We provide details on how we set the value of the two hyperparameters in Appendix B

---

references from WMT and use BTPG to create 200 bilingual translation pairs as test cases and their pseudo-references for evaluation ?
SSF
TSF
TA
COMET-20
BART+BITIMT
2.75
2.64
2.44
23.28
Ref BTPG
2.96
2.90
2.83
34,37
2.93
2.83
2.85
36.38
4.2.1 Evaluation Methods
Table 2: Quality evaluation results of test cases and their pseudo-references. SSF/TSF/TA are human annotated scores averaged among annotators.
We conduct human evaluations to evaluate the test cases and their pseudo-references generated by BTPG from three perspectives: source sentence fluency, target sentence fluency and translation ad- equacy. For translation adequacy, we also apply an automatic metric as an evaluation supplement.
BART+BiTIMT. This approach is combination of two text infilling models that generate æ' and r' by first infilling the masked source ă, then infilling the masked reference f. For source side infilling, we apply the widely used BART model (Lewis et al., 2020). As for the target side, adopt the BiTIMT model (Xiao al., 2022) to infill & on top of x' filled by BART craft r'.
Source Sentence Fluency (SSF). It measures whether the test case x' is semantically meaningful and grammatically correct. For human evaluation, we ask human annotators to score the fluency of x' based on three- point rating scale, in which 1/2/3 represents unsatisfiicdfaairstttiiiii, respectively.
Ref. We are also interested in the quality of x! and r' crafted by BTPG compared the source sentence z and its reference r. Specifically, we use the 200 original translation pairs sampled from WMT as x' and r' for comparison.
Target Sentence Fluency (TSF). Similar to SSF,
it measures whether the pseudo-reference r' of x' is semantically meaningful and grammatically correct. Likewise, adopt the same three-point scale score each r' during human evaluation.
4.2.3 Results
Table 2 lists the quality evaluation results. We observe that Ref and BTPG largely outperform BART+BiTIMT across the board, demonstrating the limitation of prior text infilling approaches that merely consider filling the masked positions with- out ensuring the fluency of the completed sentence. This characteristic also strictly lowers the transla- tion performances of BART+BiTIMY' proven by its low TA and COMET scores. Conversely, x' and r' crafted by BTPG obtain comparable fluency scores with Ref, which ensures the quality of test cases and their pseudo-references constructed by BTPG. Surprisingly, BTPG achieves slightly higher TA and COMET scores than Ref, demonstrating that the pseudo-referencee r' can act as the reference translation of crafted test cases x'. This conclusion is crucial since it supports our idea conduct MT behavioral testing following the principled defini- tion in $2 using the references generated test cases.
Translation Adequacy (TA). It captures whether the crafted pseudo-reference r' of x' has translated all the information conveyed by x'. For human evaluation, we ask annotators to compare x' and r' word-by-word and give a score for r' following the same scoring scale SSF. As for automatic evaluation, since we do not have ground truth for r'. we apply the aforementioned wm120-COMET- qe-da metric to measure translation adequacy where a higher score indicates a more adequate translation.
For each human evaluation task, we invite three annotators who proficient in both English and Chinese rate the given text and they achieve Krippendorff's alpha score (Krippendorff, 2011)< of 082/0.80/0.80 on SSETSE/TA, respectively, indi- cating a high inter- agreement. See Appendix C for detailed instructions of our human evaluations.
4.2.2 Baselines
To illustrate the effectiveness of BTPG, we com- pare with two baselines:
4.3 Testing MT Systems with BTPGBT
Subsequently, apply BTPGBT to examine vari- ous capabilities outlined in Table 1. For each capa- bility except General (which is tested in Table 5), we randomly sample 1000 source sentences and their references from WMT and filter those that do not contain segments required by the tested ca-
To obtain more general evaluation results, choose target the General capability when generating test cases skip the filtering step in $3.3.
"Wo do not use the latest reference-free wmr22- COMETRéwi-da metric since it was unavailable at the time of our experiments (May 2023).

---

Task/Capability
Prompt Template
Direct Translation
"system", "content": "You are a helpful assistant that translates English to Chinese." ", "role": "user", "content"; "[SRC]"
"role":
POS
"role": "system", "content": "You are a helpful assistant that translates English to Chinese.", "role": "user", "content"; "You are given an English sentence and its Chinese translation. In each sentence, an (POS word/phrase) has been masked with the '<mask>' token. Your task is to first fill in the masked token in the English sentence using an (POS word/phrase] other than (the or original POS word/phrase) without modifying any of the unmasked tokens. Then, use the filled English sentence to fill in the masked token in its corresponding Chinese translation. If necessary, make modifications to the filled Chinese translation to ensure fluency while preserving the meaning. Finally, please output the filled English sentence and its filled Chinese translation in the format of 'Filled English: \n Filled Chinese:". An English Sentence: [MASKED ENGLISH]. An Chinese Translation: [MASKED CHINESE]"
Table 3: Prompt templates for querying the ChatGPT model. Braces will be filled by modified segments in real usage.
pability (e.g., noun/noun phrase when testing the "Noun" capability). Then we use BTPGBT to con- struct test cases targeting these capabilities to test different MT systems and obtain the corresponding Pass Rates. For each capability in Table 1, we use a slightly different prompt for ChatGPT based on its instructions and list the prompt template targeting the POS capability in Table 3 as example. The prompts targeting other capabilities are shown in Appendix SA. Noting that we only generate one test case for given source sentence this exper- iment, yet crafting more test cases can be simply done by repeating the generation process.
all the capabilities, indicating the need for more advanced strategies in the design of robust MT sys- tems. On the other hand, although ChatGPT largely outperforms Transformer, it still makes more trans- lation errors compared to the top commercial MT system, particularly on test cases targeting Noun and Adv. This result illustrates that there still exists room for ChatGPT to become stable translator. To surprise, the commercial MT systems fail on 6%-17% of test cases targeting different capa- bilities, even when they are under regular testing and improvement. This finding further illustrates the importance of performing fine-grained error diagnosis with BTPGBT. Note that DeepL outper- forms other commercial systems on all capabilities except Others, showing its strong ability to handle different types of edits in the input sentence.
MT Systems. In this work, we evaluate various commercial translation systems, including Google Translate (Google)* 9, Microsoft Azure Translate (MS-Translator) Deepl. Translate (DeepL.) 11 and Tencent Transmart (Tencent) (Huang al., 2021) 12, We also evaluate Transformer- based (Vaswani et al., 2017) NMT model as an evaluation (anallmmen(Tranffrrmoo
Large-scale Behavioral Testing. Since BTPGBT can generalize large numbers of test cases and their pseudo-references efficiently, we also conduct behavioral testing on MT systems at scale to obtain unbiased error diagnosis. Concretely, we craft 20540 test cases targeting the General capability mentioned in Table 1 to test MT systems. As shown in Table we notice that the performances MT systems are similar to Table 4, where DeepL still outperforms other systems and all the systems outperform Transformer. These results further demonstrate that behavioral testing results in Table 4 are unbiased and could provide reliable error diagnosis of MT systems.
13,
As mentioned in $3.1, ChatGPT has recently shown strong ability in machine translation tasks. Therefore, also evaluate ChatGPT using BTPGBT. The prompt for instructing ChatGPT as translator shown in Table (Direct Translation).
Results. Table 4 lists the behavioral testing re- sults on different MT systems. We observe that all the other systems outperform Transformer on
"https://translate. www.gooooo...ooooooooo... com/
http::////laaa..iicrsssoos com./e com/en-us/azuure//////oo us/azure/
committveseeviesstrraaatrr/
4.4 Effectiveness of BTPGET
lhttpps:/wwwwdeeppp.co com.tp/ssaammtttttaaarrrr
Phttps://transmart. qg. con/ /h- CN/ index
To further demonstrate the effectiveness BTPGBT in diagnosing translation errors, we in-
USee Appendix D for details of this model.

---

MT Systems
Noun
Verb
Adj
Adv
Prep
Others
NER
Tense
Google
88.26
88.83
89.83
84.19
85.20
94.00
88.94
89.04
MS- Transaaalt
87.25
83,25
91.49
85,58
86.23
91.07
87.00
89.37
DeepL.
92.11
89.63
91.91
86.51
89.29
93.27
91.03
91.36
Tencent
88.87
84.84
88.80
85.58
88.27
91.07
88.34
87.04
ChatGPT
87.05
86,70
90.66
84,65
83.67
92.68
88,34
89.70
Transformer
70.24
69.68
71.99
68.84
69.39
79.80
71.30
70.76
Avg
85.63
83.82
87.45
82.56
83.68
90.32
85.83
86.21
Size
494
376
482
215
196
683
669
301
Table 4: Pass rates (%) of behavioral testing targeting eight different capabilities. Under each capability, the best results among all MT systems boldfaced. Size refers the number of test cases. Adj. Adv and Prep refers the adjective, adverb and preposition capability in Table 1, respectively.
MT System
Pass Rate
a: Johnson is teetering on the edge favour with his own MPs; if further photos surface, they may push him over the edge.
Google
87.56
MMS-Translator
86.30
DeepL.
89.65
Tencent
86.02
ChatGPT 86.53
w': Johnson balancing the edge of favour with his
Transformer 69.36
MPs; if further photos surface, they may push him over the edge.
Table 5: Large-scale behavioral testing results on 20540 test cases targeting the General capability.
Meaning: Johnson is balancing with his MPs; if further photos surface, they may push him the edge.
Method
Precision
Recall
Figure 2: An example of behavioral testing of DeepL on the Verb capability. Deepl. makes an error in y that it doe not translate "on the edge of favour" in x'.
Dep
34.67
49.06
BTPGBT
82.61
71.70
Table 6: Precision (%) and recall (%) of Google's erro- neous translations found by different methods.
that using pseudo-references to evaluate the quality of test case translations is beneficial for detecting more true translation errors.
vestigate its testing results from two perspectives: (1) the proportion of actual erroncous translations within the erroneous translations diagnosed by BTPGBT (Precision). (2) how many actual erro- neous translations can BTPGBT identify (Recall). To this end, we first randomly select 100 distinct source sentences in WMT and pick their test cases crafted in the previous large-scale behavioral test- ing, then evaluate Google's translations on these test cases with BTPGBT. To calculate Precision and Recall scores, we manually annotate all the actual erroneous translations produced by Google on these test cases. Besides BTPGBT, we also ex- periment with the method designed by He et al. (2020) which identifics translation errors by com-- paring syntactic structures between original trans- lations and translations of test cases (Dep), using its recommended configurations. As shown in Ta- ble 6, BTPGBT achieves much higher precision and recall scores compared to Dep, demonstrating
4.5 Error Tendency
Notably, all MT systems in Table 4 tend to gener- ate more translation errors towards the capabilities Verb and Ady, while achieving relatively high pass rates Others. In this section, we study this phe- nomenon from the linguistic perspective. For the Others capability, we investigate the corresponding test cases and find that 98.60% of the test cases are related to numeric words substitution. Since numeric words usually do not affect the structures of sentences, it is not hard for MT systems to cor- rectly translate different numeric words without affecting the translation of sentences, thus leading to high pass rates test cases targeting the Others capability.
Conversely, verb often forms collocations with different prepositions depending on the context. This factor makes challenging for MT systems to correctly interpret the functions and meanings

---

MT Systems
Adj
Verb
NE
erroneous translations whose translation errors ap- pear at the modified position. Results are shown Table We observe that few translation errors iden- tified by BTPGBT appear at the modified position, indicating that existing MT systems are proficient at translating individual segments in a sentence, even when these segments are replaced. However, they lack the ability to comprehend changes in sentence's structure caused by segment modifi- cations, which consequently leads to translation errors non-modified positions.
Google
7.14
16.67
16.67
DeepL
5.88
15.79
14.82
ChatGPT
15.39
8.57
4.44
Table 7: Percentage (%) of erroneous translations that appear at the modified positions.
verbs across various sentences, thus leading to errors when translating test cases with verb-based edits. An illustrative example is shown in Figure 2. Switching the verb "teetering" in æ to "balancing" should only change the sentence translation at the modified position. However, this edit makes the strongest MT system Deepl forget translate "on the edge of favour" in v/. probably because DeepL. pays much attention the collocation "balance with" in æ'. The translation difficulty of verbs may also affect MT systems' capability to translate ad- verbs and lead to the low pass rates on test cases targeting the Adv capability, since adverbs are of- ten used to modify verbs in sentence. In conclu- sion, through splitting evaluations of translation behaviors, researchers can systematically diagnose translation errors in MT systems, thereby facilitat- ing more effective improvements.
Conclusion
In this paper, we introduce novel behavioral test- ing framework BTPGBT to diagnose general trans- lation errors for MT systems. The key idea BTPGBT is to auto-generate test cases and their pscudo-refcrences, which facilitates the diagnosis of translation errors targeting various capabilities of MT systems. To this end, we design the BTPG approach to craft test cases and pseudd-refeeencee, whose effectiveness has been proven by extensive experiments. Experimental results on six differ- ent MT systems further demonstrate that BTPGBT can provide general and faithful behavioral testing results and lead to some insightful findings.
4.6 Appearing Position of Errors
Limitations
As demonstrated Table 4, BTPGBT enables the identification numerous translation errors MT systems. Given that these errors arise from test cases with segment-level edits, there are two poten- tial reasons for such translation errors:
In our proposed behavioral testing framework BTPGET, we incorporate a CharGPT-equipped module to automatically craft test cases and their pseudo-references for general error diagno- sis. However, ChatGPT does not keep the same translation and text generation quality on all lan- guages, which may hinder us applying BTPGBT on low-resource translation tasks. This character- istic requires us to introduce additional methods under these scenarios. We will study this direction in our future work.
The segment-level edit directly induces the translation егrоr, resulting in incorrect trans- lation at the edited position, as exemplified in Figure 2.
The segment-level edit indirectly induces the translation eπror. In this case, even though the edited position might be correctly trans- lated, it affects the correct translation of other unedited parts of the test case and thus leads to translation errors.
Ethical Considerations
Since the processes of crafting pseudo-references is totally automatic, some toxic and harmful test cases might be generated when querying the Chat- GPT model. It is also possible that some toxic and harmful translations MT systems when they make translation errors on the test cases. These potential issues require the actual users to perform comprehensive data post- processing before conducting behavioral testing
To dive deeper into this phenomenon, we investi- gate whether the translation errors diagnosed by BTPGBT appear at the edited position, shedding light on why MT systems fail on such test cases. We annotate all the erroneous translations outputted by Google, DeepL and ChatGPT on three capabil- ities in Table 4, and calculate the percentage of
outputted by commercial

---

using BTPGBT, and make sure that the found trans- lation errors will not be used under non-dcbugging situations.
Nuno M Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023. Hallucinations in large multilingual translation models. ArXiv preprint, 0bs/2303.16100)
Acknowledgement
Shashij Gupta, Pinjia He, Clara Meister, and Zhendong Su. 2020. Machine translation testing via pathologi- cal invariance. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, pages 863-875.
This research has been made possible by funding support from the Research Grants Council of Hong Kong under the General Research Fund project 16204720 and the Research Impact Fund project R6003-21.
Pinjia He, Clara Meister, and Zhendong Su. 2020. Structure-invariant testing for machine translation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 961- 973.
References
Boris Beizer. 1995. Black-box testing: techniques for functional testing of software and systems. John Wi- ley & Sons, Inc.
Pinjia He, Clara Meister, and Zhendong Su. 2021. Test- ing machine translation via referential transparency. In 2021 IEEE/ACM 43rd International Conference Software Engineering (ICSE), pages 410-422. IEEE,
Ondfej Bojar, Rajen Chatterjee, Christian Federmann. Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz. Mattco Negri. Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169-214, Copenhagen, Denmark. Association for Computa- tional Linguisties.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Has- san Awadalla. 2023. How good are gpt models at machine translation? comprehensive evaluation. ArXiv preprint, abs/2302.09210.
Guoping Huang, Lemao Liu, Xing Wang, Longyue Wang. Huayang Li, Zhaopeng Tu, Chengyan Huang, and Shuming Shi. 2021. Transmart: A practical in- teractive machine translation system. arXiv preprint arXiv:2105. 10..2110011000 13072.
Chi Chen, Maosong Sun, and Yang Liu. 2021a. Mask- align: Self- supervieed neural word alignment. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4781- 4791, Online. Association for Computational Lin- guistics.
Pin Ji, Yang Feng, Jia Liu, Zhihong Zhao, and Baowen Xu. 2021. Automated testing for machine translation via constituency invariance. In 2021 36th IEEE/ACM International Conference on Automated Software En- gineering (ASE), pages 468 479. IEEE
Guanhua Chen, Yun Chen, and Victor O. K. Li. 202 lb.
Lexically constrained neural machine translation with explicit alignment guidance. In Fhirty-Fitth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Thine Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 12630-12638. AAAI Press.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study, ArXiv preprint,
abs/2301.08744 088445..
Kevin Knight. 2000. Statistical machine translation. In Proceedings of the Fourth Conference the As- sociation for Machine Translation in the Americas: Tutorial Descriptions, Cuernavaca, Mexico. Springer.
Chris Donahue, Mina Lee, and Percy Liang. 2020. En abling language models fill in the blanks. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2492- 2501, Online. Association for Computational Lin- guistics.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Siyu Lai, Zhen Yang. Fandong Meng, Xue Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2022. Generat- ing authentic adversarial examples beyond meaning- preserving with doubly round-trip translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4256 4266, Seattle, United States. Association for Computational Linguistics.
Denis Emelin, Ivan Titov, and Rico Sennrich. 2020. De- tecting word sense disambiguation biases in machine translation for model-agnostic adversarial attacks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7635-7653, Online, Association for Computa- tional Linguistics.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Länguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
and Jie Hao. 2020. OPPO's machine translation sys- tems for WMT20. In Proceedings of the Fifth Confer- ence Machine Translation, pages 282-292, Online. Association for Computational Linguisties.
BART: Denoising
Zeyu Sun, Jie M Zhang, Mark Harman, Mike Papadakis, and Lu Zhang. 2020. Automatic testing and improve- ment of machine translation. In Proceedings of the ACM/TEEE 42nd International Conference on Soft- ware Engineering, pages 974-985.
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu. Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. ArXiv preprint, fos/2303.13780)
Betty Van Aken, Sebastian Herrmann, and Alexander Löser. 2022. What do you see this patient? be- havioral testing of clinical NLP models. In Proceed- ings of the 4th Climical Natural Language Processing Workshop, pages 63-73, Seattle, WA. Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? ArXiv preprint, abs/2302.00676.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008.
Vikas Raunak, Matt Post, and Arul Menezes. 2022. SALTED: A framework for SAlient long-tail transla- tion error detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5163-5179, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Jun Wang. Benjamin Rubinstein, and Trevor Cohn. 2022a. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Anmual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 2576-2590, Dublin, Ireland. Association for Compu- tational Linguistics.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference Machine Translation ( (WMT). pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Jun Wang, Chang Xu, Francisco Guzmán, Ahmed El-Kishky, Benjamin Rubinstein, and Trevor Cohn. 2021. As easy as 1, 2, 3: Behavioural testing of NMT systems for numerical translation. In Find- ings of the Association for Computational Linguis- rics: ACL-IICNLP 2021, pages 4711-4717, Online. Association for Computational Linguisties.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Shuo Wang. Peng Li, Zhixing Tan, Zhaopeng Tu, Maosong Sun, and Yang Liu. 2022b. A template- based method for constrained neural machine trans- lation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3665-3679, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 41-58, Online. Association for Computational Linguistics.
Yanling Xiao, Lemao Liu, Guoping Huang, Qu Cui, Shujian Huang, Shuming Shi, and Jiajun Chen. 2022. BiTIIMT: A bilingual text-infilling method for in- teractive machine translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 1958-1969, Dublin, Ireland. Association for Compu- tational Linguistics.
Danielle Saunders and Bill Byrne 2020. Reducing gen- der bias in neural machine translation as domain adaptation problem. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7724-7736. Online. Association for Computational I Linguistics.
Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. 2021. Crafting adversarial examples for neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing ( (Volume 1:
Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei Dang, Xue Zhengshan,

---

Long Papers), pages 1967-1977, Online. Association for Computational Linguistics.
six MT systems on test cases targeting the Noun, Verb and NER capabilities. Specifically, we set 1): a as 0.5 2.5/0.6/0.7/0.. while fixing 3 0.05; 2): 8 as 00.02/0.05/000 /0.0///0////// 00.00/0000000./ /0.11 while fixing a = 0.8, and list the results in Table 9, Table 10 and Table 11, respectively.
Wanrong Zhu, Zhiting Hu, and Fric Xing. 2019. Text infilling, ArXiv preprint, abs/1901.00155.
A Additional Prompts for ChatGPT
From the three tables, we observe that the val- ues of the two hyperparameters will not affect the conclusions obtained from Table 4:
In this section, we list the prompt templates used for querying ChatGPT to craft test cases target- ing the Tense and NER capability in Table 8 for reference supplement Table 3.
Transformer obtains the worst results.
B Details on Setting Hyperparameter Values
most cases, ChatGPT makes more transla- tion errors compared to the top commercial MT system.
In $4.1, we mentioned that the values of the hyperparameters and /3 0.8 and 0.05, re- spectively. In this section, we provide more details about how these two values are selected.
Deepl. outperforms other commercial MT sys- tems.
MT systems make more errors cases targeting the Verb capability.
Unlike behavioral testing on NLP classification tasks where errors can be directly derived from the prediction outputs, we cannot directly identify testing errors from MT outputs. Therefore, we proposc and β to help identify translation errors from the MT outputs automatically. As shown in Equation 2:
In conclusion, we point out the following points regarding the hyperparameter value setting:
a set to 0.8 since we use COMET as the quality evaluation metric and we regard y as a good translation Qual(y) 2 0.8. 8 is set to 0.05 since we believe that COMET difference lower than 0.05 is reasonable shift caused by the modifications in the test cases.
a controls user's degree of acceptance of MT systems' outputs. If Qual(y) 2 a, we say that the user regards y as a good translation and accepts for further testing. In the paper, a is set to 0.8 since we use COMET as the quality evaluation metric and we regard y as good translation if Qual(y) 2 0.8.
The values of these two hyperparameters will not affect the conclusions obtained from the experiments.
3 controls a user's tolerance on translation er-
C Detailed Human Evaluation Instructions
rors and if Diff(y, y¹) < B3 we say that the
quality of y (the MT system's output the modified test case) is similar to y and the MT system passes the behavioral test. In this pa- per, /3 is set to 0.05 since we believe that a COMET difference lower than 0.05 is a rea-
The detailed instructions of our human evaluation tasks are shown in Table 12.
D Details of the Transformer-based MT System
sonable shift caused by the modifications in the test cases. If Diff(y, y') < 0.05, we re- gard and y have similar translation quality and the MT system passes the behavioral test,
In this section, we provide details of the Transformer-based MT system introduced $4.3, which contains six 512-dimensional encoder and decoder layers, respectively. The word-embedding size is set to 512. Two separate BPE models applied to generate two vocabularies for En- glish (~48K tokens) and Chinese (~59K tokens). We train Transformer using the WMT20 En-Zh news translation corpus that includes 31M sen- tence pairs (Bojar et al., 2017) for 300K steps with 6K warmup steps using the cosine learning rate
Moreover, we would like to illustrate that the value setting of the two hyperparameters will not affect the conclusions drawn from the evaluation results. To demonstrate this point, we perform an experiment by 1): varying the value of while fixing 3; 2): varying the value of 3 while fixing a, and obtaining the corresponding pass rates for the

---

Task/Capability
Prompt Template
Tense
"role": "system", "content": "You are a helpful assistant that translates English to Chinese.", "role": "user", "content"; "You are given an English sentence and its Chinese translation. In each sentence, a verb/verb phrase has been masked with the '<mask>' token. Your task is to first fill in the masked token in the English sentence using a past perfect tense verb/verb phrase without modifying any of the unmasked tokens. Then, use the filled English sentence to fill in the masked token in its corresponding Chinese translation in the past perfect tense. If necessary, make modifications to the filled Chinese translation to ensure the correctness of tense while preserving the meaning. Finally, please output the filled English sentence and its filled Chinese translation in the format of 'Filled English: \n Filled Chinese:", An English Sentence: [MASKED ENGLISH]. An Chinese Translation: [MASKED CHINESE]"
NER
"role": "system", "content": "You are a helpful assistant that translates English to Chinese.", "role": "user", "content": "You are given an English sentence and its Chinese translation. In each sentence, an (named entity type) has been masked with the '<mask>' token. Your task is to first fill in the masked token in the English sentence using an (named entity type) other than (the original named entity) wi withuut modifying any of the unmasked tokens. Then, use the filled English sentence to fill in the masked token in its correspanding Chinese translation. If necessary, make modifications to the filled Chinese translation to ensure fluency while preserving the meaning. Finally, please output the filled English sentence and its filled Chinese translation in the format of 'Filled English: An Filled Chinese:". \n English Sentence: [MASKED ISG].SH]. An Chinese Translation: [MASKED CHINESE]"
Table 8: Prompt templates querying the ChatGPT model. For NER capability, we use Stanza identify named entities in sentence and theenamed entity type" refers one of the 18 named entity types provided by Stanza.
0.5, 8 = 0.05
0.6,
0.7.
0.8,
0.8,
a 0.8,
0.8,
MT System
8 0.05
8 = 0.05
3 = 0.02
8 0.08
8 0.11
8 = 0.05 (Table 4)
Google
94.74
94.33
93.31
73.08
91.09
91.30
88.26
AIS-Translator
94.94
94.74
93.52
73.48
89.27
89.27
87.25
DeepL.
95.95
95.95
95.14
79.15
93.18
93.18
92.11
Tencent
94.53
93.93
91.90
72.67
90.49
91.09
88.87
ChatGPT
90.89
90.89
89.87
64.58
90.69
90.89
87,05
Transformer
85,83
84.21
80.16
$4.45
72.47
73.68
70.24
Avg
92.81
92.34
90.65
69.57
87.86
88.24
85.63
Table 9: Pass rates (%) of behavioral testing targeting the Noun capability under different hyperparameter settings. Under each setting, the best results among MT systems boldfaced. The column refers to corresponding results listed in Table 4.
scheduling strategy. During training, we use the Adam optimizer (31 = 0.9, 32 = 0.98) with a learning rate of le-4. The dropout rate set to 0.3 and the label smoothing rate is set to 0.2. The max- imum number of tokens in a batch is set to 65536. Transformer achieves average BLEU score of 38.66 on the WMT20 En-Zh test set 14.
FoFor reference, the rank 1st system (Shi et al., 2020) on this set achieves an average BLEU score of 43.20.

---

a 0.5,
a 0.6,
a 0.7,
0.8,
a 0.8,
a 0.8,
a 0.8,
MT System
B 0.05
8 0.05
= 0.05
g = 0.02
8 0.08
8 0.11
0.05 (Table 4)
Google
93.62
93.35
92.02
67.02
92.02
92.55
88.83
MS- Translator
89.89
89.89
88.30
66.46
88.56
90.96
83.25
DeepL
94.95
94.95
94.42
69.68
91.76
92.29
89.63
Tencent
90.96
90.43
89.36
67.82
89.10
89.89
84.84
ChatGPT
90.69
90.69
90.16
60.64
92.02
92.82
86.70
Transformer
85.37
82.71
79.26
51.33
71.54
73.94
69.68
Avg
90.91
90.34
88.92
63.83
87.50
88.74
83,82
Table 10: Pass rates (%) of behavioral testing targeting Verb capability under different hyperparameter settings. Under each setting, the best results among all MT systems boldfaced. The last column refers to the corresponding results listed in Table 4.
0.5.
0.6,
0.7,
0.8.
0.8.
0.8.
0.8,
MTT System
8 0.05
0.05
0.05
3 0.02
8 0.08
0.11
0.05 (Table
Google
95.52
95.37
95.07
76.98
91.33
92.08
88.94
ASSTraanlltorr
95.07
94.77
93.57
75.19
89.39
89.99
87.00
Decpl.
95.96
95.96
95.67
77.73
93.57
93.87
91.03
Tencent
94.62
94.62
93.57
76.68
90.28
91.03
88.34
CharGPT
93.27
93.27
92.98
63.68
92.38
93.27
88.34
Transformer
87.44
87.00
83.11
58.45
73.54
74.29
71.30
Avg
93.65
93.50
92.33
71.45
88.42
89.09
85.83
Table 11: Pass rates (%) of behavioral testing targeting the NER capability under different hyperparameter settings. Under each setting, the best results among all MT systems are boldfaced. The last column refers to the corresponding results listed in Table

---

Source sentence fluency: 1/2/3, For each source sentence, you need to determine which description category it belongs to and mark it with the corresponding score. Your need not consider the translation, just consider the source sentence itself.
Score
Description
(unsatisfied)
1) The text is totally broken, or contains severe grammar errors.
2) The text is very hard understand.
(fair)
The text is basically fluent, but contains few grammar errors that do not affect understanding.
2) The text is basically fluent, but contains some repeated context.
The text basically fluent. but contains perverse fake content that obviously different from the commonsense.
(satisfied)
The text is fluent and do not have grammatical errors/repeated context/perverse or fake content. It not hard understand the meaning the sentence.
Translation Fluency: 1/2/3. For cach translation, you need determine which description category belongs and mark it with the corresponding score. You need not consider the source sentence, just consider the translation it self.
Score
Description
(unsatisfied)
The totally broken, or contains severe grammar errors.
2) The text is very hard understand.
(fair)
The text is basically fluent, but contains few grammar errors that not affect understanding.
2) The text is basically fluent, but contains some repeated context.
3) The text basically fluent. contains perverse or content that obviously different from the commonsense.
(satisfied)
1) The text is fluent and do not have grammartical crrors/repeated context/pprverse or fake content. It not hard to understand the meaning of the sentence.
Translation Adequacy: 1/2/3. For each translation, you need to determine which description category it belongs to and mark it with the corresponding score. Please consider the source sentence and the translation together. You need not to consider the fluency of the translation if there is no difficulty in understanding.
Score
Description
(unsatisfied)
1) The translation is irrelevant to the source sentence.
or
2) Many parts of the source sentence are missed in the translation.
2 (fair)
1) Few parts of the source sentence are missed in the translation.
or
2) All parts of the source sentence have heen translated, hut the tonses/syntactic structures of the source sentence and the translation are different.
or
3) Some parts that are not in the source sentences have been translated.
3 (satisficd)
1) All the parts of the source sentence are translated correctly. The tenses and the syntactic structures of both the source sentence and the translation are also the same.
Table 12; Detailed human evaluation instructions.


Reviews:
[{"review_id": "XrCoQYSOkC", "reviewer": "Reviewer_5gEN", "summary": "", "questions": "* Please provide the number of the translation errors for which the precision and the recall in Table 5 was calculated.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments and feedback! We address your concerns as follows:\n\n### Reasons to reject:\n\n> The quality test of the generated test set is only done for the category “general”\n\nWe first want to clarify the relationship of the **General** capability and other capabilities. Our motivation of evaluating the **General** capability is to test MT systems under a more complex and random scenario. Different from test cases targeting other capabilities that only edit one specific type of segment in the source sentence, test cases targeting the **General** capability may edit one or several random segments (mentioned in its \"Description/Test Case Generation Instruction\" column in Table 1) simultaneously and thus leads to more general and diverse testing examples. \n\nNext, based on the definition of the **General** capability, two kinds of test cases could be constructed:\u000b\n\n1. Test cases entailing modification to just one segment. These can also serve as test cases for other distinct capabilities such as **POS**. Importantly, during our experiments, we ensured that there was no overlap between test cases for the **General** capability and other capabilities.\n\n2. Test cases involving modifications to several segments at once, which are not suitable as test cases for the other specific capabilities.\n\nIn addition, we pick out test cases that fulfill the requirement of the **POS** capability (modifying one segment with a certain POS tag)  from the 200 tested pairs used in Section 4.2, and calculate their quality scores separately as follows:\n\n POS (24 test cases)\n|     |  **SSF** | **TSF** | **TA** | **COMET-20** | \n|  ----  | ----  | ---- | ---- | ---- | \n|  BART+BITIMT | 2.83 | 2.86 | 2.64 | 32.42 |\n|  Ref |**2.96**  |**2.95** |2.80 |41.23 |\n|  BTPG |**2.96**  |2.93|**2.86** |**41.36** |\n\nWe can see that the quality evaluation results above are similar to Table 2. Namely, BTPG showcases results in line with Ref on \"SSF\" and \"TSF\" while surpassing both Ref and BART+BITIMT in \"TA\" and \"COMET-20\" evaluations. Therefore, we conclude that test cases targeting the **General** capability are comprehensive and representative and are suitable for the quality evaluation (Section 4.2). \n\nIn conclusion, we conduct quality evaluation on test cases targeting the **General** capability since such test cases are more comprehensive and representative. Moreover, we show that results obtained from test cases targeting the **General** capability are similar to results targeting specific capabilities like **POS**, which further proves our quality evaluation strategy. We will make this point clearer in the revised version.\n\n> ... it is not very clear what we can get from the behavioral test using the generated test cases.\n\nFirst, we want to again emphasize the contribution of this paper, i.e, we conduct behavior testing in MT, which reveals more translation error cases for well-known MT systems (including the Google translate) with less human efforts. To this end, we first propose a novel method that can automatically generate large numbers of test cases from some off-the-shelve MT test sets (also mentioned by Reviewer Qgqn). Then we use this method to conduct behavior testing for six well-known MT systems. Compared to previous fine-grained error analysis works in MT that requires much human efforts on crafting high-quality references, our method alleviate this issue with much cheaper costs. \n\nRegarding the latter contribution, the \"pass rate\" of behavioral testing in Table 3 is crucial for diagnosing MT systems since it directly tells us how many errors MT systems may encounter on a batch of test cases. And these founded errors can lead to some insightful findings (Section 4.3, Section 4.5) just like previous NLP behavioral testing works have done (Ribeiro et al., 2020) [1]. \n\n> Analysis of low error rates on verbs is however just exemplified with a single instance and we do not know how many of the test failures can be explained by it.\n\nWe manually examine all the errors generated by MS-translator and Google targeting the **Verb** capability in Table 3. The percentages of translation errors caused by verbs' collocations with different prepositions are 60.32% and 40.48%, respectively. This result illustrates that this error type does dominate the translation errors targeting the **Verb** capability.\n\n> Section 4.5 contradicts with the interpretation of the “verb” category result in Section 4.3.\n\nIn Section 4.3, we focus on sentence-level translation errors caused by verb modification regardless of the error position, which can be directly identified by the \"pass rate\" as shown in Table 3. Within these identified discrepancies, we can pinpoint two specific translation error types associated with the collocation of the modified verb with prepositions:\n\nType 1. The modified verb is translated incorrectly due to its collocation with prepositions in the test case, making the MT system not passing the behavioral test. \n\nType 2. The modified verb is translated correctly. However, it collocates with a preposition that is originally collocated with another segment $s$ in the test case, thus making the MT system incorrectly translates $s$ and not passing the behavioral test. \n\nTaking the case in Figure 2 as an example. The modified verb *“balancing”* in the test case $x'$ makes DeepL forget to translate *“on the edge of favour”* in $y'$, probably because DeepL pays more attention to the collocation *“balance with”* in $x'$ than the original collocation *\"on the edge of favour with\"* (Line 512-518).\n\nTo dive deeper into the above sentence-level translation errors, in Section 4.5, we investigate how these two kinds of errors distribute in all the founded error cases. Specifically, we question whether an error is predominantly positioned at the modified site (Type 1) or elsewhere (Type 2). This analysis is not contradicting with the analysis in Section 4.3. We will make this point clearer in the revised version. \n\n### Questions for the authors\n\nWhen calculating erroneous translations in Table 5 (Line 548-555), we experiment on 135 test cases constructed from the 100 source sentences.\n\nThe number of translation errors founded by Dep and BTPGBT are 75 and 46, respectively. Among these errors, 26 and 38 are regarded as true by humans, respectively. Therefore, the precision values of Dep and BTPGBT are 26/75 $\\approx$ 34.67% and 38/48 $\\approx$ 82.61%, respectively. \n\nNext, according to human evaluation, the Google translation system makes 53 actual translation errors on all the test cases. Therefore, the recall values of Dep and BTPGBT are 26/53 $\\approx$ 49.06% and 38/53 $\\approx$ 71.70%, respectively.\n\n> Typos\n\nThanks for the suggestions. We will revise these typos in the revised version.\n\n### References in the rebuttal\n\n[1] Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. arXiv preprint arXiv:2005.04118.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is clear.\nThe generated test set is shown to be of good quality.", "reasons_to_reject": "The quality test of the generated test set is only done for the category “general” while, as far as I understand, one of the advantage of the behavioral test is that it can test a certain aspect of the systems’ capability by comparing the result across different test set categories. The author explains the reason in footnote 8 but I could not get the meaning of “more general evaluation result.” Since there is no guarantee that the quality of the “general” category is at the same level of the other, more targeted, categories, it appears to be a weak point of the paper.\n\nAnother concern I had about the paper is that it is not very clear what we can get from the behavioral test using the generated test cases. The author does not draw much information from the evaluation results of the six MT systems (Section 4.3). It is suggested that the relatively low “pass rate” on verb category is due to the fact that the accurate translation of a verb often requires to take into account the collocation with prepositions. It is however just exemplified with a single instance and we do not know how many of the test failures can be explained by it. Furthermore, in Section 4.5, the author says most of the translation failure identified by their method is **not** on the modified position of the test cases (e.g., the replaced verbs in the “verb” test cases). Isn’t it contradict with the interpretation of the “verb” category result in Section 4.3?", "questions_to_authors": "* Please provide the number of the translation errors for which the precision and the recall in Table 5 was calculated.", "paper_topic_and_main_contributions": "This paper presents a method for creating a test set for behavioral testing of machine translation systems.\nThe test set is produced by first masking a pair of corresponding words/phrases in a translation pair and then filling them by using ChatGPT.\nQuality evaluation of the produced test set revealed that the rewritten translation pairs are as good as their original translation pairs.\nSix MT systems are compared using the automatically generated test set.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L210: follow the steps -> follow**ing** the steps ?\n\nL255: between x and r -> ??\n\nL406: (?)  [bibtex entry is missing?]\n\nTable 3, header: Adp -> Adj ?\n"}, "extra_text": {}}, {"review_id": "zTDajz4zhE", "reviewer": "Reviewer_59vL", "summary": "", "questions": "Question A: When determining modification positions, as you always choose the longest segment when there are overlaps, doesn’t it prevent you from selecting single-word segments?\nQuestion B: How exactly do you generate multiple test cases from the same parallel sentence? What happens if there is only one possible phrase pair that can be masked?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments and feedback! We address your concerns as follows:\n\n### Reasons to reject\n\n> The proposed algorithm is difficult to reproduce. \n\nWe promise to make all our code, generated test cases, and translation outputs from the six MT systems publicly available to ensure complete reproducibility of our experiments. For the issues regarding reproducing the algorithm, we will answer them in the following.\n\n> Neither in section 3.1 nor in the rest of the paper, it is clearly stated how to proceed when more than one non-overlapping segment can be extracted from a sentence.\n\nWe have provided the relevant details in the \"Test Case Generation Instruction\" column of Table 1 (Lines 328-330) . For the **POS**, **Tense**, and **NER** capabilities, only a segment containing a word with a POS tag/verb/named entity in the source sentence will be modified, irrespective of how many segments are derived from this sentence. However, for the **General** capability, if there are multiple segments extracted from the source sentence, more than one of them can be modified.\n\n> The ChatGPT prompts, that are one of the main components of the algorithm, are not shown in the main paper.\n\nWe put all the prompt templates in the appendix (Table 7) and will put them in the main paper in the revised version.\n\n> In section 3.3, it is not clear how Diff(r, r’) is computed\n\nSince the proposed BTPG method enables us to follow the principle definition to conduct behavioral testing (Section 2.1), we use equation (1) to calculate $Diff(r,r')$. For this calculation, we employ the reference-free variant of COMET to determine $Qual(r)$ and $Qual(r')$ (refer to Line 336-344 and 397-402). We will make the distinction clearer in the revised version.\n\n> ... how the number of test cases generated from the same source sentence can be increased (it is mentioned in Section 4.3), as no mention is made about how the prompts should be changed.\n\nWhen generating multiple test cases from the same source sentence, the change of the prompt is only on the target segment that will be modified (mentioned in Table 7). Since we can extract multiple candidate segments from the same source sentence, we just need to use different segment as the target segment and repeat the generation process to obtain multiple test cases, as described in line 256-257, line 293-295 and line 457-460.\n\n> Regarding the experiments on Large-scale behavioral testing, there are not enough details about how the 20540 tests were obtained, which were the original sentences and how many segments were extracted from each parallel sentence.\n\nWe use the dataset *WMT* to generate the 20540 test cases targeting the **General** capability (Line 355-360). As described in Line 243-257, all the aligned segments that fulfill our requirements are extracted from each parallel sentence. \n\nWe also provide more statistics of the data and the segments. Overall, there are 6862 source sentences and their references in *WMT*, where 41240 segments can be extracted in total (6.01 for each parallel sentence in average). Moreover, considering the combinations between segments,we can construct 28484862025 masked pairs ($\\hat{x}$, $\\hat{r}$) targeting the **General** capability for further test case generation. ChatGPT's diverse blank-filling behavior also allows us to craft a vast number of unique test cases with ease.\n\n>  The analyses carried out using the proposed algorithm are not really informative since the translation errors may not happens at the modified position.\n\nFirst, we want to again emphasize the contribution of this paper, i.e, **we conduct behavior testing in machine translation, which reveals more translation error cases for well-known MT systems (including the Google translate) with less human efforts**. \n\nTo this end, we first propose a novel method that can automatically generate large numbers of test cases from some off-the-shelve MT test sets (also mentioned by Reviewer Qgqn). Then we use this method to conduct behavior testing for six well-known MT systems. Compared to previous fine-grained error analysis works in MT that requires much human efforts on crafting high-quality references, our method alleviate this issue with much cheaper costs.\n\nRegarding the latter contribution (Section 4.3, 4.4, 4.5), the \"pass rate\" in Section 4.3 is crucial for diagnosing MT systems since it directly tells us what and how many errors MT systems may encounter on a batch of test cases targeting various capabilities, then lead to insightful findings (Section 4.3, Section 4.5) just like previous NLP behavioral testing works have done (Ribeiro et al., 2020) [1]. Specifically, two main types of translation errors are identified in our experiments:\n\n**Type 1**.  The modified segment is translated incorrectly. \n\n**Type 2**.  The modified segment is translated correctly. However, it makes the MT system incorrectly translates another segment and not passing the behavioral test. \n\nTo dive deeper into the above sentence-level translation errors, in Section 4.5, we investigate how these two kinds of errors distribute in all the founded error cases, i.e., whether a translation error occurs at the modified position (Type 1) or at other positions (Type 2) and draw several conclusions in Section 4.4 and Section 4.5 and obtain some insightful findings from our behavioral testing results.\n\n> ChatGPT should not be evaluated since its translation quality could be over-estimated.\n\nFirst, we'd like to clarify that **the focus of this paper is to evaluate existing MT systems, including commercial MT systems and strong open-sourced MT systems**. Since these MT systems are not incorporated in the test case generation process, their evaluation results are not biased. \n\nFor ChatGPT, we admit that its evaluation results might be over-estimated. However, as shown in Table 3, ChatGPT's pass rates are significantly lower than the best commercial MT system, which is in line with the conclusions in existing works that ChatGPT performs worse than the strongest commercial MT systems [2] [3]. Such observations affirm that the over-estimate concern linked to ChatGPT isn't substantial, further showing the robustness of our proposed method and we put ChatGPT's results in our paper to make our evaluation more comprehensive.\n\n### Questions for the authors\n\n### Question A\nGiven that a sentence typically comprises more single words than phrases, numerous single-word segments remain in the candidate list, even when we consistently opt for the longest segment in cases of overlap. For reference, of the 41240 segments derived from *WMT*, 35302 are single-word segments, while 5938 are phrasal segments.\n\n### Question B\n\nIn our experiments, if there is only one possible segment that can be masked in the source sentence, we only generate one test case with this source sentence. However, given the diversity of ChatGPT's outputs, we can easily generate multiple test cases by querying ChatGPT for multiple times, while making sure that there is no overlap in the crafted test cases. \n\nWhen a source sentence has multiple segments, as mentioned above, we use different segment as the target segment and modify the prompt of ChatGPT correspondingly, then repeat the generation process to obtain multiple test cases (line 256-257, line 293-295 and line 457-460).\n\n### Typos\nThanks for your suggestion. We will revise these issues in the revised version.\n\n### References in the rebuttal\n\n[1] Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. arXiv preprint arXiv:2005.04118.\n\n[2] Jiao, W., Wang, W., Huang, J. T., Wang, X., & Tu, Z. (2023). Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745.\n\n[3] Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., ... & Awadalla, H. H. (2023). How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presented algorithm can help to easily detect errors in MT systems without additional parallel data", "reasons_to_reject": "The proposed algorithm is difficult to reproduce. Neither in section 3.1 nor in the rest of the paper, it is clearly stated how to proceed when more than one non-overlapping segment can be extracted from a sentence. The ChatGPT prompts, that are one of the main components of the algorithm, are not shown in the main paper. In section 3.3, it is not clear how Diff(r, r’) is computed, since later it is stated that reference-based COMET is used as the evaluation metric. It is also not clear how the number of test cases generated from the same source sentence can be increased (it is mentioned in Section 4.3), as no mention is made about how the prompts should be changed. Regarding the experiments on Large-scale behavioral testing, there are not enough details about how the 20540 tests were obtained. It is not clear which were the original sentences and how many segments were extracted from each parallel sentence.\n\nThe analyses carried out using the proposed algorithm are not really informative when it comes to drawing conclusions about MT systems (Sec. 4.3). They represent how often a sentence built by modifying a phrase that contains a word with a particular POS is translated “correctly”. Firstly, the error may come from another word from the phrase and, secondly, according to Table 6, the errors usually happen in parts of the target sentence that have not been modified during the creation of the test case.\n\nConcerning technical soundness, chatGPT should not be used as an MT system being evaluated since it has been used to generate the test cases, thus its translation quality could be overestimated.", "questions_to_authors": "Question A: When determining modification positions, as you always choose the longest segment when there are overlaps, doesn’t it prevent you from selecting single-word segments?\nQuestion B: How exactly do you generate multiple test cases from the same parallel sentence? What happens if there is only one possible phrase pair that can be masked?", "paper_topic_and_main_contributions": "The paper presents a method to automatically create behavioral tests for MT from an existing parallel corpus. The method involves word-aligning the parallel sentences, selecting parallel phrases (similarly to what was done in SMT, but selecting only syntactic phrases) and replacing them, both in source and target, with likely segments in that context by means of chatGPT. A test is passed if the automatic evaluation metric score is similar to that of the original parallel sentence, as long as the original parallel sentences obtained a good enough score. A human evaluation showed that the generated sentence pairs are adequate and fluent.\n\nThe main contributions of this work are the following:\n- The algorithm itself, that can be used to create tests for MT systems and automatically detect specific errors in MT systems, going beyond the usual automatic evaluation metrics\n- The validation, by means of human evaluation, of the capabilities of large language models (chatGPT) for mask infilling in a multilingual way", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please avoid including information that is crucial to understand the paper in Appendixes, such as the chatGPT prompts, related works and the training data the Transformer system is trained on."}, "extra_text": {}}, {"review_id": "WgKB4eUVVs", "reviewer": "Reviewer_Qgqn", "summary": "", "questions": "* On what basis did you choose 0.8 and 0.05 for the hyperparameters alpha and beta in your experiments? This is crucial since the pass/fail percentages for the systems tested are heavily influenced by these parameters. I am willing to increase my score if the authors can convincingly describe how the parameters were chosen and how they could be optimised if someone were to reproduce this work (e.g., with other languages or MT evaluation metrics).\n* When did you obtain the ChatGPT outputs? Assuming you used it via OpenAI's API, which model did you use exactly?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments and feedback! We address your concerns as follows:\n\n### Reasons to reject\n\n> ChatGPT is to be expected to work with other high-resource, but not (or not as good) with low(er)-resource language pairs.\n\nWhile we agree that ChatGPT's direct translation results on low-resource languages may not be as good as those on high-resource languages and have incorporated this point in the limitation section, recent works have proposed several ways to remedy this issue. For example, Lu et al. (2023) [1] propose to incorporate the knowledge of external multilingual dictionaries to design prompts for ChatGPT on low-resource translation task, which largely enhances ChatGPT's performances on such tasks. In future works, we're considering integrating such advancements into our methodology to enhance its capability for low-resource language translation tasks.\n\n> On what basis did you choose 0.8 and 0.05 for the hyperparameters alpha and beta in your experiments? And how they could be optimised if someone were to reproduce this work (e.g., with other languages or MT evaluation metrics)\n\nFirst of all, we'd like to emphasize that the exact values of these two hyperparameters are derived from the user's tolerance level towards translation errors. Therefore, we choose values for $\\alpha$ and $\\beta$ based on our tolerance towards translation errors without tuning. We will detail this point in the following.\n\nWe then explain the meaning of these two hyperparameters (Line 149-157) and why we select 0.8 and 0.05 for them. Unlike behavioral testing on NLP classification tasks where errors can be directly derived from the prediction outputs, we cannot directly identify testing errors from MT outputs. Therefore, we propose $\\alpha$ and $\\beta$ to help identify translation errors from the MT outputs automatically. \n\nAs shown in Equation 2, \n\n* $\\alpha$ controls an user's degree of acceptance of MT systems' outputs. If $Qual(y) \\geq \\alpha$, we say that the user regard $y$ as a good translation and accept it for further testing. **In the paper, $\\alpha$ is set to 0.8 since we use COMET as the quality evaluation metric and we regard $y$ as a good translation if $Qual (y) \\geq 0.8$**. \n\n* $\\beta$ controls an user's tolerance on translation errors and if $Diff(y, y') \\leq \\beta$, we say that the quality of $y'$ (the MT system's output on the modified test case $x'$) is similar to $y$ and the MT system passes the behavioral test. **In this paper, $\\beta$ is set to 0.05 since we believe that an COMET difference lower than 0.05 is a reasonable shift caused by the modifications in the test cases**. If $Diff(y, y') \\leq \\beta$, we regard $y$ and $y'$ have similar translation qualities and the MT system passes the behavioral test. \n\nMoreover, we'd like to illustrate that the value setting of the two hyperparameters will not affect the conclusions draw from the evaluation results, and perform an additional experiment to prove this point. Specifically, we vary the value of $\\alpha$ (while fixing $\\beta$) / vary the value of $\\beta$ (while fixing $\\alpha$) and obtain the corresponding pass rates for the six MT systems on test cases targeting the **Noun**, **Verb** and **NER** capabilities. We set $\\alpha$ as 0.5/0.6/0.7/0.8/ (while fixing $\\beta=0.05$) / $\\beta$ as 0.02/0.05/0.08/0.11 (while fixing $\\alpha=0.8$) and list the results as follows (values in the last column are the results in Table 3):\n\n **Noun**\n|   MT System     | $\\alpha=0.5,\\beta=0.05$ | $\\alpha=0.6,\\beta=0.05$ | $\\alpha=0.7,\\beta=0.05$ |$\\alpha=0.8,\\beta=0.02$ | $\\alpha=0.8,\\beta=0.08$ | $\\alpha=0.8,\\beta=0.11$ | $\\alpha=0.8,\\beta=0.05$ (**our setting**) |\n|--------|  :----------:  |  :----------: |  :----------: |  :----------: |  :----------: |  :----------: |  :----------: |\n| Google | 94.74    | 94.33    | 93.31    |  73.08    | 91.09    | 91.30    | 88.26    |\n| MS-Translator     | 94.94    | 94.74    | 93.52    |  73.48    | 89.27    | 89.27    | 87.25    |\n| DeepL  | **95.95**    | **95.95**    | **95.14**    | **79.15**    | **93.18**    | **93.18**    | **92.11**    |\n| Tencent| 94.53    | 93.93    | 91.90    |  72.67    | 90.49    | 91.09    | 88.87    |\n| ChatGPT   | 90.89    | 90.89    | 89.87    | 64.58    | 90.69    | 90.89    | 87.05    |\n| Transformer  | 85.83    | 84.21    | 80.16    |  54.45    | 72.47    | 73.68    | 70.24    |\n\n **Verb**\n|   MT System     | $\\alpha=0.5, \\beta=0.05$ | $\\alpha=0.6, \\beta=0.05$ | $\\alpha=0.7, \\beta=0.05$ | $\\alpha=0.8, \\beta=0.02$ | $\\alpha=0.8, \\beta=0.08$ | $\\alpha=0.8, \\beta=0.11$ | $\\alpha=0.8, \\beta=0.05$ (**our setting**)  |\n|--------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|\n| Google | 93.62                     | 93.35                    | 92.02                    | 67.02                    | **92.02**                    | 92.55                    | 88.83                    |\n| MS-Translator     | 89.89                     | 89.89                    | 88.30                    | 66.46                    | 88.56                    | 90.96                    | 83.25                    |\n| DeepL  | **94.95**                     | **94.95**                    | **94.42**                    | **69.68**                    | 91.76                    | 92.29                    | **89.63**                    |\n| Tencent| 90.96                     | 90.43                    | 89.36                    | 67.82                    | 89.10                    | 89.89                    | 84.84                    |\n| ChatGPT   | 90.69                     | 90.69                    | 90.16                    | 60.64                    | **92.02**                    | **92.82**                    | 86.70                    |\n| Transformer  | 85.37                     | 82.71                    | 79.26                    | 51.33                    | 71.54                    | 73.94                    | 69.68                    |\n\n **NER**\n\n|        | $\\alpha=0.5, \\beta=0.05$ | $\\alpha=0.6, \\beta=0.05$ | $\\alpha=0.7, \\beta=0.05$ | $\\alpha=0.8, \\beta=0.02$ | $\\alpha=0.8, \\beta=0.08$ | $\\alpha=0.8, \\beta=0.11$ | $\\alpha=0.8, \\beta=0.05$ (**our setting**) |\n|--------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|\n| Google | 95.52                     | 95.37                    | 95.07                    | 76.98                    | 91.33                    | 92.08                    | 88.94                    |\n| MS-Translator     | 95.07                     | 94.77                    | 93.57                    | 75.19                    | 89.39                    | 89.99                    | 87.00                    |\n| DeepL  | **95.96**                     | **95.96**                    | **95.67**                    | **77.73**                    | **93.57**                    | **93.87**                    | **91.03**                    |\n| Tencent| 94.62                     | 94.62                    | 93.57                    | 76.68                    | 90.28                    | 91.03                    | 88.34                    |\n| ChatGPT   | 93.27                     | 93.27                    | 92.98                    | 63.68                    | 92.38                    | 93.27                    | 88.34                    |\n| Transformer  | 87.44                     | 87.00                    | 83.11                    | 58.45                    | 73.54                    | 74.29                    | 71.30                    |\n\nFrom the above results, we observe that the values of the two hyperparameters will not affect the conclusions in our paper:\n1. Transformer obtains the worst results.\n2. In most cases, ChatGPT makes more translation errors compared to the top commercial MT system.\n3. DeepL outperforms other commercial MT systems.\n4. MT systems make more errors on test cases targeting the **Verb** capability.\n\n**In conclusion, we point out that**:\n\n1. The exact values of these two hyperparameters are selected based on the degree of users' tolerance towards translation errors.\n\n2. $\\alpha$ is set to 0.8 since we use COMET as the quality evaluation metric and we regard $y$ as a good translation if $Qual (y) \\geq 0.8$. $\\beta$ is set to 0.05 since we believe that an COMET difference lower than 0.05 is a reasonable shift caused by the modifications in the test cases. We do not tune these two hyperparameters in our work.\n\n3. The values of these two hyperparameters will not affect the conclusions in our paper.\n\n**When reproducing our method with a new metric or on a new task, we recommend users to**:\n\n1. Know what value of this metric represents good translation on a certain task, and what value represents large difference between translation outputs.\n\n2. Understand your own degree of tolerance towards translation errors, and transfer it into hyperparameter values.\n\n3. Make sure that different values of $\\alpha$ and $\\beta$ will not affect the conclusions obtained from the evaluation results.\n\n> The proposed method relies heavily on ChatGPT, which makes the results presented in the paper hard to reproduce.\n\nWe promise to open-source all the codes, all the generated test cases, all the translation outputs generated by the six MT systems for exactly reproducing our experiments. When reproducing our work with other translation tasks, our proposed method can be easily adapted and thanks for your appreciation on this point.\n\n### Questions for the authors\n\n### Q1:\nPlease refer to our second parts of reply above.\n\n### Q2:\n\nWe obtained ChatGPT 's outputs in April and May, 2023. The model we used was Open AI's \"gpt-3.5-turbo\" (it seems that it is now called \"gpt-3.5-turbo-0613\").\n\n### Typos\n\nThanks for the suggestions. We will revise these typos in the revised version.\n\n\n### References in the rebuttal\n\n[1] Lu, H., Huang, H., Zhang, D., Yang, H., Lam, W., & Wei, F. (2023). Chain-of-Dictionary Prompting Elicits Translation in Large Language Models. arXiv preprint arXiv:2305.06575.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The authors convincingly show that their method enables the generation of valid test cases through human evaluation in terms of source sentence fluency, pseudo-reference fluency, and translation adequacy between the two.\n* The proposed method scales well. While previous work in automatic test case construction has focussed primarily on simple modifications (such as numbers), the method proposed here can, in theory, create an infinite number of test cases by varying any linguistic feature.", "reasons_to_reject": "* The authors experiment with a single language direction (English to Chinese). Since ChatGPT is a seminal component of the proposed pipeline, it is to be expected that the approach will work with other high-resource, but not (or not as good) with low(er)-resource language pairs.\n* The hyperparameters alpha and beta are crucial since they determine whether or not a constructed test case fails. The authors present the percentage of successful test cases for six commercial MT systems, but these numbers all depend on the aforementioned hyperparameters. The paper contains no indication of why the values of 0.8 and 0.05 are used and how they were derived/optimised.\n* The proposed method relies heavily on ChatGPT, which makes the results presented in the paper hard to reproduce. (The method presented in the paper as such, however, could be easily reproduced.)", "questions_to_authors": "* On what basis did you choose 0.8 and 0.05 for the hyperparameters alpha and beta in your experiments? This is crucial since the pass/fail percentages for the systems tested are heavily influenced by these parameters. I am willing to increase my score if the authors can convincingly describe how the parameters were chosen and how they could be optimised if someone were to reproduce this work (e.g., with other languages or MT evaluation metrics).\n* When did you obtain the ChatGPT outputs? Assuming you used it via OpenAI's API, which model did you use exactly?", "paper_topic_and_main_contributions": "This paper proposes and evaluates a method to generate block box test cases for machine translation (MT) systems. Given an off-the-shelve test set (the authors use the WMT21/22 news translation test sets), test cases are generated by modifying specific features of segments in the test set by, for example, replacing a named entity in both the source segment and its reference translation. The replacement is based on a number of open-source tools (Mask-Align, Stanza) as well as a closed chat-optimised LLM (ChatGPT). An MT system passes a test case if an automatic MT quality evaluation metric (COMET in this paper) (1) assigns a ‘good’ score to the system's translation of the original source segment and (2) the difference between the score for the translation of the original source segment and the modified source segment is ‘small’. ‘good’ and ‘small’ are defined as hyperparameters alpha and beta in the paper.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Remove bold formatting of ‘2’ in Table 2.\n* L457: ‘Noting’ should be ‘Note’."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a behavioural evaluation framework for MT. The method first masks a pair of aligned words or phrases in the source and the reference translation and fills them in with ChatGPT. The evaluation test: the model passes the test if (i) the original translation quality is good, (ii) the translation quality of the new pair is similar to the old quality. For English and Chinese, the authors show that the new sentences, i.e. the ones used for evaluation, are of good quality. The authors test six MT systems.

The reviewers feel rather ambivalent about the paper. While they acknowledged that the test set could help finding MT errors without additional parallel data, they all had the same concerns. Firstly, it is not clear what to take out of the evaluation results: (i) often the models do not pass the tests because of an error outside of the modified span, (ii) many words/phrases cannot be translated in isolation (e.g., verbs often require collocations with prepositions). Secondly, the core of the method is using ChatGPT to translate: while this might work for the single language pair considered in the paper (English-Chinese), it is largely not applicable for other languages. While the authors replied very extensively and were extremely proactive during discussion, I believe these concerns are (i) valid and (ii) not addressed (e.g., mentioning the use of ChatGPT in the limitation section does not revoke the concern).
Decision: Accept-Findings