Ask Language Model to Clean Your Noisy Translation Data
Authors: Quinten Bolding
Baohao Liao
Brandon James Denis
Jun Luo
Christof Monz
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Translation
Large Language Models
Data Generation
Noise
TL;DR: Apply LLMs to clean noisy translation dataset with few-shot prompting.

Abstract: TTransformer models have demonstrated remarkable performance in neural machine translation (NMT). However, their vulnerability to noisy input poses a significant challenge in practical implementation, where generating clean output from noisy input is crucial. The MTNT dataset is widely used as a benchmark for evaluating the robustness of NMT models against noisy input. Nevertheless, its utility is limited due to the presence of noise in both the source and target sentences. To address this limitation, we focus on cleaning the noise from the target sentences in MTNT, making it more suitable as a benchmark for noise evaluation. Leveraging the capabilities of large language models (LLMs), we observe their impressive abilities in noise removal. For example, they can remove emojis while considering their semantic meaning. Additionally, we show that LLM can effectively rephrase slang, jargon, and profanities. The resulting datasets, called C-MTNT, exhibit significantly less noise in the target sentences while preserving the semantic integrity of the original sentences. Our human and GPT-4 evaluations also lead to a consistent conclusion that LLM performs well on this task. Lastly, experiments on C-MTNT showcased its effectiveness in evaluating the robustness of NMT models, highlighting the potential of advanced language models for data cleaning and emphasizing C-MTNT as a valuable resource.

Ask Language Model to Clean Your Noisy Translation Data
Quinten Bolding
Baohao Liao
Brandon James Denis R Jun Luo Christof Monz
Language Technology Lab, University of Amsterdam
Huawei Amsterdam Research Center
quinten bolding gmail. com com
b b.liao@uva.nl n1
Abstract
models exhibit noticeable decline in translation quality when exposed to noisy input. This ham- pers their performance real-world scenarios, where human users unintentionally introduce mis- spellings and grammatical errors during text input (Karpukhin et al., 2019). Therefore, evaluating the robustness of NMT models against noisy inputs becomes crucial before their deployment.
Transformer models have demonstrated remark- able performance in neural machine translation (NMT). However, their vulnerability noisy input poses significant challenge in practical implementation, where generating clean output from noisy input is crucial. The MTNT dataset (Michel and Neubig, 2018) is widely used as a benchmark for evaluating the robustness of NMT models against noisy input. Neverthe- less, its utility is limited due to the presence of noise in both the source and target sentences. To address this limitation, we focus cleaning the noise from the target sentences in MTNT, making more suitable as benchmark for noise evaluation. Leveraging the capabilities large language models (LLMs), we observe their impressive abilities in noise removal. For example, they can remove emojis while consid- ering their semantic meaning. Additionally, we show that LLM can effectively rephrase slang, jargon, and profanities. The resulting datasets, called C-MTNT, exhibit significantly less noise in the target sentences while preserving the se- mantic integrity the original sentences. Our human and GPT-4 evaluations also lead consistent conclusion that LLM performs well on this task. Lastly, experiments on C-MTNT showcased its effectiveness in evaluating the robustness of NMT models, highlighting the potential of advanced language models for data cleaning and emphasizing C-MTNT valu- able resource.'
Despite the importance of assessing the silience of NMT models against noise, the available evaluation datasets remain limited. To the best of our knowledge, MTNT (Michel and Neubig, 2018) stands as one of the few well-established resources for evaluating NMT models' performance the presence of noise. The noise distribution in MINT closely resembles real-world use cases, but its ap- plicability is constrained by the presence of noise in the target sentences. For instance, French- English pair may appear as: "REEEEEFEEEECH les normies sur mon eiffel y'en a marre" + "REEEEEEE bored of the normies on my eiffel", which is not desirable. Our expectation is that an effective NMT model capable of translating a noisy source sentence into a clean target sentence.
In order to enhance the applicability of MINT for evaluating NMT models, we propose clean- ing the target side of this dataset. In contrast to conventional cleaning approaches (Xu and Koehn, 2017; Khayrallah et al., 2018; Kochn al., 2020) that typically involve filtering out undesirable sen- tences and retaining only high-quality ones, we aim to remove noise from sentences without reducing the overall sample number. Language-aware rule- based approaches, which rely on predefined rules to eliminate noise, have been widely employed common method for such cleaning (Miao et al., 2021; Mavrogiorgos et al., 2022). While these methods can effectively remove certain types of noise, it becomes impractical define rules for every possible noise source. Moreover, some natu- ral noise introduced by human input is not easily
1 Introduction
Neural machine translation (NMT) has witnessed significant progress (Bapna al., 2018; Hieber et al., 2020; Liao et al., 2021) ) in recent years, particularly with the introduction Transformer (Vaswani et al., 2017). Despite their impressive performance on clean benchmarks (Barrault et al., 2019; Huang et al., 2020; Liao et al., 2020), these
Work done while doing this master thesis Huawei. Up-to-date version at https://ariiv.orr/bs//2211 13469.

---

identified by rule-based approaches alone.
Noise Tye
Esmppl
Spelling byperrppicall croee
scnne "aanos"" ""recaee" "Tacooe"
Another highly promising approach involves leveraging large language models (LLMs) (Tou- vron et al., 2023a; Chen et al., 2023). Pre- vious works have already demonstrated the ef- fectiveness of LLMs in various tasks, including Q& A(Robinson et al., 2022), text summarization (Pilault et al., 2020), and data generation (Meng et al., 2022: Chung al., 2023; Tang al., 2023). However, applying an LLM to clean the target sen- tences poses several challenges that need to be ad- dressed diligently:
Weed untissoe insetiioll repciions
"Ineee"
"Toovs nover"
Ommmuiccal orvess
"s "o
Spizen langaage
"wart 10" "Wonzz", "going to"
"potnz"
Inkenet alass
"Io bu bauol" - "Ild", "Slaking heal" "sall" "Beddi" "reddl"
Coptullaaion
Explidh Scottisk
swinding
loweii"
Aargon
Reddie "upwone"
"ab", "gIM""
Profaniiechlar, (ecmutimos maskrd)
"Plck"
Table I: Noise types and examples from MTNT.
Furthermore, our research highlights another re- markable potential of LLMs in generating high- quality parallel data with limited monolingual re- sources. This finding has significant implications for low-resource domains and languages, where acquiring parallel corpora is often challenging.
Comprehensive noise removal: LLMs should be capable of thoroughly cleaning the target sentence by eliminating all forms of noise, in- cluding removing semantically meaningless emojis, translating emojis with semantic con- tent into words, correcting misspellings, etc.
After the cleaning, we conduct human and GPT- (OpenAI, 2023) evaluations. Both evaluations draw the same conclusion that LLM has a great capability for such cleaning task, especially with method (1).
Semantic preservation: cleaned target sen- tence should retain similar semantic informa- tion as the original noisy target sentence.
In the end, we conduct comprehensive experi- ments train NMT models on some noisy training datasets, and evaluate them on different evaluation sets, including clean benchmarks and the bench- mark constructed with a rule-based noise removal method. C-MTNT consistently demonstrates better noise evaluation manner on the NMT models.
Alignment with the source sentence: The cleaned target sentence should convey the same intended meaning as the noisy source sentence, ensuring accurate translation and faithful representation of the original content.
To the best of our knowledge, this is the first study apply LLMs the context of cleaning data, specifically addressing the challenges out- lined above. Our findings highlight the potential of leveraging advanced language models for data- cleaning tasks and emphasize C-MTNT as a valu- able resource for evaluating the NMT model in real-world scenarios.
In this paper, we propose apply GPT-3.5 (Ope- nAI, 2021) to clean the noisy target sentences the MTNT dataset. Our approach addresses the chal- lenges noise removal, semantic preservation, and alignment with the source sentence. Inspired by the success of prompting methods (Brown et al., 2020; Wei et al., 2022; Schick et al., 2023), we design few-shot prompts guide GPT-3.5 to clean the noisy target sentences in three ways: (1) Utilizing information from both the source and target side for cleaning, (2) Cleaning the target sentence indepen- dently, and (3) Generating a clean target sentence by translating the noisy source sentence. Through a comprehensive analysis, we demonstrate the ef- fectiveness of our cleaning methods, particularly with methods (1) and (3). These methods success- fully remove emojis, while considering their seman- tic meaning, and also rephrase slang, jargon, and profanities appropriately. The resulting datasets, named C-MTNT, exhibit significantly reduced lev- cls noise compared to the original sentences, while still preserving their semantic integrity.
Data Generation and Cleaning
The primary objective of this study to harness the capabilities of LLMs in effectively removing noise and generating parallel language datasets to eval- uate the robustness of NMT models against noisy input. In this section, we will give overview of our data source MTNT and delineate our approach to LLM interaction.
2.1 MTNT Dataset
MTNT (Michel and Neubig, 2018), which incor- porates noisy comments gathered from Reddit alongside professionally sourced translations, has emerged as a benchmark for assessing the perfor- mance of NMT models when exposed to noisy

---

input. This dataset was developed in response to the scarcity of publicly available parallel corpora containing naturally occurring noise. encom- passes three distinct languages, namely English, French, and Japanese, and offers parallel data for two language pairs: English-French and English- Japanese. As shown in Table 1, we present com- prehensive analysis of noisy types within MTNT. Notably, these noise types manifest in both source and target sentences, which not expected since we want evaluate the ability of an NMT model to translate a noisy source sentence into a clean target sentence. Consequently, here we devised an approach aimed at cleaning the target sentences in MTNT to enhance its assessment capability.
Context: You'll seceive sentence Reddit, social and discussion webeite. cantance may centain various types of noise like spelling errors, slang, emojis, and profanities.
Billngual Task: You are givan input sentences: noise translated (tgt) version also containing noise, your task to eliminate noise such weird characters, inzorrect capitelization, emoji's, and spelling/gremetiaal errore from the (tgt) sentence. Raturn the cleaned [tgt] sentence as the output.
(era) sentence
Output Format: desired output format
clean (tgt] sentence is arclosed in quotation represented by following regular expression: ("{3+))
examples inputs their desired outpute, along with the reasoning Eoz each paig:
Example: Here
2.2 Approach
Our approach to cleaning MINT entails meticu- lous consideration of various available settings and resources. It includes the assessment of its effec- tiveness in scenarios where bilingual resources are accessible, as well as the investigation of its feasi- bility in cases where only source or target data is available. To explore the capabilities of LLM, we incorporate three methods specifically tailored to different data scenarios, accounting for the varying availability of language resources.
Inputé: "I'n scc000 happyryy 82, "Je suis tellement heureuxixx!! !!
Desired output: "Je su Le" Tement heureux!"
Reasoning: The notay French sentence contains
word "heuzeux", emoti and unnecessary exclamation mark. The clean Prench sentence renoves the extra letters, emoji and the exclanation mark to convey the sene nessage accurately.
Request: These are the inpute (arc (erc_sent), [tgt_ sent). Flease return the desired output in the correct format.
Bilingual cleaning:
This approach involves providing both noisy source and target sam- ples as input, with the focus on cleaning the target sample while preserving alignment with the source sample.
Figure Prompt design for bilingual cleaning. In prac- tice, the full prompt includes four examples. (sre) and [tgt) indicate the source and target languages, respec- tively. Important parts are highlighted. Full examples. task descriptions and requests for all three proposed methods are in Appendix A, B and C. The other API inputs stay the same.
Monolingual cleaning: In this approach, noisy target sample is given input, and clean target sample is generated output. It demonstrates the ability of LLM clean sen- tences without relying on the original source sample that may contain excessive noise.
of diverse inputs and their corresponding handling strategies. Based on the optimal performance of four-shot examples (Brown et al., 2020), we man- ually curate collection of four-shot examples for each method. The full list of examples used in our approach can be found in Appendix A.
This method generates new par- allel data by taking a noisy source sample as input and producing a clean target sample as output. It showcases LLM's capability of noise ignorance.
Translation:
Prompt design. As shown in Figure 1, each prompt consists of multiple components. The call follows specific layout, starting with a brief de- scription of the context, and providing essential information about the samples, domain, and lan- guage. This is followed by concise formulation of the method: (1) Bilingual cleaning of target sam- ples; (2) Monolingual cleaning of language sam- ples; (3) Data generation through translation. Next,
Chain-of-thoughh prompting. Inspired by the recent achievements of prompting methods (Brown al.. 2020; Schick al., 2023), craft set of few-shot examples that incorporate a coherent chain of thought (Wei et al., 2022). These exam- ples serve to facilitate the model's comprehension

---

we present framework for the desired output, fol- lowed by the few-shot examples for each method, which include the input example, chain-of-thought reasoning for the output, and the example output itself. Finally, we insert the input sample(s) and request the desired output.
addition, we also compare the new target samples from C-MTNT to the original ones by evaluating their semantic similarity.
3.1 Baseline
In addition to our LLM approach, we utilize the iinguge__tool_yttiee module?, which is an open- source grammar tool used for spelling, grammar, and overall language correction. With this rule- based baseline, we want to determine the perfor- mance gap between the rule-based method and our LLM-based approaches.
Language model. The prompts are utilized to interact with OpenAT's GPT through API calls. Specifically,, we use the original GPT-3.5 (text- davinci-003) variant (OpenAI, 2021). In this way, we want to show that some publicly released pre- trained LLMs, like Llama 2 (Touvron et al., 2023b), might also have this ability.
Semantic similarity. To ensure that the text gen- erated by our approach maintains the original mean- ing without unintentional hallucinations (Mündler al., 2023), we set a threshold based on LASER (Artetxe and Schwenk, 2019). LASER is language agnostic, allowing us to compare samples not only within the same language but also across different languages. We measure the cosine similarity be- tween the representations of original and cleaned sentences, as shown in Equation 1. (For bilingual and monolingual cleaning, the original sentence is the noisy target sentence. For translation, the original sentence the noisy source sentence.)
3.2 Quantitative Noise Measurement
We focus on several quantifiable noise types measure the amount of noise a set of samples and obtain an objective overview. These types are present in both the source and target sentences of MTNT, including spelling and grammatical errors, emojis, internet slang, and profanities.
We apply the language Les___ooo___yy_leeee measure the misspellings and grammatical errors. To count the occurrences of emojis in the sentences, we use the emoji library?. For detecting profanities within the text, we employ betterrproofanin for En- glish profanities, and profanity_ pccuukkty_chc""tte for French available and Japanese profanities. As there libraries for detecting internet slang, we compile lists of popular internet slang for each language from the past ten years.
toolkit²
sim (1, e2) = e1 E2
(1)
|e l2 |e2l2
where e1 and e2 are representations of original and newly generated sentences, respectively. Based on previous work on sentence embedding similarity (Okazaki and Tsujii, 2010; Xia et al., 2015), we set threshold for the LASER score as 0.7. This threshold is selected to strike a balance between preserving the meaning of sentences and allowing sufficient variations. If sim(e1, e2) < 0.7, re- peat the API call but include notice in the request: "Please ensure that your response accurately reflects the meaning of the original input sentence.", to en- sure that the meaning of the new sentence aligns closely with the original one. This process contin- ues until sim e2) 2 0.7 or reaching the maxi- mum number of iterations of 10.
As shown in Table 2, we contend that the LLM methods possess the capability simulate natural language as appears in clean benchmarks such Newstest201 14000 TED (Pryzant et al., 2018), KFTT (Neubig, 2011), and JESC (Cettolo et al., 2012). thereby generating clean target sentences. While conventional language correction tools excel in rec- tifying spelling and grammatical errors, they are inadequate in effectively eliminating or paraphras- ing slang, profanities, emojis. Conversely, the LLM methods demonstrate proficiency in address- ing such language phenomena, as also evidenced by some samples in Appendix D. As a result, the target sentences in C-MTNT exhibit significantly less noise compared to MTNT, leveling the cleanli- ness of the reference benchmarks.
3 Analysis on C-MTNT
In this section, analyze the generated data to evaluate its quality and suitability as a noise eval- uation benchmark. We compare our method a rule-based baseline and quantitatively assess the level of noise present in the target sentences. In
com/jxmorris 1 /2/aangagee_oon tarpgge__too__phhhhheee Butps//giitabboom/aaceddDD com/carpedm2220mmmjjj
bttps:/githuu..oom/jomrrri
" "hups://gitub.. com//nguuyentaanibbbttt profftttttrrr
" "hups://github wwtealuub.oo/v/ea//eeeeeeeeeeeee comm///hou8822/pooftitttttt-t
httt::/www
wtw.wwwwwwwtttttt..ttttwwt statmt. wmt S/test.tgz.

---

0.00
0.20
Curmdizo-bull
Morolingal
Bringel
Teradetim
Comeiter-bol
Meewinggs
hilingaa
Trendaiion
Cometor--tos
Munolinga
Bilingaal
Theekdizr
FR-AN
from different input data: bilingual, monolingual target, and monolingual source. The translation method cxhibits the lowest similarity score between original and clean sentences. In contrast, the mono- lingual method shows the minimal deviation be- tween original and clean sentences, while the bilin- LASER similasty
Winkler
Rouge1
Jaccanl
I BLEU
Figure 2: Semantic similarity between noisy (originally from MTNT) and cleaned English, French, and Japanese sentences. The detailed numbers for each bar plot are in Appendix
Lang.
Eval, Set
Spell/Gram.
Emojis
Slang
Profanities
ferent language pairs, revealing distinct deviations
Nowstest2014
0.415
0.000
0.571
0,173
among different methods. These deviations stem
MINT
1.712
0.031
0.816
0.616
Corretion-tool
0.112
0,031
0.818
0.618
EN
Bilingual
0.687
0.000
0.584
0,533
Transtation
0.798
0.019
0.508
Monolingual
0.748
0.019
0.716
0.399
Newsiest2014
2.878
0.000
0.133
0.431
MINT
7.125
0.227
2.225
5.522
Corretionn tool
0.100
0.227
2.139
5.508
Bilingual
0.552
0.016
0.691
0.535
Translation
gual method falls in between. We argue that the larger deviation from the bilingual and translation mainly from rephrasing and word reordering (see Appendix D for detailed samples). Despite these variations, all methods retain a substantial portion of the original semantic structure.
Monalingual
0.485
0.000
0.715
0,551
TED
0.049
0.000
3.493
3.879
KFTT
4.269
JESC
D.036
0,103
7.881
12,084
MTNT
0,051
0.051
0.794
5.682
JA
Conreciio--tol
0.000
0,051
0.794
5.684
Bilingual
D.052
0,012
1.033
5,783
Translation
0.053
0.041
1.119
5,873
Monolingual
Notably, similarity scores from the correction tool the highest for all metrics among all meth- ods, since this rule-based method can only clean or remove noise but lacks the ability to rephrase challenging noise types like slang, emojis, and pro- fanities (see Table for its results on slang, profani- ties, and emojis). Most cleaned sentences stay very similar to the original noisy ones. Complemented by the findings in Section 3.2, our cleaming meth- ods show their impressive ability in reducing noise while preserving semantic similarity.
Table 2: Noise frequency per 100 tokens in the target sentences of the evaluation sets. The best scores are highlighted. Results of Newstest2014, TED, KFTT, and JESC only serve as baselines from clean benchmarks.
Notable is the lower performance in the gener- ated Japanese target sentences. We attribute this two factors: insufficient capture of slang and profanities, and the known variations in perfor- mance of GPT-4 (OpenAI, 2023) across differ- ent languages (Koubaa, 2023). GPT-4 performs much worse on Japanese tasks compared to En- glish tasks. A similar performance discrepancy is expected with GPT-3.5 (OpenAI, 2021).
3.4 Human and GPT-4 Evaluations
Apart from evaluating C-MTNT by measuring its noise amount and its semantic preservation, here conduct human and GPT-4 (OpenAI, 2023) eval- uations.
3.3 Meaning Preservation
Due to the limited research budget, we only con- ducted the human evaluation with the help of the first four authors of this paper on some sampled sentences instcad of all sentences. 100 sentences from C-MTNT Fr-+En are sampled to generate three files. These three files are about binary com- parisons of bilingual vs. monolingual, bilingual vs. translation, and monolingual vs. translation. The order of sentences, including their indexes
Our second objective is to preserve the original meaning during cleaning. We apply multiple met- rics to measure the sentence similarity, including LASER (Artetxe and Schwenk, 2019), BLEU score (Papineni et al., 2002), Rouge-1 score (Lin, 2004), Jaro Winkler distance (Jaro, 1989), and Jaccard score (Hamers al., 1989).
Figure 2 illustrates similarity scores across dif-

---

Loft better
7e
Right bettrr
Blingual
30.0%
37.0%
29.0%
Monaiingul
Bringual
32.2%
36.0%
29.0%
60.0%
20. 0.005
34.0%
Transitioon
Monolinua
27 1..0%
39 5%
33.5%
5%
21.0%
44.0%
Torrebttoon
(a) Human and GPT-4 evaluations on 100 sampled Fr >En pairs. Left: huraa valuation. Right: GPT-4 evaluation.
Bringusl
42.3%
02.0%
7%
35.8%
35 35.7%
28 .9.8%
Monalinuul
Biingud
Transktionn
Monolingaal
Translation
(b) GPT-4 evaluation all samples. Left: En-+Fr. Right: Fr-=En.
Figure 3: Human and GPT-4 evaluations on C-MTNT, Overall, the Bilingual cleaning method results the most cleaned target sentences.
and which cleaning method comes first, is ran- domly shuffled. There is no chance for the an- notator to guess which sentence corresponds to which method, and which file corresponds to the comparison between which two methods. Notably, we prefer binary comparison ranking over three methods, since it's easier for human annotators. In addition, the cleaned sentences from the correction tool are excluded, since they are too easy to be beaten. Four annotators are asked to give their pref- erences for each comparison, based on our three criteria of comprehensive noise removal, semantic preservation, and alignment with the source sen- tence. If both sentences show similar level of noise, they are asked to give a "Tie".
model's robustness against noisy input. Let's re- emphasize our expected NMT model: Irrespective of whether the source sentence is clean or noisy, the model has the ability generate coherent target sentence that is free of errors distortions. We first mimic the woal-world scenario train a of NMT models on datasets that contain both noisy and clean source sentences but with only clean target sentences, then evaluate these models on C-MTNT and other benchmarks.
4.1 Model and Training Details
All models are trained with the fairseq toolkit (Ott et al., 2019). The architecture based on the vanilla transformer, with 6 encoder and decoder layers. The hidden dimension set to 512, with 1024 for the frow-froward intermediate output. We use Adam optimizer (Kingma and Ba, 2015) with its default hyperparameters. Dropout with prob- ability of 0.3 and label smoothing factor of 0.1 (Gao al., 2020) is applied. We train all models for 20 epochs with learning rate of 5e that is scheduled by inverse square root with 4K warmup steps, and set a batch size as 128. Subword tok- enization performed using SentencePicce (Kudo and Richardson, 2018) with BPE subwords. For all languages, we use vocabulary size of 16K without sharing embeddings.
We also prompt GPT-4 (See Appendix G for the prompt) on the same sampled sentences to check whether GPT-4 draws a similar conclusion. As shown in Figure 3a, human and GPT-4 evaluations share similar preference: bilingual translation monolingual. Compared to GPT-4, human anno- tators prefer vote for "Tie". We argue the main reason that most cleaned sentences are very simi- lar with a low level of noise, which further justifies the effectiveness of our proposed methods. Since human annotators and GPT-4 share a similar pref- erence, we further evaluate all C-MTNT sentences with GPT-4 in Figure 3b. The conclusion is similar to the above discussion, i.e. bilingual translation monolingual.
4.2 Training Data
For the English French translation directions, we utilize the same training data as Michel and Neubig (2018), which comprises the europarl-v7"
4 Machine Translation Experiments
this section, we further investigate the suitabil- ity of C-MTNT as a benchmark to evaluate NMT

---

14.0%
cal errors), and synonym substitution (grammatical errors).
12.0%
10.0%
Character augmentation (Char.) involves character-level augmentation methods, including random or controlled techniques with a probability of 0.5 for cach choice (Karpukhin et al., 2019).
Perfomarn: Cain
8.0%
6.0%
4.0%
2.0%
Contextual word embedding augmentation (Con.) utilizes language models, specifically BERT bhert-base-uncasedl (Devlin et al., 2019), substi- tute some word embeddings with their contextual word embeddings (Kobayashi, 2018). We employ the French BERT 10 for French source sentences.
0.0%
0.3
parameter
Non- Contrastie Charocter -
Contrastive Character
Non contrative Error A
Contrastive Emor
Figure 4: Performance vs. augmentation strength, a, for models trained with different augmentation strategies on the French-to-English translation direction. Scores are computed on the bilingual C-MTNT evaluation set.
MTNT-based error replacement (Err.) is inspired by the symbolic strategies (Shorten et al., 2021). Errors are identified with lan- and only the most valuable occurring more than once are retained in a dic- tionary for augmentation. By replacing the correct forms with the mapped common errors, we inten- tionally introduce these errors into the sentence.
and news- www.ss-ommettayy11000 1 corpora. The training set consisted of 2.2M samples, with 55M French tokens and 52M English tokens noon-tokenized). The WMT15 newsdiscussdev2015 serves as the validation set, used to select the best checkpoint. The trained models are evaluated on the C-MTNT, MTNT, and newstest2014 test (eval.) sets.
For synonym substitution (Syn.), we employ WordNet (Miller, 1995) and NLTK (Bird et al., 2009), to randomly select and replace words with their synonyms.
Regarding the English-to-Japanee translation direction, we follow the data construction approach of Michel and Neubig (2018) 8), combining the train- ing and validation data from the KFTT (Neubig, 201 JESC (Pryzant et al., 2018), and TED talks (Cettolo al., 2012) corpora. The Japanese seg- ments in each dataset are tokenized using the MeCab library (Kudo, 2005). This results in a training set of 3.9M samples, consisting of 35M English tokens without tokenization. We use the training and dev sets associated with each corpus the training and validation sets, respectively, and evaluate the models on MTNT, C-MTNT, and the respective test from each corpus.
These techniques only tailored for English and French, which also the main reason for our exclusion of Japanese-to-Englin direction. The source sentences are augmented with a proba- bility of a = 0.1, augmenting upproximately 10% of the tokens in each sample. This process gen- erates four augmented versions: Zeh, Re, že. and 2x, representing sentence augmentation with char- acter, contextual word embedding, error, and syn- onym augmentation, respectively. The selection of 0.1 is based on previous works (Karpukhin et al., 2019; Wei and Zou, 2019) and our similar finding (see Figure 4). These augmented sentences are combined with the original clean sentences, re- sulting in four new training sets, {æ, )ch), }, (x, ) xr, Ze J, and {x, 2.). Each is used to train model, capable of handling some specific types of noise.
4.3
Data Augmentation
The above-mentioned training data are clean, con- tain negligible noise, and can't resemble the real- world use case. Therefore, we introduce noise with different augmentation methods to the source sen- tences, including character augmentation (spelling/- typographical errors, capitalization), contextual word embedding augmentation (word omission/in- sertion/repetitiinn). MTNT-basce error replacement (spoken language, jargon, internet slang, grammati-
Notably, the distribution of introduced noise is not possible to totally resemble the noise distribu- tion in the MTNT (or C-MINT) source sentences. since we only introduce some types of noise to each new set with a pre-defined rule, i.c. the augmenta- tion method. This setting is desired and makes the evaluation more general and practical.
"htt::/wwww.stammt.or wrt/ Straing v10.tgz

---

Non-
Contrasareeevve
Contrastive
Eval.
Base.
Char.
Syn.
Con.
Err.
Char.
Syn.
Con.
Err.
Newstes12014
29.2
29. 924
29. 29.
28.9.1.
29.2au
29. 29.40.
29.2no
28.5.15
29.61
MINT
23.1
25 0g.5
23.62a
23.2nz
23.624
24.87s
22.9.7
22.7.14
23 23.623
Correction tool
23.2
25.2
23.72.
23.4n7
24.348
24.9h4
23.0az
23.0a9
23.826
FR-EN
Bilingual
25.3
28.3120
26.75*
27.06a
26.759
27.9104
27.069
26.138
27.695
Translation
26.2
29.0107
27.6ss
27.86.1
27.446
28.600
27.4a.2
26.71x
28.1
Monolingual
21.4
23.7
21.71.
21.5ux
21.822
23.076
20.9
20.8
21.71a
Newstes12014
30.3
33.190
31.74.
31.5so
31,953
32.616
32.468
32.367
32.674
MINT
20.1
22.4115
20.7
21.2s7
21.8t5
22.2
21.46.
21.1ss
22.311.
Correction tool
19.7
22.0116
20.55g
20.9,o
21.4ks
21.91.1
21.06.
20.9kn
22.0
EN-FR
Bilingual
19.7
22.1
20.75.
21.0hg
21.610.1
22.0122
21.36a
21.174
22.615.0
Translation
19.3
21.9135
20.5
20.877
21.192
21.7
20.9k.
20.770
22.5
Monolingual
16.6
18.511
17.239
17.663
17.9k0
18.511
17.55
17.556
18.813.4
TED
14.4
14.9,3
15.0a.
14.2.1.6
14.400
14.95a
14.40.
14.2.16
14.61.
KFTT
24.9
25.7sa
24.5.17
25.0na
24.2.2.
25.52a
24.7.08
25.0na
24.6.13
JESC
15.2
15.0444
14.8
14.9
15.0
15.1
14.8
14.9
14.9
MTNT
8.8
9.024
9.13.5
9.02.
9.0p.
9.026
9.128
9.026
8. 8.9u.
EN-JA
Correction tool
8.8
9.02s
9.1
9.0g6
8.90s
9.127
9.128
9.026
8.9u
Bilingual
12.6
14.0113
13.91a1
13.35.
13.132
14.1
13.7a3
13.35.
13,467
Translation 14.6
16.1102
15.9gs
15.5k.
15.773
16.4121
16.1
15.5k.
15.77x
Monolingual 9.3
10.1
10.0
9.742
9.2.14
10.1a.
10.1s4
9.74.2
9.2.1.1
Table 3: BLEU scores on various evaluation sets. The subscript number is the relative performance gain G (%), calculated as Equation 2. The best and second-best G' for each augmented dataset are highlighted and underlined. respectively. Results of Newstest2014, TED, KFTT, and JESC only serve as baselines from clean benchmarks.
4.4 Contrastive Learning
MTNT. Therefore, we focus on the relative perfor- mance gain:
In addition to the straightforward training on newly constructed sets, we also train models with con- trastive learning, which is inspired by previous works (Chen et al., 2020; Hwang et al., 2021; Hu and Li, 2022) that recognize the effectiveness of contrastive learning in improving the robustness of NLP and NMT models. By employing this method, we can analyze the performance of C-MTNT on a wider range of models trained with different ap- proaches and settings.
G (s, s6)/sb
(2)
where is the BLEU score from the model trained only on the data without any augmentation, i.e. the training dataset in Section 4.2, and s, is the BLEU score from the model trained on the augmented dataset. If model trained with the augmented dataset obtains higher G on an evaluation set, we can say the evaluation set is an ideal noise evalu- ation set. The reason is: The augmented dataset mimics our expected data distribution. L.c noise only exists in the source side. A model trained on this dataset is supposed to have the ability to translate noisy source sentences into clean target sentences. If this model obtains high G on an evaluation set, it shows that this evaluation set also fulfills the expected data distribution.
For contrastive learning, the Transformer en- coder takes both original and augmented source sentences inputs and calculates the contrastive loss based on their output representations. Simi- lar to straightforward training, we train a separate model on each set with contrastive learning. More details and experiments on contrastive training are in Appendix F.
As shown in Table 3, all trained models tend to have significantly higher G for bilingual and translation C-MTNT. We also show the average G over four models trained with different augmented datasets in Figure 5. It is even more evident that bilingual and translation C-MTNT offers higher G across all cleaning methods.
4.5 Results and Analysis
Before introducing our results in Table 3 for clean benchmarks, MTNT and C-MTNT, we first want to emphasize that the BLEU scores across differ- ent benchmarks are incomparable because these benchmarks contain different evaluation sentences. Though MTNT and C-MTNT contain the same source sentences, their target sentences are differ- ent since we apply LLM to clean the target side of
Some may argue that C-MTNT achieves higher
G score because the augmented training dataset has a similar distribution to C-MTNT, making it

---

10.0%
8.0%
6.0%
4.0%
20%
0.0%
Contrnedive
ENJA
ENJA
MINT
Comedion tool
Monslingul
Bilingel
Translation
Figure 5: Average relative performance gain G over four models that trained different augmented datasets.
easier to evaluate the trained models on C-MTNT. However, this argument is not valid for two rea- sons: (1) Bilingual and translation C-MTNT con- sistently offer higher G across all models trained with different augmented datasets (see Table 3).
et al., 2022; Yoo et al., 2021; ; Chung et al., 2023). These efforts have helped address the issue of data scarcity in various domains.
However, it is crucial to note that the aforemen- tioned research works lack least one of the two novel aspects addressed in this paper. Firstly, our research focuses on evaluating the robustness of NMT models to real-world noise introduced by hu- man users. Secondly, we explore the generation or cleaning of parallel data specifically for NMT purposes. These unique aspects of robustness eval- uation and parallel data generation/cleaning con- tribute to the existing literature in novel way.
It's almost impossible intentionally make every augmented dataset has a similar distribution as C- MTNT where the source sentence contains natural noise; (2) Monolingual C-MTNT offers lower G, sometimes even lower than MINT and the bench- mark constructed from the correction tool. This shows that cleaning with LLM doesn't always work. It's better have guidance, like guidance from source sentence for the bilingual and trans- lation methods. According to our observation, if we clean the noisy target sentence in monolingual way without any guidance, LLM tends to introduce extra information or delete important information, which hurts translation because the cleaned target sentence doesn't tt align well with the source sen- tence. In sum, C-MTNT generated by bilingual and translation methods shows its superiority noise evaluation benchmark, encouraging NMT model to translate noisy source sentence clean target sentence.
Conclusion
In this work, we propose three methods to apply LLM to clean noisy MT benchmark, MTNT, where natural noise exists in both source and tar- get sentences. We aim clean the target side of MTNT to make it more suitable benchmark in evaluating NMT models' robustness against noisy input. With meticulous design of some few-shot prompts, we guide GPT clean the noisy target sentence with only the noisy source sentence, only the noisy target sentence, or both noisy source and target sentences. By measuring the noise frequency in the cleaned target sentences, measuring the se- mantic similarity between noisy and cleaned target sentences, and evaluating with human annotators and GPT-4, we show that our proposed methods can effectively remove natural noise with LLM, while preserving the semantic structure. Our fur- ther investigation of the newly created benchmark, C-MTNT, on some trained models also shows its effectiveness as a noise evaluation benchmark for NMT models.
5 Related Work
The application of LLMs, such as GPT-3.5 (text- davinci-O03) (OpenAl, 2021), ),, in downstream tasks has garnered significant attention and led exten- sive research this field (Wang et al., 2022; Schick et al., 2023). Researchers have conducted compre- hensive investigations to explore the capabilities of LLMs, and built upon their various applications (Coyne and Sakaguchi, 2023; Mündler et al., 2023; Jiao et al., 2023).
Researchers have also observed the potential of LLMs to generate high-quality data, leading to a focus on expanding existing datasets, augment- ing data, or generating entirely new data (Meng
Limitations
Despite the contributions and potential benefits of our research, there are several limitations that

---

should be acknowledged. Firstly, our approach relies on the use of pre-trained LLMs for data cleaning and generation. While LLMs have shown promising capabilities, they are not immune to bi- ases and limitations present in the training data. As a result, our proposed dataset may still contain biases similar to those found in the original MTNT dataset, even after our efforts to mitigate them.
8.1 Biases in Pre-trained Large Language Models
Pre-trained LLMs, such as GPT-3.5, are trained on vast amounts of internet text, which inevitably in- troduces biases present in the training data. These biases can manifest in different forms, including but not limited cultural, gender, racial, and politi- cal biase.. The models can inadvertently reproduce and amplify these biases when generating new con- tent or translating text.
Furthermore, our assessment of the robustness of NMT models against noisy input relies on the uti- lization of C-MTNT, which is created using our pro- posed methodology, and MTNT. While C-MTNT offers valuable insights into the performance of NMT models, it is crucial to acknowledge that it may not comprehensively represent all poten- tial sources of noise encountered in real-world set- tings. -umanngeeeerated noise exhibits variability and contextual dependencies, and our dataset may not encompass the entire spectrum of noise that NMT models may face during actual deployment. The same can be said for MTNT.
It is crucial to acknowledge that biases present in LLMs can influence the quality and fairness of the generated data, potentially perpetuating societal inequalities and reinforcing existing stereotypes. The responsible use of LLMs requires diligent ex- amination and mitigation of these biases to ensure equitable outcomes and avoid further marginaliza- tion or harm to underrepresented groups.
8.2 Mitigating Biases in Data Generation
While we employ LLMs for data cleaning and gen- eration in our proposed dataset, it is essential to note that biases similar to those in MTNT may be present in the generated data. Despite efforts to mitigate biases, the LLMs may not fully capture the complexities and nuances of language, leading to potential biases in the generated sentences.
Additionally, research focuses evaluating the robustness of NMT models in specific language directions, namely English ++ French and English >> Japanese. While these directions provide valu- able insights, generalizing the findings to other lan- guage pairs should be done with caution. Different languages may exhibit unique linguistic character- istics, which can influence the performance and robustness of NMT models. Therefore, further re- search is needed to investigate the generalizability of our findings across broader range of languages and translation directions.
We carefully evaluated the generated data for any biased content and took steps minimize biased outputs. Additionally, we encourage the involve- ment of diverse annotators and domain experts dur- ing the evaluation and curation of the dataset to ensure a broader perspective and mitigate the in- fluence of individual biases. We also encourage translators and reviewers who are well-versed in the target languages and cultural nuances en- sure the translations accurately reflect the intended meaning while avoiding biased or offensive con- tent. Moreover, we actively seek feedback from the affected communities and stakeholders to address any concerns and rectify biases that might arise.
In summary, while our research contributes to the assessment of NMT model robustness and the generation of high-quality evaluation datasets, it is important recognize the limitations associated with biases in LLMs, the potential incompleteness of our dataset, and the need for further investigation into different language pairs.
Acknowledgements
8 Ethical Considerations
We thank all EMNLP reviewers for their great feed- back. The first author, Quinten Bolding, finished this work while doing his thesis at Huwai Amster- dam Research Center. This work was supported by Huawei's infrastructure. The thesis supervisor, Baohao Liao, funded in part by the Netherlands Organization for Scientific Research (NWO) under project number VI.C. 192.080.
The utilization of pre-trained LLMs in natural lan- guage processing tasks, including data generation and machine translation, presents several ethical considerations that must be carefully examined. In this section, we discuss the ethical implications as- sociated with the use of LLMs and highlight the potential biases that may arise in C-MTNT.

---

References
Steven Coyne and Keisuke Sakaguchi. 2023, An analy- sis of gpt-3's performance in grammatical error cor- rection. CoRR, abs/2303. 14342.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguistics, 7:597-610.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL- 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages 4171-4186. Association for Computational Linguistics.
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural ma- chine translation models with transparent attention. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, Brus- sels, Belgium, October 31- November 4, 2018, pages 1028-3033 Association for Computational Linguis- tics.
Yingbo Gao, Baohao Liao, and Hermann Ney. 2020. Unifying input and output smoothing in neural ma- chine translation. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), Decem- ber 8-13, 2020, pages 1361-4372. International Com- mittee Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Kochn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine transla- tion (WMT19) 19). In Proceedings of the Fourth Confer- ence on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019- Volune 2: Shared Task Papers, Day I, pages 1-61. Association for Compu- tational Linguistics.
Lieve Hamers, Yves Hemeryck, Guido Herweyers, Mare Janssen, Hans Keters, Ronald Rousseau, and André Vanhoutte. 1989. Similarity measures scien- tometric research: The jaccard index versus salton's s cosine formula. Inf. Process. Manag., 25(3):315- 318.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Nut- ural Language Processing with Python. O' O'Reilly.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005. 14165.
Felix Hieber, Tobias Domhan, Michael J. Denkowski. and David Vilar. 2020. Sockeye 2: A toolkit for neural machine translation. pages 457-458.
Dongyang Hu and Junhui Li. 2022. Contrastive learn- ing for robust neural machine translation with ASR errors. In Natural Longuage Processing and Chi- nese Computing 11th CCF International Confer- ence, NLPCC 2022, Guilin, China, September 24-25, 2022, Proceedings, Purt I, volume 13551 of Lecture Notes Computer Science, pages 81-91. Springer.
Xiao Shi Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. 2020. Improving transformer optimization through better initialization. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 Proceedings of Machine Learning Research, pages 4475 4483. PMLR.
Mauro Cettolo, Christian Girardi, and Marcello Fed- erico. 2012. WIT3: web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation, EAMT 2012, Trento, Italy, May 28-30, 2012, pages 261-268. European Association for Ma- chine Translation.
Yongkeun Hwang, Hyungu Yun, and Kyomin Jung. 2021. Contrastive learning for context-awaid neu- ral machine translationusing coreference information. arXiv preprint 209.::20000007.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709.
Matthew A Jaro. 1989. Advances in record- -oook-linkage methodology as applied to matching the 1985 census of tampa. florida. Journal of the American Statistical Association, 84(406):414444
Zekai Chen, Mariann Micsinai Balan, and Kevin Brown. 2023. Language models few-shot learners for prognostic prediction. CoRR, abs/2302.
John Joon Young Chung, Fee Kamar, and Saleema Amershi. 2023. Increasing diversity while main- taining accuracy: Text data generation with large language models and human interventions.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt A good translator? A preliminary study. CoRR, abs/2301.0874

---

Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. CoRR, abs/1902.01500
Konstantinos Mavrogiorgos, Argyro Mavrogiorgou. Athanasios Kiourtis. Nikolaos Zafeiropoulos, Spyri- don Kleftakis, and Dimosthenis Kyriazis. 2022. Au- tomated rule-based data cleaning using NLP. In 32nd Conference of O'enn Innovations Association, FRUCT 2022, Tampere, Finland, November 9-11, 2022, pages 162-168. IEEE.
Huda Khayrallah, Hainan Xu, and Philipp Kochn. 2018. The JHU parallel corpus filtering systems for WMT 2018. In Proceedings of the Third Conference on Ma- chine Translation: Shared Task Papers, WMT 2018, Belgium, Brussels, October 31- November 1, 2018, pages 896-899, Association for Computational Lin- guistics.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. In NeurIPS.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhengjie Miao, Yuliang Li, and Xiaolan Wang. 2021. Rotom: A beeaaeeerred data augmentation frame- work for entity matching, data cleaning, text classifi- cation, and beyond. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021, pages 1303-1316. ACM.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. CoRR, abs/1805.06201.
Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. CoRR. 0ass/1809.003
Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky. Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the WMT 2020 shared task on parallel corpus filtering and alignment. In Proceedings of the Fifth Conference on Machine Translation, WMT@ EMNLP 2020, Online, Novem- ber 19-20, 2020, pages 726-742. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11)339-41.
Niels Mündler, Jingxuan He,, Slobodan Jenko, and Mar- tin T. Vechev. 2023. contradcccyy hallucinations of large language models: Evaluation, detection and mitigation. CoRR, abs/2305.15852.
Anis Koubaa. 2023. Gpt-4 vs. gpt-3.5: A concise show- down.
Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.comm cottk/t
Taku Kudo and John Richardson. 2018. Sentencepiece:
Naoaki Okazaki and Jun't auhichi Tsujii. 2010. Simple and efficient algorithm for approximate dictionary match- ing. In COLING 2010, 23rd International Confer- ence on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 851-859 Tsinghua University Press.
A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31- November 4, 2018, pages 66-71. Asso- ciation for Computational Linguistics.
OpenAl. 2021.
GPT-3 API Documentation. https: /bepta openai com/ hom//oss// api- api-reference/ / int roduction.. Accessed
Takumitsu Kudo. 2005. Mecab: Yet another part-of- speech and morphological analyzer.
on: February 23, 2023.
Baohao Liao, Yingbo Gao, and Hermann Ney. 2020. Multi-agent mutual learning at sentence-level and token-level for neural machine translation. In Find- ings the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1715-1724. Association for Computational Linguis- tics.
OpenAl. 2023. GPT-4 technical report. CORR, abs/2303.08774.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2019, Minneapo- lis, MN, USA, June 2-7, 2019, Demonstrations, pages 48 48-53. Association for Computational Linguistics.
Baohao Liao, Shahram Khadivi, and Sanjika Hewavitha- rama. 2021. Back-translation for large-scale multilin- gual machine translation. In Proceedings of the Sixth Conference on Machine Translation, WMT@ EMNLP 2021, Online Fvent, November 10-11, 2021, pages 418 424. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6- 6-12, 2002, Philadelphia, PA, USA, pages 31 311-318. ACL.
Chin-Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches ouf, pages 74-81.

---

Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020. On extractive and abstractive neural document summarization with transformer lan- guage models. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9308-9319. Association for Computa- tional Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR. 0oss2211.105661
Reid Pryzant, Youngjoo Chung, Dan Jurafsky, and Denny Britz. 2018. JESC: Japanese-Engee English subtitle corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.1 19903.
Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022. Leveraging large language models for multiple choice question answering.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. CoRR, abs/1901. 1196.
CoRR, abs/2220.123355
Timo Schick, Jane Dwivedi-YY Yu. Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: tools.
Peipei Xia, Li Zhang, and Fanzhang Li. 2015. Learning similarity with cosine similarity ensemble. Inf. Sci., 307:39-52.
Language models can teach themselves CoRR, abw/2302.0477 2.761..
Hainan Xu and Philipp Koehn, 2017. Zipporah: a fast and scalable data cleaning system for noisy web- crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2017, Copenhagen, Den- mark, September 9-11, 2017, pages 2945 2950. As- sociation for Computational Linguistics.
Connor Shorten, Taghi M. Khoshgoftaar, and Borko Furht. 2021, Text data augmentation for deep learn- ing. J. Big Data, 8(1):101.
Ruixiang Tang, Xiaotian Han, Xiaoqiam Jiang, and Xia Hu. 2023. Does synthetic data generation of lims help clinical text mining? CoRR, 0bs/2303.04330..
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woo-Myoung Park. 2021. Gpt3mix: Leveraging large-scale language models for text aug- mentation. Findings of the Association for Com- putational Linguistics: EMNLP 2021. Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 2225-2239, Association for Computa- tional Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almabairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jemya Lee, Di- ana Liskovich, Yinghai Lu, Yuming Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu. Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama Open foundation and fine-tuned chat models. CoRR, abs/2307.00 009888.

---

A Few-Shot Examples
love it!"
Reasoning: The noisy French sentence uses non-standard spelling ("troo" for "très") and excessive letters in the words "bil" and "jkiff." The clean English sentence corrects the spelling and removes the extra letters for more natural and concise expression.
Bilingual cleaning for FR-EN. For the bilingual cleaning in the Brench-to-English translation direction, the following examples:
Input: "Jss tro contenteee!",
"Im s0000 happyyyy! @
Desired output: "I'm m so happy!
@
Reasoning: The noisy French sentence uses abbreviations ("Jss" for "Je suis") and excessive letters in the word "contenteee," The clean English sentence replaces the abbreviations and removes the extra letters to convey the same meaning clearly.
Bilingual cleaning for EN-FR. For the bilingual cleaning in the English-to- to-French translation direction, we use the following examples: Input: "I'm m 800000 happyryy!@ " "Je suis tellement heureuxxxx !@
Desired output: "Je suis tellement heureux!"
Input: "Tkt, e trooo bil! @ "Don't worry, it's awesooome!
Reasoning: The noisy French sentence contains excessive letters in the word "heureux" and an unnecessary exclamation mark. The clean French sentence removes the extra letters and the exclamation mark convey the same message accurately.
@
Desired output: "Don't worry, it's great!"
Reasoning: The noisy French sentence uses internet slang ("Tkt" for "T' inquiète") and excessive letters in the word "trooo." The clean English sentence replaces the slang with more standard expression and removes the extra letters for better comprehension.
Input: "Can't wait to see youuuu! @ " "J'ai trooooop håte
de te voiiiiir Desired output: "J'ai tellement hâte de te voir!"
@
Reasoning: The noisy French sentence includes excessive letters in the words "trooop" and "voiiilir." The clean French sentence removes the extra letters to maintain the same meaning more concisely.
Input: "Jvoudré vnir ave vs! "I wud luv 2 cum wth u guys! Desired output: "I would love to come with you!"
@
Reasoning: The noisy French sentence uses abbreviations ("Jvoudré" for "Je voudrais") and non-standard spelling ("vnir" for "venir") The clean English sentence replaces the abbreviations and corrects the spelling for clearer communication.
Input: "Let'a grab bite laterrrr! = "Allons
manger un morceau plu tarrrrd!
Desired output: "Allons manger un morceau plus tard!" Reasoning: The noisy French sentence has excessive letters in the words "plu" and "tarrrrd" and unnecessary fast food emojis.
"Ct troo bil, jkiff trp!0 ", "it's sooo gre8, luv it!O
Input:
Desired output: "It's amazing,

---

The clean French sentence removes the extra letters and the emojis while maintaining the same meaning.
Input: "Let's grab bite
laterrrz! -
"
? Hi
<
f
à! -
Desired output: "K k R L
Input: "This movie is amaziiing! & 3 ", "Ce film est troccop géniaaaaal! & "
Reasoning: The noisy Japanese sentence includes excessive elongation in the word " t> $ a 55 and unnecessary fast food emojis. The clean Japanese sentence removes the excessive elongation and provides a more polite and appropriate phrase to convey the same meaning.
Desired output: "Ce film est tellement génial!" Reasoning: The noisy French sentence contains excessive letters in the words "troooop" and "géniaaaaal" and unnecessary fire and heart emojis. The clean French sentence removes the extra letters and the emojis to convey the same message accurately.
Input: "This movie is amaziiing! 1 ", # Conon A 1
Desired output: " L レ7#
Bilingual cleaning for EN-JA. For the bilingual cleaning the English-ton Baglish-to-Japaan to-Japanese translation direction, we use the following examples:
Reasoning:
The noisy Japanese
sentence uses excessive elongation in the word " = D and includes unnecessary fire and heart emojis. The clean Japanese sentence removes the excessive elongation and provides more appropriate and accurate expression for the same meaning.
Input: "I'm 80000 happyyyy!
< LW
Desired o " " !
Reasoning: The noisy Japanese sentence uses excessive
elongation in the word "# - < and includes an unnecessary exclamation mark. The clean Japanese sentence removes the excessive elongation and uses more polite form to convey the same meaning accurately.
Translation for FR-EN. For the generative translation method in the French-to-English trans- lation direction, we use the following examples: Input: "Heyyy, ça trop
biennn! @ Jsuis trop hypeééé pour ce soir! M
Desired output: "Hey, I'm doing great! I'm so excited for tonight!"
Input: "Can't wait to see youuuu! @ Desired output: L
Reasoning: The noisy sentence contains excessive letters in words and emojis. The clean sentence removes the extra letters and emojis convey the same message more clearly.
Reasoning: The noisy Japanese sentence includes excessive elongation in the word "I & t "" and an unnecessary exclamation mark. The clean Japanese
sentence removes the excessive elongation and uses a more polite form for a clearer and more appropriate expression.
Input: "OMG jpeux pas croire, e'est trooop ouf!
Desired output: "Oh my God, can't believe it, it's so amazing!"

---

Reasoning: The noisy sentence uses internet slang ("OMG," "trooop, " "ouf") and excessive punctuation ("!!"). The clean sentence replaces the slang with more standard expressions and removes the excessive punctuation for better comprehension.
Input: "OMG can't even m,, this party is gonna be LIT AF! "
Desired output: "Mon Dieu, je ne peux pas le croire, cette fête va être incroyable !"
Reasoning: The noisy sentence uses internet slang ("OMG," "rn," "LIT AF") and abbreviations. The clean sentence replaces the slang with more standard expressions and removes the abbreviations for better comprehension in French.
Input: "Mdr t'es trop marrant, tu me fais tp rire @
Desired output: "Haha, you' re so funny, you make me laugh a lot." Reasoning: The noisy sentence contains internet slang ("Mdr," "trop," " "tp") and laughing emoji. The clean sentence replaces the slang with more common expressions and removes the emoji for a more formal and clear communication.
Input: "lol ur so funny, always make me laugh @ @
Desired output: "lol tu es tellement drôle, me fais toujours rire."
Reasoning: The noisy sentence contains internet slang ("1ol," "ur") and a laughing emoji. The clean sentence replaces the slang with more common expressions and removes the emoji for more formal and clear communication in French.
Input: "Héé, on se voit au restau tout de suite?"
Desired output: "Hey, can we meet at the restaurant right away?- क
Reasoning: The noisy sentence has intentional misspellings ("Héé," "restau") and fast food emojis. The clean sentence corrects the spellings and removes the emojis to convey the same message accurately.
Input: "Heyy, wanna grab some fud later? - 2
Desired output: "Hey, tu veux qu' on aille manger plus tard ?" Reasoning: The noisy sentence has intentional misspellings ("Heyy," "fud") and fast food emojis. The clean sentence corrects the spellings and removes the emojis to convey the same message accurately in French.
Translation for EN-FR. For the generative translation method in the English-to-French trans- lation direction, we use the following examples: Input: "Heyy, what's s up? @ I'm sooo exicteddd to go out tonight!
O "
Desired output: "Salut, quoi de neuf ? Je suis tellement. excité (e) de sortir ce soir !" Reasoning: The noisy sentence contains excessive letters in words and emojis. The clean sentence removes the extra letters and emojis to convey the same message more clearly in French.
Translation for EN-JA. For the generative translation method in the English-to-Japanees translation direction, use the following exam- ples:
Input: "Heyyy, what's s up? I'm
s000 excited to go out tonight!
Desired output: " T
Reasoning: The noisy sentence

---

contains excessive letters in words and emojis. The clean sentence removes the extra letters and emojis to convey the same message more clearly in Japanese.
Desired output: "Hey, what's up? I'm so excited to go out tonight!"
Reasoning: "The noisy sentence contains excessive letters in words and emojis. The clean sentence removes the extra letters and emojis convey the same message more clearly."
Input: "OMG I can't t even In, this party is gonna be LIT AF! & à " Desired output: " ,. RY ?? Σグ~ー= 1 11 E
Reasoning: The noisy sentence uses internet slang ("OMG," "rn," "LIT AF") and abbreviations. The clean sentence replaces the slang with more standard expressions and removes the abbreviations and emojis for better comprehension in Japanese.
Input: "OMG can't t even rn, this party is gonna be LIT AF! A
Desired output: "Oh my God, I can't t even right now, this party is going to be awesome!"
Reasoning: "The noisy sentence uses internet slang (' ('OMG', 'rn', 'LIT AF") and abbreviations. The clean sentence expands the slang and abbreviations for better comprehension."
Input: "lol you' re so funny, you always make me laugh @ Desired output: "* B サントЫ
Hut. (
Reasoning: The noisy sentence contains internet slang ("lol") and a laughing emoji. The clean sentence replaces the slang with more common expressions and removes the emoji for a more natural and clear communication in Japanese.
Input: "lol funny, always make laugh @ x
Desired output: "Laugh out loud, you're so funny. You always make me laugh." Reasoning "The noisy sentence contains internet slang (' ('lol', 'ur') and a laughing emoji. The clean sentence replaces the slang with its full form and removes the emoji for more formal expression."
Input: "Heyyy, wanna grab some food later? -
Desired output: " K # ?nカTn?? Reasoning: The noisy sentence has intentional misspellings ("Heyyy") and fast food emojis. The clean sentence corrects the spellings and removes the emojis for a more natural and clear communication in Japanese.
Input: "Heyy, wanna grab some fud later? = s
Desired Output "Hey, do you want to grab some food later?" Reasoning: "The noisy sentence has intentional misspellings (' H'Hey'', ffud") and food emojis. The clean sentence corrects the spellings and removes the emojis to convey the same message accurately."
Monolingual cleaning for FR-EN. For the monolingual cleaning in the French-to-Be English translation direction, we use the following exam- ples:
Input: "Heyy, what's up? I'm sooo exicteddd to go out tonight!
Monolingual cleaning for EN-FR. For the monolingual cleaning in the English-to Freeeshhto-French

---

translation direction, we use the following exam- ples:
same message accurately.
Input: "Saluttt, ça va? Je
Monolingual cleaning for EN-JA For the monolingual cleaning in the English-to-Japanen translation direction, we use the following ex- amples: Input:
suiis trp excitéééé pr sortir ce soiiiir! V
Desired output: "Salut, ça va? Je suis trop excité pour sortir ce soir!"
" Hh
Desired output: ?
"
Reasoning: The noisy sentence contains excessive letters in words and emojis. The clean sentence removes the extra letters and emojis to convey the same message more clearly.
Reasoning: The noisy sentence contains informal language (" $ h"" instead of "T $ h²) and excessive use of at the end. The clean sentence removes the informal elements and expresses the same meaning more formally.
Input: "Tkt, j'te dm dans 2min, ok? @
Desired output: "T' inquiééé, je te donne des nouvelles dans minutes, d' d'accord?"
Input: "Hh 5 Ior
Desired output: " و 7 BnWw!
Reasoning: The noisy sentence uses internet slang ('Tkt', 'j'te', 'mm') and abbreviations. The clean sentence expands the slang and abbreviations for
*
Reasoning: The noisy sentence includes the use of "t 5 $" (a casual intensifier) and the English acronym "LOL." The clean sentence removes the casual intensifier and replaces "LOL" with the Japanese equivalent " (meaning "laugh")
better comprehension.
Input: "Mdrr t'es tro drol, tu m' fais tp zire
Desired output: "Mort de rire, tu es vraiment drôle, tu me fais trop rire."
Input: "P^л, h B1! Desired output: " è m !
Reasoning: The noisy sentence contains internet slang ('Mdrr', 'tro', 'tp') and laughing emoji. The clean sentence replaces the slang with its full form and removes the emoji for a more formal expression.
Reasoning: The noisy sentence uses "P i >>" (a casual laughter expression) and an emoji. The clean sentence replaces "P > л with the more standard "*" and removes the emoji.
Input: "Héé, on se retrouve au medo plutar? ب me
Input: "koLe,
#
Desired output: "Hé, est-ce qu'on on peut se retrouver au McDonald's plus tard?"
Desired output: "LL # #L=50?"
Reasoning: The noisy sentence has intentional misspellings (' ('Héé', 'plutar') and fast food emojis. The clean sentence corrects the spellings and removes the emojis to convey the
Reasoning: The noisy sentence contains misspelling (" 2 hr" instead of " L x 5 /T") and fast food emojis. The clean sentence corrects the spelling and removes the emojis while maintaining the same meaning.

---

B Task Descriptions
D Samples
In Table 4, we show several samples from MTNT, and show how each distinct method tends to clean these samples in different ways. In some samples abbreviations are corrected, in others emojis removed and some are changed based language or word choice.
The task description for the bilingual cleaning method is follows:
Your task is to clean the given [tgt) sentence. You will receive two sentences as input: the (src) sentence containing noise, and the translated tgt version of that sentence, also containing noise. Your task is to clean only the (tgt) sentence and return it as output.
E Detailed Similarity Scores
For the convenience of the latter works that plan use our method as a baseline, we list the detailed similarity scores from Figure Table 5, and 7.
The task description for the generative translation method is as follows:
Rouge-1
Bilngual
US3
0.90
0.75
0.00
0.54
Your task is to translate the given noisy (src) sentence to the correct (tgt) version, thereby removing all noise. You will only return the clean (tgt) sentence as output.
Transiaiion
0.89
0.81
0.61
0.44
0.26
Monolingral
0.85
0.91
0,2
0.21
0,00
Conrection-tool
0.99
0.98
0.99
0.7
0.94
Table 5: Similarity scores hetween noisy and cleaned French-to- to---lish- target samples.
The task description for the monolingual cleaning method is as follows:
LASER
Winkler
Kilinua
Transtation
0,89
0.79
0.54
0.37
0.23
Your task is to clean the (tgt) sentence that you will receive as input: You will then return the clean version of the (tgt) sentence as output.
Monelingsal
0.97
0.93
0.58
0.00
0.73
Correction sool
0.99
0.97
0.93
0,88
0,34
Table 6: Similarity scores hetween noisy and cleaned English-to-Frenee target samples.
For each task (sre) refers to the source language (English French) and (tgt) refers the target language (English, French, or Japanese).
LASER
Winkler
Bilingul
Translation
0,83
0.70
0.48
0.35
Monolingoal
0.96
0.93
0.90
0.55
0.76
Correction-co spol
1.00
1.00
1.00
1.00
0.98
C Requests
Table 7: Similarity scores between noisy and cleaned English-to-lapanese target samples.
This section shows the requests we used in the prompts. The request for the monolingual cleaning method and generative translation methods
F
Contrastive Learning
F.1
Training Loss
are the same and as follows: This is Please
the input sinput_sentl/ return the desired output in the correct format. The only difference that the (input_sent) refers to the target sentence in the case of the monolingual cleaning method, whereas the (input (input_sent) refers to the source sentence for the generative translation method. The request for the bilingual cleaning task is different because of the multiple inputs. The request for this method is as follows: These are the inputs (sre_sent), (tgt_sent). Please return the desired output in the correct format.
The contrastive loss (Chen al., 2020) is computed based on two source sentences: the original source sentences & and the augmented source sentences 2.
Car = - Σ log exp(sim(e,t,ec)/7
Σ exp(sim(e )/T)
where x² is the original sentence, zf is the aug- mented version of x², and is the temperature factor. Therefore, the positive sample the corre- sponding augmented sentence, while the negative samples are the augmented versions of other orig- inal source sentences from the same mini-batch.

---

ext and egt are the average representations along the sequence dimension from the encoder outputs. Apart from the contrastive loss, the standard cross-entropy loss is calculated as:
the optimal À varies for different augmentation methods. We set À 0.01 by default since this value works better for most methods.
10%
N
Lee =
Σ (log Po(y'la") + log Po()"(z))
8%
6%
Pefomerce
We combine both losses as the final loss:
4%
2%
L= Loc + ofr
0%
0.00
0.01
0.82
0.03
where À is an interpolation factor. We incorporate the augmented source inputs 2 to ensure that the model can still generate correct translations with noisy input.
-2%
-4%
parameler
Charactee
Eroo Aug. Coatest
Synsaym in
F.2 Optimal Hyperparameters
Figure 7: Performance loss balance factor for models trained with different augmentation strategies on the French- to hoo-Englih translation direction. Scores are computed on the bilingual C-MTNT evaluation set.
We conduct thorough experiments to choose the optimal hyperparameters for contrastive learning, Temperature This hyperparameter plays crucial role in adjusting the softmax function used in the contrastive learning framework, thereby af- fecting the distribution of the similarity scores be- tween augmented and original sentences. By vary- ing the value of т, we can control the concentra- tion or diffusion of the score distribution. Figure 6 shows the results from the models trained with different augmentation strategies. It is evident that 0.1 uniformly performs the best. So set т 0.1 by default.
G Prompt for GPT-4 Evaluation
" """In the following, I'm going to show you one noisy source sentence in French and one noisy target sentence in English. In addition, I also offer you two clean versions of the noisy target sentence.
Can you rank these two clean target sentences based on these three criteria:
13.0%
8.0%
1. Comprehensive noise removal: All forms of noise should be eliminated from the noisy target sentence, including removing semantically meaningless emojis, translating emojis with semantic content into words, correcting mi misspellings, etc.
Pedeemaze gen (98)
3.0%
-20%
-7.0%
-120%
parameter
Charcter
Aug
2. Semantic preservation: The clean target sentence should retain similar semantic information as the original noisy target sentence.
Syneaym
Cuntest
Figure 6: Performance vs. temperature factor r for models trained with different augmentation strategies the French-to- -eengh-t-Engglish translation direction. Scores are computed on the bilingual C-MTNT evaluation set.
3. Alignment with the source sentence: The clean target sentence should convey the same intended meaning as the noisy source sentence, ensuring accurate translation and faithful
Loss balance Our final loss consists of the standard cross-entropy loss and the contrastive loss. Here we conduct experiments to choose the op- timal loss balance factor A. As shown in Figure

---

representation of the original
content.
Noisy source sentence in French:
{0)
Noisy target sentence in
English: {1}
The first clean target sentence:
(2)
The second clean target
sentence: (3)
For your output, you don't need to give any explanation. If the first version is better, you output 1. If the second version is better, you output 2. If they are equally clean, you output 3. 3.""".format (source, target, ltte___froocceeennnee target__f ceatt_ffromceeee___e metho v2)
vl,

---

Method
Lang.
Noisy Target Sample
Clean Target Sample
"pp don't 't have many juicy stories to tell right now."
"I don't have many juicy stories to tell at the moment."
EN
"If in doubt, tinker with the doc."
"If in doubt, tinker with the document." "Social network does not equal reality, but when you add newspapers to that..."
"Social network =/= reality but when you add
Bilingual
news paper to that..."
"Je pense que plus I'on reste céto et moins on envie de ces choses."
"Je pense que plus on reste Keto, moins on envie ces choses."
FR
"el oh el merci, j' ai le même espoir pour toi aussi compère"
"Mort de rire merci, j'ai le méme espoir pour toi aussi, compadre"
"Je ne sais pas quoi faire Passe à autre chose."
"Je ne sais pas quoi faire. Allons de I'avant."
"WresiteMania it Fla LireT
4
d Wtestleeaniii T-AI
JA
<
1ン 9 A A E.s. f t
"
L 33
L. 3323
1 カー 39
21 11
2 2リ
7
"A A.R.A. 322
J b.
"p don't tt have many juicy stories to tell right now."
"I don't 't have many juicy stories to tell right now."
EN
"If in doubt, tinker with the doe."
"If you are unsure, tinker with the document." "Social network does not equal reality but when you add newspaper that..."
"Social network =/= reality but when you add
Monolingual
news paper to that..."
"Je pense que plus I'on reste céto et moins on
"Je pense que plus on reste Keto et moins on envie de ces choses."
FR
envie de ces choses."
"el oh el Lol j' le même espoir pour toi aussi compère"
"Mort de rire merci, j'ai le même espoir pour toi aussi compère"
"Je sais pas quoi faire Passe à autre chose."
"Je ne suis pas quoi faire. Pusse à autre chose."
"WreshteMania AA F0 4
Wretleeaaiaaa at is f 22l
JA
# LU
" L. 144 OL 1 7 ..
"} L 7 77 カシシーーレ
3 x x
"A h A. t. 3 L
"AAh Ahhhh ts 3
カ.
fal L
"p don't 't have many juicy stories to tell right now."
"I have few juicy stories to tell at the moment."
EN
doubt, tinker with the doc."
"When in doubt, tweak the document." "Social media does not equal reality, but when you add journals to that, on the other hand..."
"Social network =/= reality but when you add news paper to that..."
Translation
"Je pense que plus l'on reste céto moins on envie de choses."
"Je pense que plus nous restons mode Keto, choses."
FR
moins nous avons envie "Mort de rire merci.j j'ai méme espérance pour toi aussi compadre"
"el oh el Lol le même espoir pour toi aussi compère"
"Je ne sais pas quoi faire Passe à autre chose." "
"Je ne sais pas quoi faire. Avancer."
"WrestteMania #l LLTTT.
4
" f o Wiresleeaniia 7 L
JA
1ン AARAA tEd I tr
35
24 イ
23, "
"s-he サタすナトビ. 3 LINSSFW B
Table 4: Several noisy target samples from MTNT and C-MTNT with different cleaning methods. The red text is noise, the blue text indicates rephrased parts, and the green text indicates the removal or correction of noise.


Reviews:
[{"review_id": "X54YZyso2Z", "reviewer": "Reviewer_pKJv", "summary": "", "questions": "1. In Table 2, why is the numbers in the spelling error column not 0 for the correction-tool rows? Does it mean that the tool can recognize more errors than correct them?\n2. Would the collected slangs (line 295-297) be released?\n3. How are the prompt examples selected and how sensitive is the LLM to the choice of these examples?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n* Offers an interesting idea of using LLM to clean up references;\n\n* Addresses the insufficient coverage issue of traditional rule-based cleaning methods;\n\n* Offers insightful and very helpful experimental results for future research.\n\nWe have responded to your comments below and updated our draft accordingly. \n\nIf you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n\n---\n\n> Reasons To Reject: I concede the point that it's nontrivial to say which test set is more suitable as a benchmark since we are comparing different test sets, but I don't find G (introduced in eq 2) a satisfactory solution -- would a 20->22 increase in BLEU actually be better than a 30->32 increase? We would conclude 'yes' if we use G, but I'm not entirely sure. I would have been happier with some sort of human evaluation/auditing of C-MTNT than the data augmentation experiments that make up the latter half of the paper.\n\nWe totally agree with the point that 20->22 increment in BLEU isn't necessarily better than 30->32 increment, even though our metric G implies it. \n\nThanks to your advice, we conducted an emergency human and GPT-4 evaluation experiment on C-MTNT. For your convenience, we first show the conclusion here: **Overall, human and GPT-4 evaluation share a similar preference: bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods.** \n\nThe experimental setting is:\n\n(1) 100 sentences from C-MTNT Fr->En are sampled to generate three files. These three files are about binary comparisons of bilingual vs. monolingual, bilingual vs. translation, and monolingual vs. translation. The order of sentences, including their indexes and which cleaning method comes first, is randomly shuffled. There is no chance for the annotator to guess which sentence corresponds to which method, and which file corresponds to the comparison between which two methods. (Notably, we prefer binary comparison to ranking over three methods, since it's easier for human annotators. In addition, the cleaned sentences from the correction tool are excluded, since they are too easy to be beaten.)\n\n(2) Four annotators (authors of this paper) are asked to give their preferences for each comparison, based on our three criteria of comprehensive noise removal, semantic preservation, and alignment with the source sentence (Lines 90-103). If both sentences show a similar level of noise, they are asked to give a \"Tie\".\n\n(3) Due to the time limit, we couldn't compare all sentences from different translation directions, and hire a larger amount of annotators. To overcome this limit, we also prompt GPT-4 to do the comparison on all sentences on the Fr<->En corpora. En->Ja is excluded since GPT-4 is not good at Japanese, as stated in our paper (Line 314-322). We append the prompt at the end of this response, i.e. Appendix A.\n\nHere is the result from human evaluation on the sampled 100 sentences of Fr->En.  \n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **30.0%** | 46.8% | 23.2% | Monolingual |\n| Bilingual | **32.2%** | 38.0% | 29.8% | Translation |\n| Monolingual | 27.0% | 39.5% | **33.5%** | Translation |\n\nThe GPT-4 evaluation result of these 100 samples is:\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **34.0%** | 37.0% | 29.0% | Monolingual |\n| Bilingual | **40.0%** | 26.0% | 34.0% | Translation |\n| Monolingual | 35.0% | 21.0% | **44.0%** | Translation |\n\nOverall, human and GPT-4 evaluations share a similar preference: bilingual > translation > monolingual. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods. \n\nThe results from the GPT-4 evaluation on the whole C-MTNT are:\n\nEn -> Fr\n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **42.3%** | 32.0% | 25.7% | Monolingual |\n| Bilingual | **45.8%** | 21.2% | 33.0% | Translation | \n| Monolingual | 31.3% | 16.8% | **51.9%** | Translation | \n\nFr -> En\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **35.5%** | 35.7% | 28.8% | Monolingual |\n| Bilingual | **40.2%** | 22.8% | 37.0% | Translation |\n| Monolingual | 35.4% | 15.3% | **49.3%** | Translation |\n\nThe conclusion is similar to the above discussion, i.e. bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. We have added these new results to our updated draft. And the newly collected evaluation scores will also be released later.\n\n---\n\n> Question 1. In Table 2, why is the numbers in the spelling error column not 0 for the correction-tool rows? Does it mean that the tool can recognize more errors than correct them?\n\nIn the process of rectifying grammar and spelling errors, the correction tool occasionally generates sentences that deviate from conventional structures or inadvertently introduce new grammatical or spelling errors. Consequently, upon post-correction verification, a minimal number of discrepancies might still be identified, even though they are significantly reduced compared to the original noisy sentence.\n\n---\n\n> Question 2. Would the collected slangs (line 295-297) be released?\n\nDue to the time limit, we couldn't release all experimental materials upon submission. However, the release of all materials for reproduction is on our plan. If you still have other requests, please do let us know.\n\n---\n\n> Question 3. How are the prompt examples selected and how sensitive is the LLM to the choice of these examples?\n\nThank you for raising this important question regarding the selection of prompt examples and the sensitivity of the LLM to these choices.\n\n1. **How are the prompt examples selected?**\n\nThe prompt examples are initially generated by the model (GPT-3.5) itself with the following prompts:\n\n```\nGenerate 20 examples of parallel sentence pairs where each pair consists of a noisy source sentence in {source_language} and its noisy {target_language} translation. The noise could be in the form of emojis, slang, jargon, slurs, grammatical errors, spelling errors, or code-switching. Also, provide the cleaned-up version of the {target_language} translation for each example. Thus a version of {target_language} translation without any of the aforementioned noise types.\n```\n\nafter which they are subjected to a manual selection process to guarantee their quality, relevance, and efficacy. Our selection process consists of several key steps:\n\n* **Relevance to objective**: Each generated example is first evaluated against the primary objectives of the paper or study. This ensures that the examples serve as accurate representations of the phenomena we aim to explore.\n\n* **Quality of content**: Following this, we assess the quality of the example in terms of language, coherence, and sentence flow. This includes verifying that the output generated from these prompts is accurate and pertinent.\n\n* **Consistency with other data**: To ensure that the chosen examples are not outliers or anomalous data points, they are compared with other data in the study.\n\n* **Instructional value**: Finally, the instructional value of each example is appraised to confirm that it aids in explaining or demonstrating the phenomena under investigation.\n\nThrough this multi-step process, we strive to minimize any sensitivity of the LLM to the choice of examples and ensure the robustness of our findings. Sorry for not adding these details to the paper. We already included it in our updated draft.\n\n2. **How sensitive is the LLM to the choice of these examples?**\n\nWe conduct new experiments by designing new prompt examples. Similar to the human evaluation part, we only collect 100 clean sentences on Fr->En with the new prompt examples, keeping the other parts of Figure 1 the same for the ablation study. After collection, we use the same prompt in Appendix A to ask GPT-4 to give its preference, since it shares a similar preference as human annotators.\n\nFirst, we design two new sets of prompt examples for the bilingual cleaning method to check whether new prompt examples cause large variances for the same cleaning method. The results are:\n| | Left is better | Tie | Right is better |  |\n| --- | :---: |:---: |:---: |--- |\n| Original prompt examples | 18% | 60% | 12% | New prompt examples v1 |\n| Original prompt examples | 20% | 53% | 27% | New prompt examples v2 |\n\nWe do see the variance with different prompt examples. However, the major preference (>50%) of GPT-4 is located in \"Tie\", indicating that the same cleaning method with different prompt examples offers similar cleaned sentences.\n\nSecond, we design two more sets of prompt examples for the monolingual and translation cleaning methods to check whether we have a similar pattern as the previous human/GPT-4 evaluation. The results are:\n| | Left is better | Tie | Right is better |  |\n| --- | :---: |:---: |:---: |--- |\n| Original bilingual | **36%** | 40% | 24% | Monolingual with new prompt examples |\n| Original bilingual | **34%** | 35% | 31% | Translation with new prompt examples |\n| Monolingual with new prompt examples| 29% | 31% | **40%** | Translation with new prompt examples |\n\nA similar pattern, i.e. bilingual > translation > monolingual, is observed. **In sum, different prompt examples will cause a mild variance, but the observed pattern is similar.**\n\n---\n\n> Presentation Improvement: Table 3 numbers are a bit too small to read comfortably.\n\nThanks for this advice. We already adjusted the font size of Table 3 in our updated draft.\n\n---\nAppendix A. Prompt for GPT-4 evaluation\n```\ndef generate_prompt(source, target, v1, v2):\n    return \"\"\"In the following, I'm going to show you one noisy source sentence in French and one noisy target sentence in English. In addition, I also offer you two clean versions of the noisy target sentence. \n\nCan you rank these two clean target sentences based on these three criteria:\n1. Comprehensive noise removal: All form of noise should be eliminated from the noisy target sentence, including removing semantically meaningless emojis, translating emojis with semantic content into words, correcting misspellings, etc.\n2. Semantic preservation: The clean target sentence should retain similar semantic information as the original noisy target sentence.\n3. Alignment with the source sentence: The clean target sentence should convey the same intended meaning as the noisy source sentence, ensuring accurate translation and faithful representation of the original content.\n\nNoisy source sentence in French: {0}\nNoisy target sentence in English: {1}\nThe first clean target sentence: {2}\nThe second clean target sentence: {3}\n\nFor your output, you don't need to give any explanation. If the first version is better, you output 1. If the second version is better, you output 2. If they are equally clean, you output 3.\"\"\".format(source, target, v1, v2)\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I find the idea of using LLMs to clean up references quite interesting, and it addresses the issue of insufficient coverage of the traditional rule-based cleaners.\n2. The experimental results on three different ways of cleaning the references are insightful and very helpful for future research.", "reasons_to_reject": "I personally find the first half of the paper (i.e., on cleaning up the data) more convincing than the second half (i.e., on showcasing C-MTNT's effectiveness to evaluate MT robustness). I concede the point that it's nontrivial to say which test set is more suitable as a benchmark since we are comparing different test sets, but I don't find G (introduced in eq 2) a satisfactory solution -- would a 20->22 increase in BLEU actually be better than a 30->32 increase? We would conclude 'yes' if we use G, but I'm not entirely sure. I would have been happier with some sort of human evaluation/auditing of C-MTNT than the data augmentation experiments that make up the latter half of the paper.", "questions_to_authors": "1. In Table 2, why is the numbers in the spelling error column not 0 for the correction-tool rows? Does it mean that the tool can recognize more errors than correct them?\n2. Would the collected slangs (line 295-297) be released?\n3. How are the prompt examples selected and how sensitive is the LLM to the choice of these examples?", "paper_topic_and_main_contributions": "This paper investigates the possibility of using LLMs to clean up the references of MTNT, a noisy MT dataset on both the source and the reference sides. It studies three ways for prompting LLMs to clean up the reference by 1) looking only at the target; 2) looking at both the target and the source; 3) looking at the source only (i.e., translation). The analysis shows that the second approach actually delivers the best result in terms of preserving the original meaning and reducing the noise. Through experiments on data augmentation, the paper also argues  that the cleaned dataset is better at evaluating the robustness of MT systems.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 3 numbers are a bit too small to read comfortably."}, "extra_text": {}}, {"review_id": "m3yrGP14yj", "reviewer": "Reviewer_Z2kB", "summary": "", "questions": "Why did not introduce the request in lines 260-261  in the first place?   What is the maximum number of iterations?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n* Offers a helpful and cleaned dataset for exploring the application of MT models in real-life scenarios.\n\nWe have responded to your comments below and updated our draft accordingly. \n\nIf you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n\n---\n\n> Reasons To Reject: The paper should be strengthened in the writting, which is not fully clear now. For contrastive learning, which is the positive sample? Which is the negative sample? Please explain in detail in Section 4.4.\n\nSorry for the unclear explanation of contrastive learning. You can find more details in Appendix F.1. For your convenience, we show the related contents here. We calculated the contrastive loss as:\n\nnumerator = $exp(sim(e_{x^i}, e_{z^i}))/\\tau$\n\ndenominator = $\\sum_j^Nexp(sim(e_{x^i}, e_{z^j})/\\tau$\n\n$\\mathcal{L}_{ctr}$  = - $\\sum_i^{N}$log$\\frac{numerator}{denominator}$\n\nwhere $x^i$ is the original source sentence, $z^i$ is the augmented version of $x^i$. Therefore, **the positive sample is the corresponding augmented sentence, while the negative samples are the augmented versions of other original source sentences from the same mini-batch.**\n\nWe have made a modification to Section 4.4 in our updated draft for a clear explanation.\n\n---\n\n> Reasons To Reject: Please provide a more detailed description on cleaning data by leveraging the large language models.\n\nThank you for raising the question regarding the details of how we use LLM for data cleaning in our work. We recognize the need for more explicit elucidation and will add further clarifications in the revised paper.\n\n1. **Method variants**: In our approach, we have three cleaning methods tailored to different data scenarios: Bilingual, Monolingual, and Translation. These methods leverage the capabilities of the LLM to produce clean text that preserves the essential information in noisy data.\n\n* **Bilingual**: For bilingual cleaning, we input both the noisy source and target samples. The aim is to generate a clean target text that maintains its alignment with the noisy source.\n* **Monolingual**: For this approach, we only provide a noisy target text and the LLM generates a cleaner version of it.\n* **Translation**: This technique utilizes the LLM to translate a noisy source text into a clean target text, effectively ignoring the source’s noise.\n\n2. **Chain-of-thought prompting**: We leverage recent advancements in prompting methods to improve our LLM’s understanding of what we aim to achieve. Through chain-of-thought prompting, we can guide the model’s behavior and optimize its responses.\n\n3. **Prompt design**: Our prompts contain multiple components, including the description of the context, method formulation, desired output framework, few-shot examples, and then the input samples. This composite prompt design improves the cleaning operation by making it more aligned with our objectives.\n\n4. **Semantic similarity**: To ensure that the cleaned text retains the intended meaning of the original text, we compute semantic similarity scores using LASER. If the similarity score is below a certain threshold, we iterate the cleaning process with additional guidance to the model.\n\n5. **Feedback loop with LASER**: In cases where the semantic similarity score falls below our threshold (0.7), we modify the prompt to include additional guidance, ensuring the cleaned text better aligns with the original meaning. We iterate this process until the threshold is met.\n\nIn summary, our approach leverages both the versatility of LLMs and semantic similarity checks to achieve effective and meaningful data cleaning. Additional computational details and algorithmic procedures will be incorporated in the revision to provide a thorough understanding of our data-cleaning process.\n\n---\n\n> Reasons To Reject: It is hard to understand the conclusion in lines 505-515.\n\nThe logic behind the conclusion in Lines 505-515 is not straightforward. We offer a more clear explanation here. \n\nOur goal in Section 4.5 is to show which test set is more suitable as a noise evaluation dataset. However, it's not easy to compare different test sets, since they have different sentences. For simplicity, we take the original noisy MTNT test set (termed as MTNT) and MTNT cleaned by bilingual method (termed as C-MTNT) as examples. MTNT and C-MTNT share the same noisy source sentences, but C-MTNT has a cleaner version of target sentences. If a model obtains a better BLUE score on MTNT than on C-MTNT, can we say that the model performs better on MTNT? No, we can't say that, because the target sentences of MTNT and C-MTNT are different. After cleaning, their target sentence length, noise level, and so on are different. \n\nTherefore, we focus on the relative improvement $G$ rather than the absolute improvement. If a model originally performs worse on C-MTNT than on MTNT, and achieves a better BLEU score on C-MTNT after augmented training, we can say that the augmented training offers more gain to C-MTNT. It also demonstrates that the distribution of the augmented dataset is more similar to C-MTNT, so the model has more gain on C-MTNT.\n\nLet's review how we augment our dataset. We only input noise, like character augmentation, to the source sentence while keeping the target sentence clean. This is the key motivation for the whole paper. I.e. we want a model that can generate clean target sentences from noisy source sentences. If a model trained on this augmented dataset achieves a better relative improvement on a test set, we can presume that this test set and the augmented dataset have similar distributions. I.e. most noise is located in the source side rather than in the target side. And this test set is exactly what we want.\n\nAccording to Figure 4, C-MTNT by bilingual or translation cleaning is our final ideal test set, since the model trained on the augmented dataset offers the highest relative improvement on them. We have added a more clear explanation to this conclusion in our updated draft.\n\n---\n\n> Reasons To Reject: It is necessary to introduce human evaluation to verify that the new dataset is better.\n\nWe totally agree with this point.\n\nIn this rebuttal week, we conducted an emergency human and GPT-4 evaluation experiment on C-MTNT. For your convenience, we first show the conclusion here: **Overall, human and GPT-4 evaluation share a similar preference: bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods.** \n\nThe experimental setting is:\n\n(1) 100 sentences from C-MTNT Fr->En are sampled to generate three files. These three files are about a binary comparison of bilingual vs. monolingual, bilingual vs. translation, and monolingual vs. translation. The order of sentences, including their indexes and which cleaning method comes first, is randomly shuffled. There is no chance for the annotator to guess which sentence corresponds to which method, and which file corresponds to the comparison between which two methods. (Notably, we prefer binary comparison to ranking over three methods, since it's easier for human annotators. In addition, the cleaned sentences from the correction tool are excluded, since they are too easy to be beaten.)\n\n(2) Four annotators (authors of this paper) are asked to give their preferences for each comparison, based on our three criteria of comprehensive noise removal, semantic preservation, and alignment with the source sentence (Lines 90-103). If both sentences show a similar level of noise, they are asked to give a \"Tie\".\n\n(3) Due to the time limit, we couldn't compare all sentences from different translation directions and hired a larger amount of annotators. To overcome this limit, we also prompt GPT-4 to do the comparison on all sentences on the Fr<->En corpora. En->Ja is excluded since GPT-4 is not good at Japanese, as stated in our paper (Line 314-322). We append the prompt at the end of this response, i.e. Appendix A.\n\nHere is the result from human evaluation on the sampled 100 sentences of Fr->En.  \n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: |:---:| --- |\n| Bilingual | **30.0%** | 46.8% | 23.2% | Monolingual |\n| Bilingual | **32.2%** | 38.0% | 29.8% | Translation |\n| Monolingual | 27.0% | 39.5% | **33.5%** | Translation |\n\nThe GPT-4 evaluation result of these 100 samples is:\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: |:---:| --- |\n| Bilingual | **34.0%** | 37.0% | 29.0% | Monolingual |\n| Bilingual | **40.0%** | 26.0% | 34.0% | Translation |\n| Monolingual | 35.0% | 21.0% | **44.0%** | Translation |\n\nOverall, human and GPT-4 evaluations share a similar preference: bilingual > translation > monolingual. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods. \n\nThe results from the GPT-4 evaluation on the whole C-MTNT are:\n\nEn -> Fr\n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: |:---:| --- |\n| Bilingual | **42.3%** | 32.0% | 25.7% | Monolingual |\n| Bilingual | **45.8%** | 21.2% | 33.0% | Translation | \n| Monolingual | 31.3% | 16.8% | **51.9%** | Translation | \n\nFr -> En\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: |:---:| --- |\n| Bilingual | **35.5%** | 35.7% | 28.8% | Monolingual |\n| Bilingual | **40.2%** | 22.8% | 37.0% | Translation |\n| Monolingual | 35.4% | 15.3% | **49.3%** | Translation |\n\nThe conclusion is similar to the above discussion, i.e. bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. We have added these new results to our updated draft. And the newly collected evaluation scores will also be released later.\n\n---\n\n> Reasons To Reject: For the baseline, why did you just use the language_tool_python module? Why didn't you use the emoji library, better_profanity, and profanity_check?\n\nFor our baseline, we prioritized tools that not only identify errors but also provide corrections. The language_tool_python module offers this feature. \n\nUsing better_profanity, profanity_check, or simply removing profanities can significantly alter the sentence's meaning, which deviates from our criteria of semantic preservation and alignment with the source sentence (Lines 96-103). \n\nSimilarly for the emoji, the emoji library can only detect emojis without offering replacement/correction. Simply removing them presupposes they don't add value to the sentence. It is a presumption we didn't want to make, because they do offer meaning to the sentences according to our observation, mostly strengthening the emotion.\n\n---\n\n> Questions: Why did not introduce the request in lines 260-261 in the first place? What is the maximum number of iterations?\n\nThis is a really good observation. Originally, we prompted GPT-3.5 to generate C-MTNT with different cleaning methods without the request. When we checked the generated sentences, we realized that GPT-3.5  unintentionally hallucinates, especially for monolingual cleaning. We didn't want to waste the clean generation from the first iteration. Therefore, we added this request at the end to re-emphasize our desired output for other iterations.\n\nIn addition, another benefit of adding the request afterward is to minimize the chances of receiving repetitive responses from the model. While it's not a guarantee that the model will always produce the same answer, the modification helps in reducing redundancy. It's reminiscent of the saying, \"The definition of insanity is doing the same thing and expecting different results.\" \n\nThough there were no cases of the model exceeding this limit, we set **a limit of 10 tries (iterations)** per sentence to avoid excessive API calls.\n\n---\n\n> Presentation Improvements: The text-davinci-003 model is the instructGPT model, which introduces SFT and RLHF.\n\nThank you for this correction, we already made an update to our draft.\n\n---\nAppendix A: Prompt for GPT-4 evaluation\n```\ndef generate_prompt(source, target, v1, v2):\n    return \"\"\"In the following, I'm going to show you one noisy source sentence in French and one noisy target sentence in English. In addition, I also offer you two clean versions of the noisy target sentence. \n\nCan you rank these two clean target sentences based on these three criteria:\n1. Comprehensive noise removal: All forms of noise should be eliminated from the noisy target sentence, including removing semantically meaningless emojis, translating emojis with semantic content into words, correcting misspellings, etc.\n2. Semantic preservation: The clean target sentence should retain similar semantic information as the original noisy target sentence.\n3. Alignment with the source sentence: The clean target sentence should convey the same intended meaning as the noisy source sentence, ensuring accurate translation and faithful representation of the original content.\n\nNoisy source sentence in French: {0}\nNoisy target sentence in English: {1}\nThe first clean target sentence: {2}\nThe second clean target sentence: {3}\n\nFor your output, you don't need to give any explanation. If the first version is better, you output 1. If the second version is better, you output 2. If they are equally clean, you output 3.\"\"\".format(source, target, v1, v2)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A cleaned datatset is helpful for exploring the application of machine translation in real scenarios.", "reasons_to_reject": "The paper should be strengthened in the writting, which is not fully clear now.  For contrastive learning, which is the positive sample? Which is the negative sample? Please explain in detail in Section 4.4. Please provide a more detailed description on cleaning data by leveraging the large language models.\nIt is hard to understand the conclusion in lines 505-515. It is necessary to introduce human evaluation to verify that the new dataset is better.\nFor the baseline, why did you just use language_tool_python module? Why didn't you use the emoji library, better_profanity, and profanity_check?\n\n", "questions_to_authors": "Why did not introduce the request in lines 260-261  in the first place?   What is the maximum number of iterations?", "paper_topic_and_main_contributions": "Neural machine translation models exhibit a noticeable decline in translation quality when exposed to noisy input. The MTNT dataset (Michel and Neubig, 2018) is widely used as a benchmark for evaluating the robustness of NMT models against noisy input. This paper argues that the noise exists not only in the source input but also in the target output, which affects the use of this dataset. Therefore, this paper proposes to clean the noise from the target sentences in MTNT by leveraging the large language models.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The text-davinci-003 model is the instructGPT model, which introduces SFT and RLHF."}, "extra_text": {}}, {"review_id": "viw5tbv81i", "reviewer": "Reviewer_6pw7", "summary": "", "questions": "A) While this is not a decision changing question, I wanted to understand why the authors felt that changing emojis to a more meaningful representation is required (mentioned a few times: line 018, 094 etc.). Would it not hurt metric like BLUE when evaluating performance?\n\nB) I would love to hear from authors about their views on my criticism and hear their counterpoints so that I can reconsider my points allocation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n* Design suitable application case where few-shot MT can be particularly useful;\n* Offers well-set-up experiments to demonstrate the effectiveness of collected data, C-MTNT;\n* Is well written for quick understanding.\n\nWe have responded to your comments below and updated our draft accordingly. \n\nIf you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n\n---\n\n> Reasons To Reject: where the contribution of the paper lies.\n\nWe acknowledge that our work builds upon established techniques, but we'd like to highlight the novelty and advantages of our approach:\n\n* **Application novelty**: We deem both methodological and application novelty equally important. Compared to previous work [1] where few-shot learning is applied to prompt an LLM to obtain a better translation, we apply few-shot learning in building a new benchmark to evaluate a MT system, which is normally done by human annotators. We understand that some previous works, like Alpaca, already applied LLM to create a new training set. Compared to them, our job focuses on data cleaning, a more challenging task that requires the understanding of the whole sentence while still being able to detect noise and offer correction. We didn't find any related work in this specific area with LLM (We're glad for your references), since data cleaning normally requires extensive work of human annotators.\n\n* **New way to construct prompt examples**: We are really sorry for forgetting to include this novel part in the paper. Compared to chain-of-thought where human annotators are asked to offer some representative few-shot examples, the prompt examples in our paper are generated by GPT-3.5 itself. If you are interested in how we prompt GPT-3.5 to generate these examples, we kindly ask you to go to our response to Reviewer pKJv on \"Question 3. How are the prompt examples selected and how sensitive is the LLM to the choice of these examples?\". This part was already added to our updated draft.\n\n* **New benchmark**: Our original goal of this research was to evaluate the robustness of different MT systems against noisy input. However, the most suitable benchmark we could find is MTNT where noise is almost equally distributed in both source and target sides, which is not desired for real-life scenarios, like e-commerce. By designing three cleaning methods, we offer a new benchmark to evaluate the robustness of MT systems. In addition, these three methods are also suitable for different scenarios. When only source sentences are available, translation cleaning is a better choice. When both source and target sentences are available, bilingual cleaning is a better choice.\n\n* **Thorough evaluation of the new benchmark**: In the paper, we offer multiple ways to evaluate the effectiveness of the newly constructed benchmark: (1) Measure the semantic similarity between noisy and cleaned target sentences in Figure 2; (2) Quantifiably measure the amount of different types of noise in the benchmark in Table 2; (3) Conduct MT experiments to show the effectiveness of the newly constructed benchmark; (4) Thanks to your advice, we also include human and GPT-4 evaluation. You can find more details in the following question.\n\n---\n\n> Reasons To Reject: The baseline of using language tool python seems way too easy for large language models to beat, rather, an effective baseline to beat or just evaluate would have been a fair human correction/ evaluation.\n\nTraditional cleaning method serves more as a threshold in our work instead of a strong baseline. Similar to Table 2 where we include the noise distribution in clean benchmarks, we want to show a lower bound for the reader's easy comparison. Self-comparison among different cleaning methods, like monolingual, bilingual, and translation, are our focus.\n\nThanks to your advice, we conducted an emergency human and GPT-4 evaluation experiment on C-MTNT. For your convenience, we first show the conclusion here: **Overall, human and GPT-4 evaluation share a similar preference: bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods.** \n\n\nThe experimental setting is:\n\n(1) 100 sentences from C-MTNT Fr->En are sampled to generate three files. These three files are about the binary comparison of bilingual vs. monolingual, bilingual vs. translation, and monolingual vs. translation. The order of sentences, including their indexes and which cleaning method comes first, is randomly shuffled. There is no chance for the annotator to guess which sentence corresponds to which method, and which file corresponds to the comparison between which two methods. (Notably, we prefer binary comparison to ranking over three methods, since it's easier for human annotators. In addition, the cleaned sentences from the correction tool are excluded, since they are too easy to be beaten.)\n\n(2) Four annotators (authors of this paper) are asked to give their preferences for each comparison, based on our three criteria of comprehensive noise removal, semantic preservation, and alignment with the source sentence (Lines 90-103). If both sentences show a similar level of noise, they are asked to give a \"Tie\".\n\n(3) Due to the time limit, we couldn't compare all sentences from different translation directions, and hire a larger amount of annotators. To overcome this limit, we also prompt GPT-4 to do the comparison on all sentences on the Fr<->En corpora. En->Ja is excluded since GPT-4 is not good at Japanese, as stated in our paper (Line 314-322). We append the prompt at the end of this response, i.e. Appendix A.\n\nHere is the result from human evaluation on the sampled 100 sentences of Fr->En.  \n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **30.0%** | 46.8% | 23.2% | Monolingual |\n| Bilingual | **32.2%** | 38.0% | 29.8% | Translation |\n| Monolingual | 27.0% | 39.5% | **33.5%** | Translation |\n\nThe GPT-4 evaluation result of these 100 samples is:\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **34.0%** | 37.0% | 29.0% | Monolingual |\n| Bilingual | **40.0%** | 26.0% | 34.0% | Translation |\n| Monolingual | 35.0% | 21.0% | **44.0%** | Translation |\n\nOverall, human and GPT-4 evaluations share a similar preference: bilingual > translation > monolingual. Compared to GPT-4, human annotators prefer to vote for \"Tie\". We argue the main reason is that most cleaned sentences are very similar with a low level of noise, which further justifies the effectiveness of our proposed methods. \n\nThe results from the GPT-4 evaluation on the whole C-MTNT are:\n\nEn -> Fr\n\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **42.3%** | 32.0% | 25.7% | Monolingual |\n| Bilingual | **45.8%** | 21.2% | 33.0% | Translation | \n| Monolingual | 31.3% | 16.8% | **51.9%** | Translation | \n\nFr -> En\n|  | Left is better | Tie | Right is better | | \n| --- | :---: | :---: | :---: | --- |\n| Bilingual | **35.5%** | 35.7% | 28.8% | Monolingual |\n| Bilingual | **40.2%** | 22.8% | 37.0% | Translation |\n| Monolingual | 35.4% | 15.3% | **49.3%** | Translation |\n\nThe conclusion is similar to the above discussion, i.e. bilingual > translation > monolingual, which is also similar to our originally drawn conclusion from Table 3 in the paper. We have added these new results to our updated draft. And the newly collected evaluation scores will also be released later.\n\n---\n\n> Reasons To Reject: the paper misses how sensitive the cleaning experiments could be based on what prompting strategies that they follow and how much hallucinations these models suffer from if the focus was just to show the effectiveness of few shot capabilities for correction.\n\nWe agree with this point since they are well-known issues for LLMs. We conduct some experiments to answer your concerns.\n\n1. **The sensitivity of different prompting strategies**\n\nWe conduct new experiments by designing new prompt examples. Similar to the human evaluation part, we only collect 100 clean sentences on Fr->En with the new prompt examples, keeping the other parts of Figure 1 the same for the ablation study. After collection, we use the same prompt in Appendix A to ask GPT-4 to give its preference, since it shares a similar preference as human annotators.\n\nFirst, we design two new sets of prompt examples for the bilingual cleaning method to check whether new prompt examples cause large variances for the same cleaning method. The results are:\n| | Left is better | Tie | Right is better |  |\n| --- | --- |--- |--- |--- |\n| Original prompt examples | 18% | 60% | 12% | New prompt examples v1 |\n| Original prompt examples | 20% | 53% | 27% | New prompt examples v2 |\n\nWe do see some variance with different prompt examples. However, the major preference (>50%) of GPT-4 is located in \"Tie\", indicating that the same cleaning method with different prompt examples offers similar cleaned sentences.\n\nSecond, we design two more sets of prompt examples for the monolingual and translation cleaning methods to check whether we have a similar pattern as the previous human/GPT-4 evaluation. The results are:\n| | Left is better | Tie | Right is better |  |\n| --- | --- |--- |--- |--- |\n| Original bilingual | **36%** | 40% | 24% | Monolingual with new prompt examples |\n| Original bilingual | **34%** | 35% | 31% | Translation with new prompt examples |\n| Monolingual with new prompt examples| 29% | 31% | **40%** | Translation with new prompt examples |\n\nA similar pattern, i.e. bilingual > translation > monolingual, is observed. **In sum, different prompt examples will cause a mild variance, but the observed pattern is similar.**\n\n\n2. **How many hallucinations do these models suffer from**\n\nWe did see some hallucinations from the cleaned sentences. It happens more frequently for the monolingual cleaning method since there is no supervision from the target side. One trick we applied to reduce this issue is including the LASER score (Lines 258-264). If the LASER score between the cleaned and noisy sentences is too low (<0.7), we repeat the API call till it is larger than 0.7. In this way, we offer some room for cleaning while still requiring a certain degree of similarity between the cleaned and noisy sentences to avoid hallucinations. We are open to other suggestions.\n\n---\n\n> Reasons To Reject: The authors use a single BLUE metric for their evaluations. While they do show improvements on BLUE based metrics, it would have been interesting to see how the performance corresponded to LM based evaluations metrics like BLEURT-20, COMET-QE.\n\nWe totally agree with this point, since LM-based metrics show more correlation with human preference. Apart from these LM-based metrics, we also offer GPT-4 evaluation in the above question, which shows a similar finding as the BLEU metric. The results of other LM-based metrics will be included in our next updated version.\n\n---\n\n> Reasons To Reject: authors could make further improvements in their work by having leveraged the latest research on parallel corpus that already make certain perturbations in addition to MTNT and give a fair comparison on how their approach is able to perform on different augmentations in addition to the ones that they have used. There is a body of multiple similar work that authors can take a look at and evaluate how they can leverage for improving their experiments.\n\nThis is a very inspired suggestion. We do find two of your offered references [2, 3] quite related. The implementation/discussion/comparison of these two works is on our to-do list.  \n\n---\n\n> Reasons To Reject: Authors might also want to cite some of the works in the field which I found was missing.\n\nReally thank you for the offered references [1, 2, 3, 4]. We find all of them very related and already included them in our related work.\n\n---\n\n> Question A. why the authors felt that changing emojis to a more meaningful representation is required (mentioned a few times: line 018, 094 etc.). Would it not hurt metric like BLUE when evaluating performance?\n\nThis is a very good question. Whether emojis are noise depends on the application case (similar to slang). For the social media use case, we think emojis are not necessarily a noise. However, for some official use cases, like the product title in e-commerce, emojis are considered as noise. In our work, we prefer to treat emojis as noise, because (1) It makes our task more challenging, which could explore the potential of LLM; (2) We want to expand the applicable case of our benchmark, making it a suitable benchmark for different MT systems. \n\nWe believe changing emojis to some meaningful representation would hurt the metric, like BLEU, because it is easier for a MT model to directly copy the emoji from the source sentence to the target sentence (emojis are normally the same on both sides) than to translate the emojis to some words according to different contexts. This is also one reason why we care more for the relative improvement rather than the absolute improvement in Section 4.5. However, we still think it is reasonable for such changing: (1) Making our benchmark more challenging; (2) Making our benchmark more applicable for different MT systems.\n\n---\n    \n> Missing References: I feel these citations for these works should also be present in the final work.\n\nThank you for these references, we already added them [1, 2, 3, 4] to our related work in an updated draft.\n\n---\n\nReferences\n\n[1] Garcia, X., Bansal, Y., Cherry, C., Foster, G., Krikun, M., Johnson, M., & Firat, O. (2023, July). The unreasonable effectiveness of few-shot learning for machine translation.\n\n[2] Karpinska, M., Raj, N., Thai, K., Song, Y., Gupta, A., & Iyyer, M. (2022). DEMETR: Diagnosing evaluation metrics for translation. \n\n[3] Jwalapuram, P. (2023, July). Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation. \n\n[4] Riley, P., Dozat, T., Botha, J. A., Garcia, X., Garrette, D., Riesa, J., ... & Constant, N. (2022). Frmt: A benchmark for few-shot region-aware machine translation.\n\n---\nAppendix A: Prompt for GPT-4 evaluation\n```\ndef generate_prompt(source, target, v1, v2):\n    return \"\"\"In the following, I'm going to show you one noisy source sentence in French and one noisy target sentence in English. In addition, I also offer you two clean versions of the noisy target sentence. \n\nCan you rank these two clean target sentences based on these three criteria:\n1. Comprehensive noise removal: All form of noise should be eliminated from the noisy target sentence, including removing semantically meaningless emojis, translating emojis with semantic content into words, correcting misspellings, etc.\n2. Semantic preservation: The clean target sentence should retain similar semantic information as the original noisy target sentence.\n3. Alignment with the source sentence: The clean target sentence should convey the same intended meaning as the noisy source sentence, ensuring accurate translation and faithful representation of the original content.\n\nNoisy source sentence in French: {0}\nNoisy target sentence in English: {1}\nThe first clean target sentence: {2}\nThe second clean target sentence: {3}\n\nFor your output, you don't need to give any explanation. If the first version is better, you output 1. If the second version is better, you output 2. If they are equally clean, you output 3.\"\"\".format(source, target, v1, v2)\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates the effectiveness of large language models in few-shot machine translation tasks and designs experiments to demonstrate the cases where they can be particularly useful (they show sometimes monolingual C-MTNT does not work as good as expected and without controls LLMs tend to hallucinate sometimes which is a known issue)\n\nOne more job that the paper does well is in setting up experiments with language model training to demonstrate the effectiveness of C-MTNT in evaluating model's robustness for machine translations tasks. Instead of directly comparing on BLUE score, authors use relative performance gain in BLEU score which to me seemed like a fairer evaluation metric overall. \n\nOverall, the paper seems to be well written for quick understanding of what the authors are trying to achieve.", "reasons_to_reject": "While the paper has its merits there are many significant reasons for rejections. The authors' three approaches that they claim as experiments (line 112 to 116) are nothing but few shot NMT capabilities of large language models and hence while designing a modified version of C-MTNT does seem a fair step, I am not sure where the contribution of the paper lies. Also, the paper misses how sensitive the cleaning experiments could be based on what prompting strategies that they follow and how much hallucinations these models suffer from if the focus was just to show the effectiveness of few shot capabilities for correction. \n\nThe baseline of using language tool python seems way too easy for large language models to beat, rather, an effective baseline to beat or just evaluate would have been a fair human correction/ evaluation. \n\nThe authors use single BLUE metric for their evaluations. While they do show improvements on BLUE based metrics, it would have been interesting to see how the performance corresponded to LM based evaluations metrics like BLEURT-20, COMET-QE. \n\nAlso, authors could make further improvements in their work by having leveraged the latest research on parallel corpus that already make certain perturbations in addition to MTNT and give a fair comparison on how their approach is able to perform on different augmentations in addition to the ones that they have used (synonyms, character augmentations). Some of the recent work that I got to read was (https://aclanthology.org/2023.findings-acl.381.pdf, https://aclanthology.org/2022.emnlp-main.649.pdf etc.) There is a body of multiple similar work that authors can take a look at and evaluate how they can leverage for improving their experiments. \n\nAuthors might also want to cite some of the works in field which I found was missing.", "questions_to_authors": "A) While this is not a decision changing question, I wanted to understand why the authors felt that changing emojis to a more meaningful representation is required (mentioned a few times: line 018, 094 etc.). Would it not hurt metric like BLUE when evaluating performance?\n\nB) I would love to hear from authors about their views on my criticism and hear their counterpoints so that I can reconsider my points allocation.", "paper_topic_and_main_contributions": "Machine Translation approaches and their evaluations may suffer from noisy source as well as target sentences thereby limiting their performances. The paper addresses this challenge by leveraging a large language model (GPT-3.5) to clean the target side of the MTNT dataset and they present a cleaner version called C-MTNT which could be a good resource for the MT community based on authors' claims. The paper also demonstrates the few shot capabilities of large language models and how they can effectively be leveraged for generating parallel corpora tailored to MT use cases for a robust evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None to my notice"}, "extra_text": {"Missing_References": "I feel these citations for these works should also be present in the final work, but this is up to author's discretion:\n1) https://arxiv.org/abs/2302.01398\n2) https://aclanthology.org/2022.emnlp-main.649/\n3) https://aclanthology.org/2023.findings-acl.381.pdf\n4) https://arxiv.org/pdf/2210.00193.pdf"}}]

Author rebuttal:

Meta-review: This paper presents a method using an LLM (GPT) to automatically remove “noise” from the target side of noisy parallel data (the MTNT dataset, en-fr and en-ja), to make the dataset more useful for testing the robustness of MT models. They compare the performance on 3 input types: monolingual (just the target sentence), bilingual (provide both source and target) and translation (provide just the source sentence). They focus the evaluation on 4 known types of “noise”: spelling/grammar errors, emojis, slang and profanities and compare again a rule-based correction tool. The best method depends on the noise type (e.g. bilingual working best for emojis/slang and the correction tool is best for spelling/grammar errors). They then test the new corpus C-MTNT on the MT task and show that the bilingual method leads to the greatest gain in performance when training MT models to produce clean target sentences from noisy source data. 

The paper’s motivation was well appreciated by the reviewers and the experiments are interesting and useful for future research. The authors’ extensive rebuttal provided additional experimental details and justification some design choices, answering most of the reviewers’ comments. The paper was initially lacking some human evaluation which was provided during the rebuttal. I recommend including all this in the paper.
Decision: Accept-Findings