Venue: NAACL 2021

Autocorrect in the Process of Translation -- Multi-task Learning Improves Dialogue Machine Translation

Tao Wang1,2, Chengqi Zhao1, Mingxuan Wang1, Lei Li1, Deyi Xiong3\({}^{*}\)

###### Abstract

Automatic translation of dialogue texts is a much needed demand in many real life scenarios. However, current neural machine translation systems usually deliver unsatisfying translation results of dialogue texts. In this paper, we conduct a deep analysis of a dialogue corpus and summarize three major issues on dialogue translation, including pronoun dropping (ProDrop), punctuation dropping (PunDrop), and typos (DialTypo). In response to these challenges, we propose a joint learning method to identify omission and typo in the process of translating, and utilize context to translate dialogue utterances. To properly evaluate the performance, we propose a manually annotated dataset with 1,931 Chinese-English parallel utterances from 300 dialogues as a benchmark testbed for dialogue translation. Our experiments show that the proposed method improves translation quality by 3.2 BLEU over the baselines. It also elevates the recovery rate of omitted pronouns from 26.09% to 47.16%. The code and dataset are publicly available at [https://github.com/rgwt123/DialogueMT](https://github.com/rgwt123/DialogueMT).

BytcDance AI Lab +
Footnote †: Corresponding author.

\begin{tabular}{l|p{156.8pt}}  & Tao Wang1,2, Chengqi Zhao1, Mingxuan Wang1, Lei Li1, Deyi Xiong3\({}^{*}\)\\   \[{}^{1}\]ByteDance AI Lab \\   \[{}^{2}\]School of Computer Science and Technology, Soochow University, Suzhou, China \\   \[{}^{3}\]College of Intelligence and Computing, Tianjin University, Tianjin, China \\   {wangtao.960826, zhaochengqi.d, wangmingxuan.89}@bytedance.com \\   {lilei.02}@bytedance.com \\ dyxiong@tju.edu.cn \\ \end{tabular}

Footnote 1: footnotemark:

Footnote 2: Corresponding author.

## 1 Introduction

Remarkable progress has been made in Neural Machine Translation (NMT) Bahdanau et al. (2015); Wu et al. (2016); Lin et al. (2020); Liu et al. (2020) in recent years, which has been widely applied in everyday life. A typical scenario for such application is translating dialogue texts, in particular the record of group chats or movie subtitles, which helps people of different languages understand cross-language chat and improve their comprehension capabilities.

However, traditional NMT models translate texts in a sentence-by-sentence manner and focus on the formal text input, such as WMT news translation Barrault et al. (2020), while the translation of dialogue must take the meaning of context and the input noise into account. Table 1 shows examples of dialogue fragment in Chinese and their translation in English. Example (1) demonstrates that the omission in traditional translation (e.g., dropped pronouns in Chinese) leads to inaccurate translation results.

Despite its vast potential application, efforts of exploration into dialogue translation are far from enough. Existing works Wang et al. (2016); Maruf et al. (2018) focus on either extracting dialogues from parallel corpora, such as OpenSubtitles Lison et al. (2019), or leveraging speaker information for integrating dialogue context into neural models. Also, the lack of both training data and benchmark test set makes current dialogue translation models far from satisfying and need to be further improved.

In this paper, we try to alleviate the aforementioned challenges in dialogue translation. We first analyze a fraction of a dialogue corpus and summarize three critical issues in dialogue translation, including ProDrop, PunDrop, and DialTypo. Then we design a Multi-Task Learning (MTLdial) approach that learns to self-correct sentences in the process of translating. The model's

\begin{table}
\begin{tabular}{l l} \hline (1) & Nancy\({}^{3}\!\! 

encoder part automatically learns how to de-noise the noise input via explicit supervisory signals provided by additional contextual labeling. We also propose three strong baselines for dialogue translation, including repair (REPAIRdial) and robust (ROBUSTdial) model. To alleviate the challenges arising from the scarcity of dialogue data, we use sub-documents in the bilingual parallel corpus to enable the model to learn from cross-sentence context.

Additionally as for evaluation, the most commonly used BLEU metric Papineni et al. (2001) for NMT is not good enough to provide a deep look into the translation quality in such a scenario. Thus, we build a Chinese-English test set containing sentences with the issues in ProDrop, PunDrop and DialTypo, attached with the human translation and annotation. Finally, we get a test set of 300 dialogues with 1,931 parallel sentences.

The main contributions of this paper are as follows: a) We analyze three challenges ProDrop, PunDrop and DialTypo, which greatly impact the understanding and translation of a dialogue. b) We propose a contextual multi-task learning method to tackle the analyzed challenges. c) We create a Chinese-English test set specifically containing those problems and conduct experiments to evaluate proposed method on this test set.

## 2 Analysis on Dialogue Translation

There were already some manual analyses of translation errors, especially in the field of discourse translation. Voita et al. (2019) study English-Russian translation and find three main challenges for discourse translation: deixis, ellipsis, and lexical cohesion. For Chinese-English translation, tense consistency, connective mismatch, and content-heavy sentences are the most common issues Li et al. (2014).

Different from previous works, we mainly analyze the specific phenomena in dialogue translation. We begin with a study on a bilingual dialogue corpus Wang et al. (2018).1 We translate source sentences into the target language at sentence level and compare translation results with reference at dialogue level. Around 1,000 dialogues are evaluated, and the results are reported in Table 2. From the statistic, we observe two persistent dialogue translation problems: pronoun dropping (ProDrop), punctuation dropping(PunDrop). The phenomenon is consistent with the issue we collect in practical Instant Messaging (IM) chat scenarios, except for typos since the analyzed dialogue corpus has been proofread to remove typos.

Footnote 1: [https://github.com/longyuewangdcu/tvsub](https://github.com/longyuewangdcu/tvsub)

### Pronoun Dropping

Pronouns are frequently omitted in pro-drop languages Huang (1989), such as Chinese, Japanese, Korean, Vietnamese, and Slavic languages. Such phenomenon are more frequent in dialogue, where the interlocutors are both aware of what's omitted in the context. However, when translating a pro-drop language into a non-pro-drop language (e.g., English)2, it is hard to translate those omitted pronouns, resulting in grammatical errors or semantic inaccuracies in the target language. The first conversation in Table 1 is an example.

Footnote 2: [https://en.wikipedia.org/wiki/Pro-drop_language](https://en.wikipedia.org/wiki/Pro-drop_language)

### Punctuation Dropping

In dialogue scenarios, such as IM software, punctuation is often omitted and users tend to segment sentences with spaces. The problem becomes much serious in languages with no spaces, such as Chinese, Japanese, Korean, and Thai. Table 1 shows this phenomenon in Example (2).

### Dialogue Typos

Typo repairing is another fundamental but very challenging practical problem. In dialogue translation, typos or misspellings are very common, which dramatically undermine the quality of translation output produced by machine translation. Table 1 shows this phenomenon in Example (3).

## 3 Approach to NMTDial

This section aims to propose a unified framework that facilitates NMT to correct noisy inputs in dialogue neural machine translation (NMTdial). The framework includes three different methods, which are REPAIRdial, ROBUSTdial and MTLdial.

\begin{table}
\begin{tabular}{|l|c|} \hline
**Types of phenomena** & **Frequency** \\ \hline Correct & 88.1\% \\ PreDrop & 4.3\% \\ PunDrop & 3.2\% \\ Incorrect segmentation & 2.4\% \\ Other translation errors & 2.0\% \\ \hline \end{tabular}
\end{table}
Table 2: Manual evaluation of dialogue samples.

 



### Contextual Perturbation Example Generation

The most challenging problem for NMTdial is the data distribution gap between training and inference stage, where the training data are clean sentence-level pairs while the test data are noisy dialogue-level conversations.

To bridge the distribution gap, the first step is to generate perturbation examples based on training instances. The data generation mainly consists of two steps. The first step is to obtain sub-documents with cross-sentence context, and the second step is to generate examples with word perturbations within sub-documents. Figure 0(a) shows a complete process.

**Cross-sentence Context** It is difficult to acquire dialog-level parallel training data. As an alternative approach, we use parallel document data to catch dependencies across sentences.

Formally, let \(x_{d}=\{x^{(1)},x^{(2)},\cdots,x^{(M)}\}\) be a source-language document containing \(M\) source sentences. And \(y_{d}=\{y^{(1)},y^{(2)},\cdots,y^{(M)}\}\) is the corresponding target-language document containing the same number of sentences as that of the source document. To get more context information, we randomly sample consecutive sub-document pairs \((x_{d},y_{d})\) of \(N\) sentences (i.e., snippet pairs from aligned documents). We set \(N\in[1,10]\) in this paper.

We use a special token <sep> as the separator to concatenate sentences into a parallel sub-document \(\{(x_{d},y_{d})\}\), as shown in Figure 0(a).

**Contextual Perturbation** We then consider generating perturbation example \(x^{\prime}_{d}\) from \(x_{d}\) with respect to sub-document context. For ProDrop, PunDrop and DialTypo, we build a Chinese pronoun table T\({}_{\texttt{ProDrop}}\), a common punctuation table T\({}_{\texttt{FunDrop}}\) and a Chinese homophone table T\({}_{\texttt{DialType}}\) respectively.

For ProDrop and PunDrop, we traverse source sentences of \(x_{d}\), discard pronouns/punctuation in these sentences with a probability of 30% and record deletion positions with corresponding labels (see details below); to construct a typo, we choose a word with a probability of 1%, of which 80% is replaced with one of its homophones according to T\({}_{\texttt{DialType}}\) and 20% is replaced with another random word. We determine these percentages by observing the generated perturbation data. For annotation labels, we tag correct words with 0, words of DialTypo with 1, ProDrop words with 2 and PunDrop words with 3.

Finally we get \(x_{d}\), \(x^{\prime}_{d}\) and their corresponding label sequences \(\ell_{x}\), \(\ell^{\prime}_{x}\). \(\ell_{x}\) is a sequence of all 0s.

### NMTdial Base Models

With the created training data, we first introduce two methods for NMTdial as our strong baselines, which will be elaborated here for model comparison.

**REPAIRdial** A natural way for NMTdial is to train a dialog repair model to transform dialogue inputs into forms that an ordinary NMT system can deal with. REPAIRdial involves training a repair model to transform \(x^{\prime}_{d}\) to \(x_{d}\) and a clean translation model that translates \(x_{d}\) to \(y_{d}\). As a pipeline method, REPAIRdial may suffer from error propagation.

**ROBUSTdial** We extend the robust NMT

Figure 1: Overall diagram of NMTdial. (a) demonstrates the process of data generation, and (b) displays the three proposed methods. 1⃝Z⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃝X⃃�X⃃X�X⃃X⃃X�X⃃X⃃X�X⃃X�X⃃X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�XX�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�X�XX�X�X�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XX�XXXX�XX�XXXX�XXXX�

(Cheng et al., 2018) to dialogue-level translation. Specifically, we take both the original \((x_{d},y_{d})\) and the perturbated \((x^{\prime}_{d},y_{d})\) bilingual pairs as training instances. So the model is more resilient on dialogue translation. During the inference stage, the robust model directly translates raw inputs into the target language.

### MTLdial

ROBUSTDial has the potential to handle translation problems caused by noisy dialogue inputs. However, the internal mechanism is rather implicit and in a black box. Therefore, the improvement is limited, and it is not easy to analyze the improvement. To address this issue, we introduce a context-aware multi-task learning method MTLdial for NMTdial.

As shown in 3 of Figure 0(b), the only difference is that we have a contextual labeling module based on the encoder. We denote the final layer output of the Transformer encoder as \(H\). For each token \(h_{i}\) in \(H=(h_{1},h_{2},...,h_{m})\), the probability of contextual labeling is defined as:

\[P(p_{i}=j|X)=softmax(W\cdot h_{i}+b)[j] \tag{1}\]

where \(X=(x_{1},x_{2},...,x_{m})\) is the input sequence, \(P(p_{i}=j|X)\) is the conditional probability that token \(x_{i}\) is labeled as \(j\) (\(j\in 0,1,2,3\) as defined above).

Here we make the labeling module as simple as possible, so that the Transformer encoder can behave like BERT Devlin et al. (2019), learning more information related to perturbation and guiding the decoder to find desirable translations.

During the training phrase, the model takes \((x_{d},x^{\prime}_{d},\ell_{x},\ell^{\prime}_{x},y_{d})\) as the training data. The learning process is driven by optimizing two objectives, corresponding to sequence labeling as auxiliary loss (\(\mathcal{L}_{SL}\)) and machine translation as the primary loss (\(\mathcal{L}_{MT}\)) in a multi-task learning framework.

\[\mathcal{L}_{SL}=-log(P(\ell_{x}|x_{d})+P(\ell^{\prime}_{x}|x^{ \prime}_{d})) \tag{2}\] \[\mathcal{L}_{MT}=-log(P(y_{d}|x_{d})+P(y_{d}|x^{\prime}_{d})) \tag{3}\]

The two objective are linearly combined as the overall objective in learning.

\[\mathcal{L}=\mathcal{L}_{MT}+\lambda\cdot\mathcal{L}_{SL} \tag{4}\]

\(\lambda\) is coefficient. During experiments, we set as follows according the best practice:

\[\lambda=max(1.0-\frac{update\_num}{10^{5}},0.2) \tag{5}\]

where \(update\_num\) is the number of updating steps during training.

We introduce multi-task learning for two reasons: 1) The labeling performance reflects the model's understanding of sentences containing the mentioned phenomena. 2) Contextual Labeling can be seen as a pre-training process based on the BERT-like model, and explicit guidance can enable the encoder to learn more about the information we annotate.

### Modeling Dialogue Context

The modes for exploring dialogue context during decoding can be divided into offline and online. For the offline setting, all sentences in a dialogue are concatenated one by one with <sep>. The concatenated sequence is translated, and the target translation for each sentence can be easily detected according to the separator <sep>.

The offline mode can be used for dialogue translation where the entire source dialogue has already been available before translation (e.g., movie subtitles). However, we continuously get new source sentences for online chat and need to generate corresponding translations immediately. We refer to this mode as the online setting.

We experiment with two online methods. One is _online-cut_ where the current sentence is concatenated to the previous context with the separator <sep>. The trained NMTdial model then translates the concatenated sequence and the last target segment is used as the translation for the current source sentence. The other is _online-fd_. _Online-fd_ is a force decoding method. It forces the decoder to use translated history and continues decoding instead of re-translating the entire concatenated sequence. _Online-fd_ brings more consistent translation.

## 4 Experiments

### Test Set

For better evaluation of NMTdial, we create a Chinese-English test set covering all issues discussed above based on the corpus we analyze in the second section. Statistics on the built test set are displayed in Table 3. Building such a test set is hard and time-consuming as we need to perform manual selection, translation and annotation.

As for translation quality evaluation, we use other metrics in addition to BLEU. For PunDrop and DialTypo, we evaluate BLEU scores on sen 

tences containing missing punctuation or typos according to the annotation information. As for ProDrop, we evaluate the translation quality by the percentage of correctly recovering and translating the dropped pronouns.

### Settings

We adopt the Chinese-English corpus from WMT20203, with about 48M sentence pairs, as our bilingual training data \(D\). We select newstest2019 as the development set. After splicing, we get \(D_{doc}\) with 1.2M pairs and corresponding perturbated dataset \(D^{\prime}\) and \(D^{\prime}_{doc}\) with 48M and 1.2M pairs respectively.

Footnote 3: This corpus includes News Commentary, Wiki Titles, UN Parallel Corpus, CCMT Corpus, WikiMatrix and Backtranslated news.

We use byte pair encoding compression algorithm (BPE) [5] to process all these data and limit the number of merge operations to a maximum of 30K. In our studies, all translation models are Transformer-big, including 6 layers for both encoders and decoders, 1024 dimensions for model, 4096 dimensions for FFN layers and 16 heads for attention.

During training, we use label smoothing = 0.1 [7], attention dropout = 0.1 and dropout [10] with a rate of 0.3 for all other layers. We use Adam [13] to train the NMT models. \(\beta 1\) and \(\beta 2\) of Adam are set to 0.9 and 0.98, the learning rate is set to 0.0005, and gradient norm 5. The models are trained with a batch size of 32,000 tokens on 8 Tesla V100 GPUs during training. During decoding, we employ beam search algorithm and set the beam size to 5. We use sacrebleu [20] to calculate uncased BLEU-4 [11].

### Results of Offline Setting

The offline mode aims at using the entire source dialogue for translation. We experiment with all the methods in the offline setting, and the results are shown in Table 4. BASE is a Transformer-big model trained with \(D\) and \(D_{doc}\). GOLD+BASE represents the oracle result on this test set. We can see that MTLdial has achieved the best results, reducing the gap between \(test_{wrong}\) and \(test_{gold}\) from 4.1 to 0.9. Compared with ROBUSTdial and MTLdial, REPAIRdial performs relatively poorly. We believe that this is due to the error propagation caused by the pipeline.

From the specific indicators, we can draw the following conclusions: 1) DialTypo has a very obvious impact on BLEU, and the gap between BASE and GOLD+BASE is more than 12 points; 2) The recovery of ProDrop is a relatively difficult task. Although compared with BASE, the current best result of 47.16% has been greatly improved, but is still far away from the golden result 97.32%; 3) PunDrop seems to be a relatively easy task for each method to address.

### Results of Online Setting

The online mode only makes use of previous context during translation. An extreme situation of online setting is that there is no context, that is, sentence-level translation. We show the results of all the methods on the test set at the sentence level in Table 5. Despite the lack of context, our approaches can still bring general benefits. We find that ProDrop relies heavily on context, especially for MTLdial, where the absence of context results in a 12.38% drop in performance. This is in line

\begin{table}
\begin{tabular}{l|c c c} \hline \multirow{2}{*}{Methods} & Overall & \multicolumn{3}{c}{Details} \\  & BLEU & ProDrop & PunDrop & DialTypo \\ \hline BASE & 32.7 & 26.09\% & 28.2 & 24.0 \\ REEPIdial & 34.0 & 29.77\% & 31.2 & 27.4 \\ RDUSTdial & 34.1 & 45.48\% & 33.0 & **28.8** \\ MTLdial & **35.9** & **47.16\%** & **34.3** & 28.7 \\ \hline \hline GOLD+BASE & **36.8** & **97.32\%** & **34.6** & **36.8** \\ \hline \end{tabular}
\end{table}
Table 4: Experiment results on our constructed dialogue translation test set in offline setting. The GOLD+BASE represents translations of completely correct inputs (without ProDrop, PunDrop or DialTypo) using BASE model, which is used to show the oracle results with Transformer on the test set.

\begin{table}
\begin{tabular}{l|c} \hline
**Item** & **Count** \\ \hline \#dialagues & 300 \\ \#sentence pairs & 1,931 \\ \#total tokens & 19,15/15,976 \\ \#average tokens & 9.92/8.27 \\ \#proDrop & 299 \\ \#PunDrop & 542 \\ \#DialTypo & 203 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics on the test set. “/” denote numbers in Chinese and English separately.

\begin{table}
\begin{tabular}{l|c|c c c} \hline \multirow{2}{*}{Methods} & Overall & \multicolumn{3}{c}{Details} \\  & BLEU & ProDrop & PunDrop & DialTypo \\ \hline BASE & 32.8(+0.1) & 19.06\%(+7.03\%) & 28.1(-0.1) & 22.3(-1.7) \\ REEPIdial & 33.8(-0.2) & 24.75\%(-5.02\%) & 32.0(+0.8) & 28.3(+0.9) \\ ROBUSTdial & 34.2(+0.1) & **36.79\%(+8.09\%)** & 32.7(-0.3) & **28.9(-0.5)** \\ MTLdial & **35.3(-0.6)** & 34.78\%(-12.3\%) & **34.3(-0.0)** & **28.6(-0.1)** \\ \hline \hline GOLD+BASE & **37.1(+0.3)** & **96.66\%(+0.66\%)** & **35.3(+0.7)** & **35.9(-0.9)** \\ \hline \end{tabular}
\end{table}
Table 5: Results on our constructed dialogue translation test set in online setting at the sentence level.

 

with our expectations, as in many cases machine translation system heavily depends on context to fulfill the dropped pronouns.

We further experiment on how context lengths can affect NMTDial. The results are shown in Figure 2. In the _online-cut_ setting, we can see that using previous few sentences as context may improve overall BLEU score, but continuously adding more preceding texts will lead to a continuous decline. _Online-fd_ performs well because using historical translation records to continue decoding can bring more consistent translation results. For the recovery accuracy of ProDrop, _online-cut_ is better than _online-fd_ in contrast, because forced decoding may cause wrong pronoun transmission.

## 5 Analysis

### Labeling Performance

To better understand how our proposed MTLdial make sense, we calculate the labeling performance on both validation and test set. Table 6 shows the overall performance. The validation set follows the same processing progress of training data, while the test set is the real dialogue data set built manually.

The proposed model obtains 54.3% F1 score on the validation set for ProDrop, 70.9% for PunDrop, and 73.2% for DialTypo. When testing on the real test data, the performance on ProDrop has declined a lot because of the difference between synthetic training/validation data and real test data. Especially noteworthy is the fact that F1 score of DialTypo drops the most, reaching 26%, because of its low recall. It may be due to the considerable difference between the typos generated by our automatic method and the actual distribution.

### Effects of Pronoun Correcting

We further explore the auto-correction of specific pronouns. As shown in Figure 3, we can find that pronouns such as I/you, which occur mostly in the corpus, generally have a higher recovery success rate. We believe this is due to the data imbalance. Compared with BASE, MTLdial has a much better performance. While ProDrop recovery accuracy has been improved, it still has not achieved 50%. The most common error is that the model does not capture any context or captures previous inappropriate context. We summarize frequently-occurring recovery errors in Table 7.

## 6 Related Work

Our work is related with both dialogue translation and robust training.

**Dialogue Translation**

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Data & & Precision & Recall & Fl \\ \hline \multirow{3}{*}{validation} & ProDrop & **61.3** & **48.7** & **54.3** \\ \cline{2-5}  & PunDrop & 80.0 & 63.6 & 70.9 \\ \cline{2-5}  & DialTypo & **85.3** & **64.2** & **73.2** \\ \hline \hline \multirow{3}{*}{test} & ProDrop & 48.6 & 32.2 & 38.8 \\ \cline{2-5}  & PunDrop & **96.6** & **87.9** & **92.1** \\ \cline{2-5}  & DialTypo & 83.3 & 31.0 & 45.2 \\ \hline \end{tabular}
\end{table}
Table 6: Labeling performance on the validation/test set.

Figure 3: ProDrop recovery performance of BASE and contextual MTLdial. Total means the total number of occurrence of corresponding pronouns in the test set. We ignore pronouns with a total occurrence number less than 5.

\begin{table}
\begin{tabular}{|l|p{142.3pt}|p{142.3pt}|} \hline  & en & \\ \hline (1) & What’s going on with Ellie? \\ \hline (2) & What’s going on hyper in my law firm. \\ \hline (3) & What’s any two’s going? \\ \hline (4) & Does asked/, I want to ask you something. \\ \hline (5) & How do they get an I/I like nearly lost anything. \\ \hline (6) & If Do they get an I/I like really lost anything. \\ \hline \end{tabular}
\end{table}
Table 7: Examples of ProDrop recovery errors.

Figure 2: Overall BLEU and ProDrop recovery performance (Accuracy) of MTLdial with different context length. Dash lines are the offline results.

 

There has been some work on building bilingual dialogue data sets for the translation task in recent years. Wang et al. (2016) propose a novel approach to automatically construct parallel discourse corpus for dialogue machine translation and release around 100K parallel discourse data with manual speaker and dialogue boundary annotation. Maruf et al. (2018) propose the task of translating Bilingual Multi-Speaker Conversations. They introduce datasets extracted from Europarl and Opensubtitles and explore how to exploit both source and target-side conversation histories. Bowden et al. (2019) present a new English-French test set for evaluating of Machine Translation (MT) for informal, written bilingual dialogue. Recently WMT2020 has also proposed a new shared task - machine translation for chats,4 focusing on bilingual customer support chats (Farajian et al., 2020).

Footnote 4: [http://www.statmt.org/wmt20/chat-task.html](http://www.statmt.org/wmt20/chat-task.html)

**Robust Training**

Neural models have been usually affected by noisy issues. Many efforts Li et al. (2017); Sperber et al. (2017); Vaibhav et al. (2019); Yang et al. (2020) focus on data augmentation to alleviate the problem by adding synthetic noise to the training set. However, generating noise has always been a challenge, as natural noise is always more diversified than artificially constructed noise Belinkov and Bisk (2018); Anastasopoulos (2019); Anastasopoulos et al. (2019).

## 7 Conclusions

In this paper, we manually analyze challenges in dialogue translation and detect three main problems. In order to tackle these issues, we propose a multi-task learning method with contextual labeling. For deep evaluation, we construct dialogues with translation and detailed annotations as a benchmark test set. Our proposed model achieves substantial improvements over the baselines. What is more, we further analyze the performance of contextual labeling and pronoun recovery errors.

## Acknowledgments

We thank the bilingual speakers for test set construction, and the anonymous reviewers for suggestions. Deyi Xiong is partially supported by the Natural Science Foundation of Tianjin (Grant No. 19JCZDJC31400) and the Royal Society (London) (NAF\R1\180122).

## References

* Anastasopoulos (2019) Antonios Anastasopoulos. 2019. An analysis of sourceside grammatical errors in nmt. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 213-223.
* Anastasopoulos et al. (2019) Antonios Anastasopoulos, Alison Lui, Toan Q Nguyen, and David Chiang. 2019. Neural machine translation of text from non-native speakers. In _NAACL-HLT (1)_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _3rd International Conference on Learning Representations, ICLR 2015_.
* Barrault et al. (2020) Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R Costa-jussa, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Jounis, et al. 2020. Findings of the 2020 conference on machine translation (wmtz0). In _Proceedings of the Fifth Conference on Machine Translation_, pages 1-55.
* Bawden et al. (2019) Rachel Bawden, Sophie Rosset, Thomas Lavergne, and Eric Bilinski. 2019. Diabla: A corpus of bilingual spontaneous written dialogues for machine translation. _arXiv preprint arXiv:1905.13354_.
* Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In _International Conference on Learning Representations_.
* Cheng et al. (2018) Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_, pages 1756-1766. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT (1)_.
* Farajian et al. (2020) M Amin Farajian, Antonio V Lopes, Andre FT Martins, Sameen Maruf, and Gholamreza Haffari. 2020. Findings of the wmt 2020 shared task on chat translation. In _Proceedings of the Fifth Conference on Machine Translation_, pages 65-75.
* Hinton et al. (2012) Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. _CoRR_, abs/1207.0580.
* Huang (1989) CT James Huang. 1989. Pro-drop in chinese: A generalized control theory. In _The null subject parameter_, pages 185-214. Springer 

* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of ICLR_.
* Li et al. (2014) Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014. Assessing the discourse factors that influence the quality of machine translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 283-288.
* Li et al. (2017) Yitong Li, Trevor Cohn, and Timothy Baldwin. 2017. Robust training under linguistic adversity. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 21-27.
* Lin et al. (2020) Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pre-training multilingual neural machine translation by leveraging alignment information. _arXiv preprint arXiv:2010.03142_.
* Lison et al. (2019) Pierre Lison, Jorg Tiedemann, Milen Kouylekov, et al. 2019. Open subtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In _LREC 2018, Eleventh International Conference on Language Resources and Evaluation_. European Language Resources Association (ELRA).
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:726-742.
* Maruf et al. (2018) Sameen Maruf, Andre FT Martins, and Gholamreza Haffari. 2018. Contextual neural model for translating bilingual multi-speaker conversations. _WMT 2018_, page 101.
* Papineni et al. (2001) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In _ACL_.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1715-1725.
* Sperber et al. (2017) Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In _International Workshop on Spoken Language Translation (IWSLT)_.
* Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 2818-2826.
* Vaibhav Saibhav et al. (2019) Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1916-1920.
* Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1198-1212.
* Wang et al. (2018) Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018. Translating pro-drop languages with reconstruction models. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Wang et al. (2016) Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, Andy Way, and Qun Liu. 2016. Automatic construction of discourse corpora for dialogue translation.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_.
* Yang et al. (2020) Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. 2020. Towards making the most of bert in neural machine translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 9378-9385.