PROSE: A Pronoun Omission Solution for Chinese-English Spoken Language Translation
Authors: Ke Wang
Xiutian Zhao
Yanghui Li
Wei Peng
Venue: EMNLP 2023 Main (2023)
Keywords: Chinese-English Spoken Language Translation
Zero-Pronoun
Mention-Aware Semantic Augmentation
TL;DR: Our study sheds light on the challenges posed by zero pronouns in Chinese-English spoken language translation and provides a valuable solution that can improve the performance of neural machine translation systems.

Abstract: Neural Machine Translation (NMT) systems encounter a significant challenge when translating a pro-drop ('pronoun-dropping') language (e.g., Chinese) to a non-pro-drop one (e.g., English), since the pro-drop phenomenon demands NMT systems to recover omitted pronouns. This unique and crucial task, however, lacks sufficient datasets for benchmarking. To bridge this gap, we introduce PROSE, a new benchmark featured in diverse pro-drop instances for document-level Chinese-English spoken language translation. Furthermore, we conduct an in-depth investigation of the pro-drop phenomenon in spoken Chinese on this dataset, reconfirming that pro-drop reduces the performance of NMT systems in Chinese-English translation. To alleviate the negative impact introduced by pro-drop, we propose Mention-Aware Semantic Augmentation, a novel approach that leverages the semantic embedding of dropped pronouns to augment training pairs. Results from the experiments on four Chinese-English translation corpora show that our proposed method outperforms existing methods regarding omitted pronoun retrieval and overall translation quality.

PROSE: A Pronoun Omission Solution for Chinese-English Spoken Language Translation
Ke Wang Xiutian Zhao' Yanghui Li
Wei Peng"
Huawei IT Innovation and Research Center (wangke215, zhaoxiutian, liyanghui, peng. wei )ehuawei com
Abstract
Input (ANEEE
-B
Neural Machine Translation (NMT) systems encounter sigmificant challenge when translat- ing pro-drop (' ('pronoun "prroouundroopingg" language (e-g., Chinese) a non-pro-drop one (e.g.- English). since the pro-drop phenomenon de- mands NMT systems to recover omitted pro- nouns. This unique and crucial task, how- ever, lacks sufficient datasets for benchmark- ing. To bridge this gap, we introduce PROSE, a new benchmark featured in diverse pro-drop instances for document-leve spoken language translation. Furthermore, we conduct in-depth investigation of the pro- drop phenomenon in spoken Chinese on this dataset, reconfirming that pro-drop reduces the performance of NMT systems in Chinese- English translation. To alleviate the nega- tive impact introduced by pro-drop, we pro- pose Mention-Awart Semantic Augmentation. a novel approach that leverages the semantic embedding of dropped pronouns augment training pairs. Results from the experiments on four Chinese-Fngie English translation corpora show that proposed method outperforms existing methods regarding omitted pronoun retrieval overall translation quality.
Google
Let's go, let's go eat togethen.
Let's go, let's eat togethen.
DeepL
Let's go and eat together.
Let's go, let's eat together.
Ours
Let's go out and have meal together.
Input
(60R
English
Google
You did good job this time, you in charge?
Transdate
Good this time, you led?
DecpL.
You a good job this time?
Nice job this time, did you lead it?
Ours
great job time. Was your lead?
Figure 1: Examples of pro-drop in daily spoken Chinese with corresponding English translations provided by our model and several mature commercial NMT systems (Google, DeepL, Baidu and ChatGPT, respectively, data collected on January 13th, 2023). The Chinese pronoums in brackets are dropped, and the dropped subject and ob- ject pronouns are marked red and green, respectively.
1 Introduction
and Koehn, 2010; Taira et al., 2012; Wang et al., 2016, 2018at 8a; Tan et al., 2019), advanced commer- cial NMT systems occasionally fail to faithfully recover dropped pronouns in the source language. In some cases, leaving missed pronouns unrecov- ered could result in severe semantic distortion and alter the intended meaning of the translated text, demonstrated in Figure 1.
In recent years, neural machine translation (NMT) technology has made significant progress low- ering communication barriers between individu- als from different language backgrounds. How- ever, NMT systems often struggle when translating sentences from a pro-drop ("pronoun-droppingg" language, such as Chinese, Korean and Japanese (Shimazu al., 2020), to non-pro-drop language, such English, French and German (Haspelmath, 2001)). While the pro-drop phenomenon has been widely studied in the research community (Nagard
To tackle this issue, researchers have proposed two primary strategies: (1) incorporating addi- tional pro-drop resolution systems provide sup- plementary syntactic information (Nagard and Kochn, 2010; Taira et al., 2012; Wang et al., 2016). For instance, Xiang et al. (2013) modeled Empty
'Equal Contribution. "Corresponding author.

---

Categories within the framework of government- binding theory; (2) treating pro-drop resolution as a regularization component of NMT task directly (Wang et al., 2018b; Tan et al., 2019). This ap- proach suggests filling dropped pronouns (Wang et al., 2018a) or predicting pro-drops in the Chinese text encoder component of seq2seq model (Wang et al., 2019). Despite the studies done resolving Chinese pro-drop in NMT so far, relevant bench- marks evaluating the effectiveness of pro-drop mit- igation are highly limited, and Chinese-Engling spoken translation datasets with line-grained anno- tation are even fewer.
2 Dataset Creation
To mitigate the scarce of benchmarks evaluating pro-drop in Chinese-English spoken language trans- lation, we collect and construct a new benchmark, PROSE, a high-quality Chinese-English bilingual dataset of four different genres, including Talk, Drama, Movie and Vlog.
2.1 Data Collection and Filtering
The raw data was collected from bilingual subtitles of publicly accessible videos the internet. We assume that these subtitles reflect authentic daily spoken expressions in Chinese and cover a diverse range of zero anaphora phenomena. Specifically, our filtering process is based on three criteria.
In this study, we present PROSE, a PRonoun Omission Solution for Chinese- spoken lan- guage translation. To facilitate research in this area, we introduce novel dataset for document- Chinese- English spoken language translation that includes abundant and diverse pro-drop instances with contextual and pro-drop annotations across four spoken language genres (talk, drama, movie, and vlog). The analysis of this dataset reveals that the negative impact of pro-drop on Chinese- English spoken language translation. Furthermore, we propose the Mention-Aware Semantic Augmen- tation approach, which utilizes mention encoder capture the semantic embedding of dropped pro- nouns and employs a mention-side data augmenta- tion technique to generate additional training pairs. Experiment results on four Eeinese-Enggish trans- lation corpora demonstrate that our proposed ap- proach significantly increase translation quality and the recover rate of missed pronouns, in compari- with baseline methods on both automatic and human evaluation metries. Additionally, we con- ducted ablation studies to provide further insights on the effect of designated losses.
The chosen domain must be spoken, rather than written, such as news articles, to preserve the colloquial features of Chinese;
To ensure high-quality English translations, we only considered source materials in Chi- nese that have undergone manual translation by professionals, rather than relying on ma- chine translations. For instance, we primarily chose movies from China that have been pro- moted overseas and short videos with profes- sional fan-made translations on YouTube.;
To enable accurate restoration of missing pro- nouns, the source material must contain con- textual sentences that provide greater context and accuracy to the translations.
We end up collecting over 20,000 videos in Chi- nese and over 2 million lines of English and Chi- nese subtitle, which can be classified into four dis- tinct spoken genres:
Our contributions are summarized as follows:
We construct document-level Chinese- English spoken translation dataset that covers multiple spoken genres and provides detailed contextual and pro-drop annotation informa- tion.
Subtitles from personal presentations on websites like TED.
Talk:
Drama: Subtitles from Chinese TV series, such as Nirvana in Fire ($).
Our analysis reveals that pro-drop negatively impacts the quality of Seiinese-Englise spoken language translation.
Movie: Subtitles from Chinese films, such as The Great Cause of the Founding of the People E
We propose a Mention-Aware Semantic Aug- mentation approach increase the recover rate of dropped pronouns when translating and hence enhance overall translation quality.
Vlog: Subtitles from short videos filmed by Chinese internet celebrities, such as Huanong Brothers ( ( ).).

---

2.2 Pro-drop Annotation
Accuracy
Talk
Drama
Movie
Viog 85.6%
Subject Ellipsis
93.4%
89.4%
90.4%
We employ DDparser (Zhang et al., 2020), a Chi- nese dependency parsing tool, to detect the omis- sions of subject or object pronouns in the source language Chinese. Subject pronouns are tagged SBV (subject-verb) or VV (verb-verb), while object pronouns tagged VOB (verb-object), POB (preposition-object), DOB (double object), or DBL (double). Dependencies that do not contain these marks are assumed to be missing either the subject or object pronoun. Although this method of labeling is not perfect, it warrants further study. Below is an example from the subtitles of a short video about cooking.
Object Ellipsis
95.3%
90.3%
91.4%
87.3%
Table I: The accuracies of the Subject Ellipsis and Ob- ject Ellipsis marking by the annotation tool.
in Table 2. CWMT2018' is the most popular Chinese-English machine translation corpus, con- taining written language such news articles, while AIChallenger is the largest spoken Chinese- English machine translation dataset to the best of our knowledge.
In comparison with those two widely used bilin- gual datasets, our dataset is 1) more representa- tive with a higher pro-drop ratio, 2) more diverse, containing four genres of spoken language, and 3) more informative, with contextual and pro-drop annotation information.
Chinese: V 1 20 S.
English: Uncle, where should I start cutting? [Subject Ellipsis]
Chinese: A 73. lipsis] [Object Ellipsis]
[Subject EI-
3 Pronoun-Dropping Analysis
English: How do start cutting this?
To gain more insights into the phenomenon of pro- drop in the translation spoken Chinese into En- glish, we examine the prevalence of pro-drop spoken Chinese and its impact on the quality of Chinese-English spoken language translation.
Chinese: AR T
&. (Subject Ellipsis/
English: Will you leave the meat on the bone?
Spoken Chinese Contains More Pro-drop Than Literary Language Formally, pro-drop refers reference position that is filled with amorphologi- cally unrealized form, and one of the most com- mon referential options in many languages such as Chinese (Wang et al., 201 8a), Japanese (Taira et al., 2012), Korean (Park et al., 2015), and Thai (Kawtrakul et al., 2002). Previous studies have re- vealed that spoken Chinese language tends to tain more pro-drops than literary language (Wang et al., 2016. 2017; Xu et al., 2021). However, quan- titative studies on pro-drops in different genres of spoken Chinese, remain scarce.
As shown, each data pair consists of Chinese text with its corresponding pronoun type missing, high- quality English translations done by human ex- perts, and the surrounding context. We apply the DDparser tool on the training set to annotate whether there is Subject Ellipsis and Object El- lipsis in the Chinese sentences, while the English sentences require no annotation. This is due that we only collected source materials in Chinese that have undergone manual translation by profession- als. The high-quality translations have completed the subject and object in English. For the test set, in addition to calculating the BLEU score with human translation, we also use human evaluation to assess Completeness, Semantic Correctness, and Overall quality (details can be found in Appendix C).
As demonstrated Table 2, both written and spo- ken languages contain a certain proportion of pro- drops, which is consistent with the unique grammat- ical phenomenon of Chinese. However, written lan- guage contains fewer Object Ellipsis than spoken language. For example, in the CWMT2018 dataset, the proportion of Object Ellipsis (i.e., 2.80%) is significantly smaller than that of Subject Ellipsis (i.c., 9.00%). Our four bilingual spoken language corpora are varied, displaying differences in the
We randomly sample 100 samples and manually check the accuracy of the Subject Ellipsis and Ob- ject Ellipsis marked by the annotation tool. The experimental results are shown in the table 1.
2.3
Data Statistics
'http://nlp.ng hwt.hlppjju.euu.... wt/..c//ccweeeooooooceeeeeee.eos cwmttwwt// / thub. www.AA/CaalleeggrrAAA Challenger_2018
The data statistics for our datasets, which include four genres spoken Chinese, are presented

---

English #Word / (Sent.
Chinese #Wrrd ASent
Subject Elipsis
Object Ellipsis
Datasets
Spoken
Type
#Doc
#Sen.
#Sent./DDoc
CWMT2018
x
9,023,454
11.54
19.65
9.00%
2.80%
AlChallenger
x
8,426,940
1 11.54
17.99
8.10%
8.90%
Train
1,613
193,965
120.25
17.38
29.35
9.11%
8.01%
Talk
PROSE (Ours)
Test
179
21,325
119.13
17.29
29.79
9.16%
7.92%
Train
22,499
2,150,956
95.60
5.90
8.86
32.77%
33.03%
Drama
Test
25
2,240
89.60
8.31
9.35
40.40%
36.52%
Train
134
T08, 162
807.18
5,66
9.53
37.38%
35.84%
Movie
Test
13,137
938.36
5.50
10.88
35.78%
35.21%
Train
667
75,051
112.52
6.67
8.95
45.91%
37.45%
Vlog
Test
74
7,536
101.84
6.85
9.21
45.38%
33.80%
Table 2: The data distribution of our Chinese-English pro-drop machine translation datasets. Doc. and Sen. indicate Document and Sentence respectively. stands for the quantity, and denotes the ratio.
rates of subject and object pronoun drop, average sentence length, average document length, and so on. For example, the average length of sentences in the three genres of spoken corpora, namely Drama, Movie and Vlog, much shorter than that of Talk (i.e., individual talks) and AlChallenger. particu- lar, the Drama, Movie and Vlog corpora in our data set contain a surprising proportion of pro-drops (about 33% to 46%), which is more extensive than the current largest Chinese-English spoken transla- tion corpus ATChallenger.
English translation errors in our online simulta- neous machine translation system. The primary use case of translation system is Chinese-to- English translation (primarily spoken Chinese) in meetings and conferences. Moreover, we have ex- perienced labeling experts to categorize the bad cases generated by the online system. can be seen from Figure 2 that the proportion of errors caused by pro-drop is relatively high, constituting more than 1 of all errors. This is one of the major factors contributing the degradation of the translation quality of our system.
Pro-drop Harms the Quality of English Spoken Language Translation Subjective and objective pronouns are frequently omitted in spo- ken Chinese, but should be recovered in non-pro- drop languages like English. The question arises whether the current NMT system is able to ac- curately translate spoken Chinese sentences with dropped pronouns into English, non-pro-drop lan- guage, as illustrated in Figure 1.
Zero Pronoun
Trans- former
Huuman Completion
Dataset
A
CWMT
9.0%
27.40
27.91
+0.51
Talk
9.2%
15.56
16.31
+0.75
Drama
40.4%
13.38
15.68
+2.30
Movie
35.8%
14.92
16.62
+1.70
Vlog 45.3%
9.38
12.25
+2.87
Table 3: Results of Eingess-Ennglsh spoken language tramslation with the omitted pronouns complemented by humans. Although the model achieved a high BLEU score of 27.40 the CWMT dataset, its performance on their dataset showed significant decline. with BLEU score dropping from 9.38 to 15.56.
Terminology
30.00 89%
Inacurate
23.81%
To investigate the potential of reinstated pro- nouns in Chinese spoken sentences to improve the quality of Chinese-English spoken language translation, we conduct experiments using spoken Chinese sentences with omitted pronouns comple- mented by humans. We first train a Transformer- base (Vaswani et al., 2017; Hassan et al., 2018 18) model the CWMT dataset, and then report the BLEU (Papineni al., 2002) scores with Sacre- BLEU3 (Post, 2018) on sets of our four cor-
70%
hthers English Polysomy
15
Trnstttionn
Ellipsis
Figure 2: Error distribution of Chinese-Engee English Spoken translation in our online simultaneous translation system. Errors caused by pro-drop (i.e, Ellipsis) account for about % of errors.
Figure 2 shows the distribution of Chinese-to-
44BLEU case.mixed LANGUAAE PAIR + num-

---

pora (i.e., Talk, Drama, Movie and Vlog). Next, the spoken Chinese test sets that detected as pro-drop are completed manually, as shown in the content in brackets in Figure I.
y
"Let's go out and have meal together." "
Text Decoder
The experimental results before and after hu- man completion are shown in Table 3. Although the model achieves 27.40 BLEU score on the CWMT dataset, its performance on our dataset shows significant BLEU score decline (from 9.38 15.56 across four genres). This indicates large discrepancy between spoken and written Chinese for neural machine translation systems that rely on data-driven approaches. For convenience, the sec- ond column in Table 3 displays the proportion of different datasets with pro-drop. Human comple- tion of dropped pronouns leads to varying perfor- mance improvement levels, with the improvement being roughly proportional to the ratio of pro-drops. Interestingly, even on the CWMT dataset, human completion has improved translation quality (i.e., +0.51 BLEU score) ), suggesting that pro-drop may also degrade the quality of the Chinese Eninise-English translation of that dataset.
+ m!!
"B [it[ass)
m
a m
Mention Encoder
Text Encoder
E
(The meal should be ready.)
X
Figure 3: The framework of mention-aware semantic augmentation, æ and y represent sentences in the source and target languages, respectively. The contextual text is denoted by c.
to maximize the log-likelihood:
4 Methodology
Lmmi(() = (((.,)) (log P(yla, (lss))ictt(((oPFP/llll....))))): (1)
4.1
Problem Definition
Given two data spaces, X and y. encompassing all possible sentences in source (Chinese) and tar-
4.2 Mention-Aware Semantic Augmentation
get (English) languages, each sample is pair of sentences belonging different languages, i... (x, 3) E (X, y). Here, {x1, 22, -|=} is the Chinese sentence containing |0 tokens, and y (31 .......,........................ , is the English sentence with lyl tokens. To identify the mentions (co- references) of entities (i.c., pronouns) in x, its sur- rounding context is expressed as e. For example, in the context of e = "Ia E in #tt 4 J (The meal should be ready)", the missing object pronoun of "cE (eat)" in the sentence æ " * E. - "Er""" can be inferred to be "t (meal)", thus the translation of the non-pro-drop sentence would be "I "Let's go out and have a meal together".
Motivated by the high prevalence of pro-drop in spoken Chinese and the consequent difficulty in automatically understanding pro-drop source tences when translated into non-pro-drop English, we present Mention-Aware Semantic Augmenta- tion (illustrated in Figure 3) as a potential solution.
Architecture This approach is built on top of Transformer (Vaswani et al., 2017) and consists of three modules: text encoder E,, text decoder D,- and an additional mention encoder Em.. The men- tion encoder Em is a 6-layer transformer encoder which translates the context e to representations Em (c) E R2 where k is the embedding dimen- sion. To obtain a real-valued output, a projection matrix A C R is applied to En(c), resulting in m - Em (c) The mention representation m and the text representation r are concatenated to- gether each time-step and sent to the decoder to calculate the cross attention. It worth noting that our mention encoder module shares parameters with the text encoder Fz. Moreover, it is agnostic to the model architecture and can casily be adapted to other text generation frameworks.
The neural machine translation task (Bahdanau et al., 2015; Gchring et al., 2017: Vaswani et al., 2017) seeks to model the translation probability P x c; () using conditional language model based on Transformer architecture (Vaswani et al., 2017), where θ represents the parameters of the model to be optimized. Formally, the training ob- jective of a given of observed sentence pairs is
refs.1 smooth.exp + test.SET tok.int + version.1.22

---

Overall, our approach leverages 1) the mention encoder to focus on completing the dropped pro- nouns in the input a from the context e in the case limited parallel corpus, and 2) representation in- terpolation the semantic space of observed sam- ples to expand the training data pairs, thus compen- sating for the lack of large-scale Chinese-English spoken language translation corpora.
Lmir -xIIsip B22 E Bx- (3)
t(X21 + (1 X)a2, + (1 {)uz)],
where is the cross entropy loss (de Boer et al., 2005). According to Appendix A, we can simplify Equation 3 as follows:
Mention-Aware Contrastive Learning We pro- pose a contrastive objective to learn the semantic embeddings m mentions in the source sentence a. Specifically, the representations of sentences containing mentions of entities should be "closer" to m than those without mentions.
Emis ()) + E2i Bez Ex 1a) (4) (()0) + +1 1).2, V1).
which enables us to avoid the requirement for label blending when combining labels y1 and y2, with drawn from 3(xx + a:). This is beneficial in scenarios where is discrete sequence. Ac- cordingly, our mention-side mixup loss minimizes the interpolations loss from vicinity distribution (Chapelle et al., 2000) defined in the representation space:
To this end, expect the similarity between and "similar" sample m= be far greater than that between m and negative sample m i.c. Sim (, m²) >>> Sim(m, m m) To obtain m we use DDparser (Zhang et al., 2020) to detect all mentioned entities in the context, and then ran- domly replace them with a special token [MASK]. m is sampled by randomly replacing non-entity words. The measure of similarity between two em- beddings, denoted Sim, is caleulated using the dot product. This can be interpreted as the angle between the two embeddings in the vector space. Consequently, the mention-awee aware contrastive ob- jective is formulated as follows:
Tmmmns (+) - Em ( (r (X. (1
(5)
(log P [x Am, + (1 A)m! ); ()).
In other words, we can utilize the presence or ab- sence of pronoun context (i.e, m and m+) to argu- ment the training samples for enhancing the robust- ness towards pronouns.
4.3 Training and Inference
(O) (2(,,))>II exp(Simif (Sim(m, mn))
(2)
Finally, we optimize the sum of the above losses:
log exp(Sim(m, m )) + exp(Sim (im(m, m
final (+)
-mmm (e) + Lm (6)
We introduce a regularization loss to further re- duce the disagreements among the mention projec- tion matrix and reduce the redundancy of param- eters: Creg (9) = || A T|12, where I is the identity matrix.
+Lrggg (i) + mmmi (+).
During inference, beam search decoding is per- formed.
Experiments
Mention-Side Mixup Interpolation Drawing in- spiration from Mixup approaches (Zhang et al., 2018; Wang et al., 2021; Wei et al., 2022), we pro- pose to sample data points from the adjacency men- tion semantic region to augment the current train- ing instance. Given pairs of samples (I1, V1) and (I2, )2), Mixup chooses random mixing propor- tion À from a Beta distribution B(a, a) controlled by the hyper-parameter a, and creates an artificial training example (Yx + (1 12)x22, Ay1 +1 2)y2) to train the network by minimizing the loss on mixed-up data points:
5.1 Baselines Comparisons
We compare our method with several state-of- the-art machine translation methods, including pro-drop machine translation methods (RecNMT (Wang et al., 2018a) and pro-dropP&T (Wang et al., 2019)), document-level machine translation methods (HanNMT (Miculicich et al., 2018)), and data-augmentattion machine translation methods (AdvAug (Cheng et al., 2020) and CsaNMT (Wei et al., 2022)). We pre-train the NMT model using the AIChallenger dataset, achieving 27.97 BLEU

---

Type
Dataset
Method
Average
A
pro-drop?
DL?
DAY
Talk
Drama
Movie
Vlog
Base
X
x
13.10
11.99
16.09
5.09
11.57
Fine-tuning
X
x
16.41
17.29
17.73
13.89
16.33
+4.76
RecNM'T (Wang et al., 2018a)
V
x
X
17.46
17.97
18.06
13.98
16.89
+0.53
pro-dropP&/ (Wang et al. 2019)
V
X
17.97
17.98
18.12
14.27
17.09
+0.76
HanNMT (Miculicich et al, 2018)
X
V
18.92
18.92
19.05
17.62
18.63
+2.30
AdvAug (Cheng et al., 2020)
X
X
18.82
18. 12
18.89
14.86
17.67
+1.34
CsaNMT (Wei et al., 2022)
X
X
18.58
18.32
19.32
17.08
18.33
+2.00
Ours
V
19.46
19.87
20.34
18.47
19.54
+3.21
Table 4: Automatic evaluation results Chinese-Englise spoken language translation dataset. The acronym "pro-drop?" stands for "pro-drop?", "DL?" for "DocumentLevel?? and "DA?" 2" for "Data-Augmeenattion..
points the set. Afterward, we optimize the parameters on our specific spoken Chinese corpus, which is relatively small in size. The implementa- tion details are shown in Appendix B.
Genre
Method
Pron.
Seman.
Overall
RecNMT
-0.32
-0.08
-0.45
ManNMT
-0.12
-0.06
-0.23
Talk
CsaNMT
0.19
-0.18
0.25
Ours
0.25
0.32
0.43
For analysis, we also show the performance of NMT models trained on different corpora, includ- ing: (1) Base:
RecNMT
-0.79
-0.42
-0.11
НапNMT
0.68
-0.30
-0,64
Drama
Training the NMT model solely small Chinese-English spoken language translation corpus. (2) Fine-tuning: Training the NMT model on the AIChallenger dataset and then fine-tuning the model on Chinese-Englin spoken corpora.
CsaNMT
-0.47
0.28
0.31
Ours
0.58
0.44
0.44
RecNMT
-0.23
-0.12
-0.01
HanNMT
0.04
-0.72
-0,58
Movie
CsaNMT
-0.66
0.52
0.24
Ours
0.85
0.32
0.35
RecNMT
-0.55
-0.33
-0,28
5.2 Automatic Evaluation
HanNMT 0.53
-0.22
-0.94
Vlog
CsaNMT -0.53
0.30
0.44
For automatic translation evaluation, we report the classical BLEU (Papineni et al., 2002) scores with SacreBLEU. The automatic evaluation results on our four-genre English spoken translation dataset are presented in Table 4.
Ours
0.55
0.25
0.78
Table 5: Human evaluation results in terms of the Best- Worst scaling. The kappa coefficient of judges 0.52.
Our experiment results show that Fine-tuning method outperforms the Base method by 4.76 BLEU points, indicating that the amount of data remains the bottleneck of translation per- formance on the task of spoken language trans- lation with limited corpus. Furthermore, the document-level machine translation method (Han- NMT) is significantly better than single-text-input. based methods (ReeNMT and pro-dropP&T) and nata-auummenattion based methods (AdvAug and CsaNMT), indicating that context information useful for pro-drop translation. Interestingly, the dan-aaggmentation--aaedd NMT methods (AdvAug and CsaNMT) also have an approximate BLEU gain of 1.34 to 2.00, demonstrating that the sam- pling method in the semantic space to expand the training dataset can well enhance the generaliza- tion of pro-drop spoken language translation. In any case, our method greatly outperforms these baseline methods, demonstrating the effectiveness our proposed approach for pro-drop translation.
5.3 Human Evaluation
We also conduct human evaluation focusing on three metrics: pronoun recovery (determining whether the translated sentence complete or con- tains missing mentions), semantic correctness (de- termining whether the translated sentence is seman- tically consistent with the source text sentence) and overall quality.
We sample 200 instances from four corpora, and hired two workers to rate the translation re- sults of pro-dropP&T, HanNMT, CsaNMT and our model based on the above three aspects. We used Pest-Worst Scaling, which has been shown to pro- duce more reliable results than ranking scales (Kir- itchenko and Mohammad, 2017). Specifically, each score computed as the percentage of times it selected as best minus the percentage of times it was selected as worst, and ranges from -1 (unani- mously worst) to +1 (umanimously best). The order in which the translated texts were presented to the

---

Drmm
fireg
Amms
Talk
Drama
Movie
Vlog
Average
19.46
19.87
20.34
18.47
19.54
2
v
/
18.97
19.28
19.12
17.92
18.82
-0.72
3
v
18.47
18.57
18.35
17.52
18.23
-1.31
4
/
19.37
19.43
19.97
18.06
19.21
-0.33
Table Ablation study of different losses.
judges was random. The details of the questions can be found in Appendix C.
on some syntax tools to provide extra information for the MT system (Nagard and Koehn, 2010; Taira et al., 2012; Wang et al., 2016), such as modeling empty categories (Xiang al., 2013). However, directly using the results of external pro-drop res- olution systems for the translation task shows lim- ited improvements (Taira et al., 2012), since such external systems are trained on small-scale data that non-homologous to MT. To bridge the gap between the two tasks, some later studies (Wang et al., 2018b; Tan et al., 2019; Xu et al., 2022) di- rectly integrated the pro-drop resolution task into the machine translation task, such reconstructing the missing pronouns (Wang al., 2018 (0)8a) in the encoder or predicting the pro-drop (Wang et al., 2019).
Table 5 indicates that HanNMT, a strong doeument-level machine translation method, per- forms better than CsaNMT and RecNMT in recov- ering missing pronouns, possibly due to its use of rich source-side context. Interestingly, CsaNMT, which utilizes data augmentation, exhibits superior semantic correctness and overall quality. Nonethe- less, our method outperforms all baselines in terms of pronoun recovery and overall quality, indicat- ing that the performance improvement is attributed pro-drop resolution. More examples of gener- ated translations of model against comparison systems are presented in Appendix D.
5.4 Ablation Study
Unlike previous methods, our method recovers the missing pro-drop information from the context and uses data augmentation in the semantic space to increase the training data. To the best of our knowledge, are the first to construct document- level Shinese-Engglise spoken translation dataset covering multiple spoken genres.
We conduct various ablation studies on our dataset as shown in Table 6, which assess the contribution of different losses. The SacreBLEU scores are reported on test sets.
The experiment results show that the removal of Mention-Side Mixup Interpolation results in 0.72 BLEU point drop, indicating that the data aug- mentation method based on mentions can increase the generalization of pro-drop translation. More- over, we find that all our losses, especially L are beneficial for improving the translation quality. This implies that our mention-aware contrastive learning is capable of capturing the lost pronoun in- formation and thus improving overall performance of NMT.
Document-Leve Machine Translation Recent works on customized model architcctures have fo- cused improving context representations for document-level translation models, such as context- aware encoders (Voita et al., 2019a), context-ane aware decoders (Voita et al., 2019b), hicrarchical history representations (Miculicich et al., 2018), and mem- ory networks (Maruf and Haffari, 2018). How- ever, Yin et al. 2021 points out that simply feeding contextual text may not be able accurately dis- ambiguate pronouns and polysemous words that require contexts for resolution. Thus, we employ contrastive learning to enforee the model incor- porate the mention-informan about the dropped pronouns.
is worth noting that the third row in Table 6 is a strong document-level machine translation base- line, indicating that the improvement of our model mainly comes from the mention-aware aware loss rather than the wide contexts in the source side.
6 Related Work
Pro-drop in Machine Translation Research on pro-drop machine translation mainly falls into two categories: (1) methods using extra pro-drop res- olution systems and (2) joint pro-drop resolution and translation training methods. The former relies
Data Augmentation in Machine Translation Our approach is also related Vicinal Risk Mini- mization (Chapelle et al., 2000), which formalizes data augmentation as extracting additional pscudo samples from the vicinal distribution of observed

---

instances (Krizhevsky et al., 2012: Zhang et al., 2018: Wang et al., 2021). In machine transla- tion, this vicinity is often defined through adver- sarial augmentation with manifold neighborhoods (Cheng et al., 2020; Wei et al., 2022). Our approach is similar in that it involves an adjaceney mention semantic region as the vicinity manifold for each training instance.
insights and expertise have been instrumental in the realization of our ideas. We are also immensely grateful to the anonymous reviewers for their con- structive feedback and comments. Their perspec- tives have greatly enhanced the quality of our work. Lastly, we extend our gratitude to all those who have indirectly contributed to this project. Your support has not gone unnoticed and is much appre- ciated.
7 Conclusion
This study provides valuable insights into the phe- nomenon of pro-drop in Chinese and its impact on Chinese-English spoken language translation. Furthermore, we introduced a new dataset that im- proves upon existing corpora in terms of representa- tiveness, diversity, and informational value. Lastly, our proposed approach, Mention-Aware Semantic Augmentation, demonstrates superior performance over existing methods in addressing the challenges posed by pro-drops.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. 2000. Vicinal risk minimization. In Advances in Neural Information Processing Sys- tems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 416- 422. MIT Press.
Our study underscores the critical importance of taking into account pro- pro-drops in NMT systems, and offers valuable benchmarks and insights to guide future advancements in this field.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. Advaug: Robust adversarial aug- mentation for neural machine translation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3961-5970. Association for Computational Linguisties.
Limitations
Our method has shown effectiveness improving translation quality in pro-drop machine translation task from pro-drop languages such as Chinese to a non-pro-drop target language, English in this case. However, due to the limited availability of data resources, the translation performance from other pro-drop languages such as Japanese (Sasano and Kurohashi, 2011), Thai (Kawtrakul al., 2002), Korean (Park et al., 2015), Italian (Tida and Poe- sio, 2011), Spanish (Palomar et al., 2001), etc. to non-pro-drop languages remains to be evaluated. Furthermore, our method may not be able to match the performance of large language models such as PaLM (Chowdhery et al., 2022), ChatGPT and GPT4 which are trained with massive machine translation corpora and other language resources.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov. and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.022. 12211.
Acknowledgement
We would like to express our deepest appreciation to Suqing Yan, Weixuan Wang, and Xupeng Meng from Huawei for their invaluable assistance and dis- cussions during the implementation process. Their
Pieter- de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. 2005. tutorial on the cross- entropy method. Ann. Oper. Rex., 134(1):19-67
compaaa...com////aa/aaa/e 4

---

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learn- ing Research, pages I 223--122. PMI.R
machine translation with hierarchical attention net- works. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2947-2954. Association for Computational Linguisties.
Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution, In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MerR, WMT@ACL 2010, Uppsala, Sweden, July 15-16, 2010, pages 252- 261. Association for Computational Linguistics.
Martin Haspelmath. 2001. 107. The European linguistic area: Standard Average European, pages 1492-1510. De Gruyter Mouton, Berlin, Boston.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang. Marcin Junczys-Downan Lewis, Mu Li, Shujie Liu, Tie- Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dong- dong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic chinese to en- glish news translation. CoRR, abs/1 803.05567.
William
Mylc Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT' 2019, Minneapo- lis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53. Association for Computational Linguistics.
Ryu lida and Massimo Poesio. 2011. A cross- Progs-lingual ILP solution zero anaphora resolution. In The 49th Annual Meeting of the Association for Compu- lational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, 1, Portland, Oregon, USA, pages 603.813.77 The Associ- ation for Computer Linguistics.
Manuel Palomar, Antonio Ferrández, Lidia Moreno, Patricio Martínez- Barco, Jesús Peral, Maximiliano Saiz-Noeda, and Rafael Muñoz. 2001. An algorithm for anaphora resolution in spanish texts. Comput. Linguistics, 2744)5545
Asanee Kawtrakul, Mukda Suktarachan, Patcharee Vara- sai, and Hutchatai Chanlekha. 2002. A state of the art of thai language resources and thai language behav- ior analysis and modeling. In The 3rd Workshop on Asian Language Resources and International Stan- dardization, @ COLING 2002, Taipei, Taiwan, August 24- Setember 1, 2002.
Kishore papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Svetlana Kiritchenko and Saif M. Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30- August 4, Volume 2: Short Papers, pages 465 470. Association for Com- putational Linguistics.
Arum Park, Seunghee Lim, and Munpyo Hong. 2015. Zero object resolution in korean. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, PACLIC 29, Shanghai, China, October 30 November 1, 2015. ACL.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 November 1, 2018, pages 186-191, Association for Computational Lin- guistics.
Alex Krizhevsky, Tlya Sutskever, and Geoffrey E, Hin- ton. 2012. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Sys- tems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114 1114.
Ryohei Sasano and Sadao Kurohashi. 2011. I. A discrimi- native approach to japanese zero anaphora resolution with large-scale lexicalized case frames. In Fifth In- ternational Joins Conference on Natural Language Processing, IJCNLP 2011, Chiang Mai, Thailand, November 8-13, 2011, pages 758 766. The Associa- tion for Computer Linguistics.
Sameen Maruf and Gholamreza Haffari. 2018 Docu- ment context neural machine translation with mem- ory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1275-1284. As- sociation for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL 2016. The Association for Computer Linguistics.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson, 2018. Document-level neural
Sho Shimazu, Sho Takase, Toshiaki Nakazawa, and Naoaki Okazaki. 2020. Evaluation dataset for zero

---

pronoun in Japanese to English translation. In Pro- ceedings of the Twelfth Language Resources and Evaluation Conference, pages 3630-3634, Marseille, France. European Language Resources Association.
Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4937-4945. AAAI Press.
Longyue Wang. Zhaopeng Tu, Xing Wang. and Shum- ing Shi 2019, One model to learn both: Zero pro- noun prediction and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, SMNLL-IJCNLP' 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 921-930. Association for Com- putational Linguistics.
Hirotoshi Taira, Katsuhito Sudoh, and Masaaki Nagata. 2012. Zero pronoun resolution can improve the qual- ity of J-E translation. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Sta- tistical Translation, SSST@ ACT 2012, Jeju, Korea, 12 July 2012, pages 111-118, Association for Com- putational Linguistics.
Xin Tan, Shaohui Kuang, and Deyi Xiong. 2019. De- tecting and translating dropped pronouns in neural machine translation. In Natural Language Process- ing and Chinese Computing- 8th CCF International Conference, NLPCC 2019, Dunhwang. China, Octo- ber 9-14, 2019, Proceedings, Purt I, volume 11838 of Lecture Notes in Computer Science, pages 343-354. Springer.
Longyue Wang. Zhaopeng Tu, Andy Way, and Qun Liu. 2018b. Learning to jointly translate and predict dropped pronouns with a shared reconstruction mech- anism. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2997-3002. Association for Computational Linguistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Mlaaa Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9. 2017, Long Beach, CA, USA. pages 5998 -608..
Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016. A novel approach to dropped pronoun translation. In NAACL HLT 2016, The 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, San Diego Califor- nia, USA, June 12-17, 2016, pages 983-993. The Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural ma- chine tramslation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing,
Longyue Wang. Zhaopeng Tu, Xiaojun Zhang. Siyou Liu, Hang Li, Andy Way, and Qun Liu. 2017. A novel and robust approach for pro-drop language translation. Mach. Transl., 31 (11-2))65-87.
2019. Hong Kong, China, November 3-7, 2019, pages 877-886. Association for Computational Linguistics.
Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. 2022. Learning to gen- eralize to more: Continuous semantic augmentation for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7930- 7944. Association for Computational I Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019b.
When good translation wrong in context: Context- aware machine translation improves on deixis, ellip- sis, and lexical cohesion. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1198-1212 Association for Computational Linguisties.
Bing Xiang, Xiaoqiang Luo, and Bowen 7hou. 2013. Enlisting the ghost: Modeling empty categories for machine translation. In Proceedings of the 51 An- nual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bul- garia, Volume 1: Long Papers, pages 822 831. The Association for Computer Linguistics.
Ke Wang. Guandan Chen, Zhonggiang Huang. Xiaojun Wan, and Fei Huang. 2021. Bridging the domain gap: Improve informal language translation via coun- terfactual domain adaptation. In 7 Thirtyy Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13970- 13978. AAAI Press.
Mingzhou Xu, Liangyou Li, Derek F. Wong, Qun Liu, and Lidia S. Chao. 2021. Document graph for neu- ral machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8435-8448 Association for Computa- tional Linguistics.
Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018a. Trans- lating pro-drop languages with reconstruction models. In Proceedings of the Thirty-Secome AAAI Confer- ence on Artificial Intelligence, (AAAI-18), the 30th in- novative Applications of Artificial Intelligence (IAAI- 18), and the Sth AAAI Symposium on Educational
Mingzhou Xu, Longyue Wang, Derek F. Wong. Hongye Liu, Linfeng Song, Lidia S. Chao, Shuming Shi, and

---

Zhaopeng Tu. 2022. Guofeng: benchmark for zero pronoun recovery and translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022, pages 11266-1 12788. Association for Computational Linguisties.
Appendix
A Proof of Equation
[())- E B22. 18 Sla (((Axs
(1 2)a2, Ayn (1 A1212]]
(7)
Em up Eng Baa Es-A(a)))
Ae(Ax1 + (1- A)22. V1)
(1 - -)e(Arr + (1 22))
(8)
EBn ~pp Erp E2~((a;a)
Ber (2) (Ax1 + (1 2) 91)
+ (1 - 2)(()21 + +11 A) V2)]
(9)
= -Em,,ippp Ea BerBBr((..5
Bx- 12 11 2)(z((Ar1+
Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André F. T. Martins, and Graham Neubig. 2021. Do context-aware translation models pay the right attention? In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guisties and the 11th International Joint Conference on Natural Longuage Processing, ACL/JCNNP 2021. (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pages 788-801. Association for Computational Linguistics.
(1 A)12,01) U1) + (1 2)((Ax1
(1 2)22,,82))
(10)
0.5 * Er Eug
E Exmplatt,a)
[e(ATT + (1 - A)z2 y + 0.5 + Eayserp
Eey E Ex [f(Ar1+
(1 2)
(11)
-0.5 Ex App E Eur E Ex-
[e(ATT + (1 A) v1)] + 0.5 + Ergyrrr
E 8pp ((1-)) [E((1 A)2z
+ Ax1, w1)
(12)
Exi t2 Earppy 1,)
(1 2/1)
(13)
{(r,P/1 Eq (8): Linearity of the loss: f(r, pV1 + (1 p)y2) = pl(x, 31) + (1 p){(z, 92). where the loss is the cross entropy loss.
Hongyi Zhangy Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018 mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net.
Eq (9): Expectation of Bernoulli(A).
Eq (10): The Beta distribution is conjugate prior for the Bernoulli.
Eq (11): Expectation of a Bernouli(0.5))
Eq (12): Symmetry of the Beta distribution in the sense that À ~ (a, b) implies (b, a).
Eq (13): Changing variable names in the ex- pectation.
B Implementation Details
We implement our method on top of the Transformer-base (Vaswani et al., 2017) imple- mented in Fairseq (Ott et al., 2019). For this, the
Shuai Zhang, Lijie Wang, Ke Sun, and Xinyan Xiao. 2020. practical chinese dependency parser based on a large-scale dataset.

---

dimension k to 512, the number of attention heads to 8, the mention encoder Em and the text encoder Et and text decoder D2 to layers, and the maximum sequence length to 200. The beam size the beam search was 5. Other included a dropout rate of 0. Adam with learn- ing rate of le-5, 31 0.9, and [32 0.999. To address the out-of- vocauulary problem, we apply byte-pair-encoding (BPE) vocabulary (Sennrich al., 2016) with 40k merge operations and in {(a + a.) 0.1. We implemented our model using PyTorch and used Tesla V100 graphic cards for training.
Human Evaluation Questions
Completeness: Does the translated sentence demonstrate syntactic completeness?
Semantic Correctness: Is the translated sen- tence semantically correct?
Overall: What is the overall quality of the translation?
Examples of Generated Translations
Examples of generated translations of our model and comparison systems are show in Table 7, Ta- ble 8, Table 9, and Table 10.

---

in 3
(Gold: Your other option is a bit safe. Your just going to get some money for sure. ) イ R 500
Input
(Gold: I'm just going to give you 500 bucks.)
4 L i t i i & & & & 1
(Gold: You can stick
your wallet and use it immediately.)
4/: 6) E. x # T
?
(Gold: So see what your intuition here.)
Another safer option: let you take another exact amount. But only $500.
HanNMT
You can put the money in your wallet or spend it right away.
Have your instincts decided?
Another safer option: let you take the exact amount again. But only take $500. You can put the money in your wallet or spend it right away. Intuition decided?
CsaNMT
Another more secure option would be to get an exact amount of money. But it can only be up to $500.
Ours
You can either put it in a wallet or spend it right away.
Have you made your decision based your intuition?
Table 7: Examples of generated translations on Talk.
# 7
(Gold: I'm so sorry. Do you want some of my breakfast? ) J HL & F (Gold: No, I'm too depressed to eat.)
Input
it 5. 3) $ म a
(Gold: I'll probably eat in, like, five minutes.)
I'm sorry. Do you want my breakfast?
HanNMT
No. I'm too depressed to cat. In five minutes.
I'm sorry. Would you like some breakfast? No. I'm too depressed to cat.
CsaNMT
Talk about it in five minutes.
Sorry, do you want to have my breakfast?
Ours
No. I'm too upset to eat.
Let's talk about it in five minutes.
Table 8: Examples of generated translations on Drama.

---

(Gold: is a girl)
Input
1 l tet -
(Gold: She has all the fingers and toes.)
(Gold: Doctor, how come she doesn't 't cry?)
is girl Sound limbs
HanNMT
Doctor, why isn't she crying?
Girl.
CsaNMT
Sound limbs
Doctor, why isn't she crying?
She is girl.
Ours
She has all her limbs.
Doctor, why isn't she crying?
Table 9: Examples of generated translations on Movie.
- F
*$ 69
(Gold: I just saw a snakehead fish, very big onc.)
Input
A v + T
(Gold: Where it now?)
A XX 3 L (1 J T
(Gold: thought they had run away, but they haven't it look!)
Bl = 66 2 # En
(Gold: They's re really fat now.)
I'm just looking at the snakchead. It's big.
Where'd 'd you go? thought they got away. They didn't. Look. They' re so fat now.
HanNMT
Just so can see the snakehead. It's big. Run where?
CsaNMT
thought they were gone, but they weren't. 't.
I'm so fat now.
T just took a look at the snakehead fish, it's so big. Where did it go? thought it had run away, but it didn't, 't look.
Ours
They are eating well now, aren't 't they?
Table 10: Examples of generated translations on Vlog.


Reviews:
[{"review_id": "z3k8H9eTPW", "reviewer": "Reviewer_DfAP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are glad that you found our dataset carefully annotated and our method novel. Below are our responses to specific comments.\n\n***\n\n>The difference between the proposed dataset and the related work is poorly discussed. For example, [1] is also proposed for the zero pronoun translation in Chinese-English translation.\n\nWe thank the reviewer for pointing out the related paper [1]. We have carefully examined the paper and decided not to include it in comprison simply because the **data and code [1] claims to release are not available**. The stated GitHub open-source address in the paper is invalid, and we are unable to obtain the dataset publicly to the best of our effort.\n\nSupposing the data of [1] is available to public, our dataset differs from the one claimed in [1] by both **size and covered domains**. The size of our dataset (25,205 documents with 2,572,372 sentences) is significantly larger than the one proposed in [1] (457 documents with 8093 sentences). [1] covers *Movie Subtitles*, *Q&A Forum*, *Government News*, *Web Fiction*, and *Personal Profile*, while our dataset covers *Talk and Interview*, *TV Drama*, *Movie*, and *Vlog*. The scope of our dataset is pro-drop in **spoken language** and all of our sources are spoken ones, while 72.8% of sentences in [1] are from **written text** sources. In addition, this study also proposes a new method, Mention-Aware Semantic Augmentation, to mitigate pro-drop errors in MT.\n\nAs demonstrrated in Table 1 and discussed at Lines 189-202, we have compared 2 widely used biliginual datasets, CWMT2018 and AIChallenger. We have further augmented our analysis and comparison in line with your comment.\n\n[1] Wang et al. 2022. GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation.\n\n***\n\n\n>The annotation process (Section 2.2 Pro-drop Annotation) is not written in detail. 1) How to label the English sentence, word alignment information or any else? 2) What's the accuracy of the Chinese label process? 3) How many types of labels are used? e.g., [Subject Ellipsis]. I think the authors should make it clear in the following version.\n\n1) As we stated in Section 2.2, we applied the DDparser tool on the training set to annotate whether there is *Subject Ellipsis* and *Object Ellipsis* in the Chinese sentences, while the English sentences require no annotation. This is due to that we only collected source materials in Chinese that have undergone manual translation by professionals. The high-quality translations have completed the subject and object in English, as described at [Lines 131-138]. For the test set, in addition to calculating the BLEU score with human translation, we also use manual evaluation to assess Completeness, Semantic Correctness, and Overall quality (details can be found in Appendix C).\n\n2) We have randomly sampled 100 samples and manually checked the accuracy of the Subject Ellipsis and Object Ellipsis marked by the annotation tool. The experimental results are shown in the table below:\n\n|      Accuracy     |  Talk | Drama | Movie |  Vlog | Average |\n|:----------------:|:-----:|:-----:|:-----:|:-----:|:-------:|\n| Subject Ellipsis | 93.4% | 89.4% | 90.4% | 85.6% |  89.7%  |\n|  Object Ellipsis | 95.3% | 90.3% | 91.4% | 87.3% |  91.1%  |\n\nThe accuracy of the annotation tool is strong. We have added this statistics in our revised version.\n\n3) Two types of lables (*Subject Ellipsis* and *Object Ellipsis*) were used.\n\n***\n\nThank you for all the great questions and suggestions! Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n\n2. The paper released a dataset for Chinese-English translation to address the problem of pronoun-dropping.\n\n3. This released dataset contains 4 topics: talk, drama, movie, and vlog with detailed pro-drop annotation and analysis.", "reasons_to_reject": "1. The difference between the proposed dataset and the related work is poorly discussed. For example, [1] is also proposed for the zero pronoun translation in Chinese-English translation.\n\n2. The annotation process (Section 2.2 Pro-drop Annotation) is not written in detail.  1) How to label the English sentence, word alignment information or any else?  2) What's the accuracy of the Chinese label process?  3) How many types of labels are used? e.g.,  [Subject Ellipsis].  I think the authors should make it clear in the following version. \n\n\n[1] GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation, EMNLP 2022", "questions_to_authors": "", "paper_topic_and_main_contributions": "The present paper focuses on the issue of \"pronoun-dropping\" in Chinese-English translation. It introduces a dataset with careful pro-drop annotation. This dataset comprises four domains: talk, drama, movie, and vlog. Additionally, the paper proposed a novel method to address the pro-drop problem, which is effectively demonstrated using the aforementioned dataset. The results of experiments conducted on four Chinese-English translation corpora reveal that the proposed method outperforms existing approaches regarding translation quality and the appropriate reinsertion of omitted pronouns.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation, EMNLP 2022"}, "extra_text": {}}, {"review_id": "0sRAOeCF3O", "reviewer": "Reviewer_x45u", "summary": "", "questions": "Could you also provide the performance of LLMs, such as GPT-3.5/4? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments! We are glad that you found our dataset benefiting future research and our proposed method inspiring. Below are our responses to specific comments.\n\n***\n> As the authors pointed out in the limitation section, the proposed method maybe is still inferior to LLM models, which is a key challenge in MT.\n\n> Could you also provide the performance of LLMs, such as GPT-3.5/4?\n\nYes, we have supplemented our experiment by adding the performances of LLMs on our dataset. Below is a table of the BLEU score results (cf. Table 4).\n\n| Method |  Talk | Drama | Movie |  Vlog | Average |\n|:------------------:|:-----:|:-----:|:-----:|:-----:|:-------:|\n| gpt-3.5-turbo-0301 | 23.12 | 21.23 | 22.44 | 20.82 |  21.90  |\n|     gpt-4-0314     | 23.72 | 22.27 | 22.27 | 20.82 |  22.27  |\n|        Ours        | 19.46 | 19.87 | 20.34 | 18.47 |  19.54  |\n\nThe results show that although the BLEU scores of GPT-3.5/4 are better than our method, the gap is not so significant. There is still a lot of room for further improvement and we will explore incorporating our method with LLM in future work. We have added the LLMs results to Table 4 in our revised version.\n\n***\n\n> The dataset is limited to Chinese-English language pair, and there are many other language pairs with pro-drop languages.\n\nWe agree with you that there are more language pairs with pro-drop languages. Besides Chinese, typical pro-drop languages include Spanish, Italian and Japanese, while English, German and French are non-pro-drop languages [1, 2]. Although a dependency parser is a prerequisite of our method, we believe that the basic idea of the method is transferable and can be generalized on other language pairs. \n\n[1] Mihoko Zushi. 2003. Null arguments: the case of Japanese and Romance.  \n[2] Gereon Müller. 2005. Pro-Drop and Impoverishment.\n\n***\n\n> The kappa value of human evaluation in Table 4 is only 0.52 which is not that convincing.\n\nWe agree with you that the kappa value of 0.52 is less than good. While we are short in time conducting a new human evaluation during the rebuttal period, we will re-evaluate the results with a larger evaluator size and report additional inter-rater agreement metrics in our revised version.\n\n***\n\n> Typos and Writing suggestions:\n\nThank you for catching those typos and unclear expressions! We have carefully corrected them.\n\n***\nThank you for all the great insight and suggestions! Please let us know if you have any remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* a new dataset for document-level Chinese-English spoken language translation, focusing on pro-drop instances, which benefits the future research on this area. \n* the Mention-aware semantic augmentation method to deal with pro-drop phenomena is trained jointly with NMT framework, the contrastive learning objective and mention-side mix-up interpolation objective can inspire other research areas as well.  \n* have done comprehensive experiments \n* well written and organized ", "reasons_to_reject": "* as the authors pointed out in the limitation section, the proposed method maybe is still inferior to LLM models, which is a key challenge in MT. \n* the dataset is limited to Chinese-English language pair, and there are many other language pairs with pro-drop languageds. \n* the kappa value of human evaluation in Table 4 is only 0.52 which is not that convincing. ", "questions_to_authors": "Could you also provide the performance of LLMs, such as GPT-3.5/4? ", "paper_topic_and_main_contributions": "This paper introduces PROSE, a new dataset for document-level Chinese-English spoken language translation, focusing on pro-drop instances. To alleviate the negative impact of pro-drop, this paper also proposes Mention-aware semantic augmentation method to deal with the dropped pronouns. Experimental results confirm the effectiveness of the proposed method. \n\n* a new dataset for document-level Chinese-English spoken language translation, focusing on pro-drop instances\n* the Mention-aware semantic augmentation method to deal with pro-drop phenomena\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Table 1: use commas to seperate figures larger than 1,000. (#doc and #sen. columns)\n* caption of Table 2 can be further improved: point out the ratio and the metric in the caption.  \n* since-wide --> source-side "}}, {"review_id": "wiSAQ7tegD", "reviewer": "Reviewer_NDb2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We are glad that you found our dataset useful for the machine learning research community. Below are our responses to specific comments.\n\n***\n> I cannot agree that the proposed method is effective yet. In my understanding, several methods such as Base can use only single sentence as an input but the authors should use concatenated two sentences as an input like the baseline of [1] for a fair comparison.\n\nWe thank the reviewer for pointing out this concern. In fact, we have compared our method with state-of-the-art **document-level machine translation models**, such as the HANNMT method as shown in Table 3, which includes context sentences as input. Additionally, we have demonstrated the effectiveness of our method by including a variant that only uses Mention-Side Mixup Interpolation to concatenate context without Mention-Aware Contrastive Learning in our ablation study results (Table 5, row 3).\n\nWe also argue that the **effect of context is inconclusive**, as introducing context may positively or negatively impact the accuracy of the results. To further investigate this effect, we have supplemented our experiment with a simple baseline method, Fine-tuning (w/ context), where we use concatenated two sentences as input. To be more specific, the method takes the last two sentences as additional context, but the output is still only the translation of the current source language input into the target language.\n\n|           Method          |  Talk | Drama | Movie |  Vlog | Average |\n|:-----------------------:|:-----:|:-----:|:-----:|:-----:|:-------:|\n|       Fine-tuning       | 16.41 | 17.29 | 17.73 | 13.89 |  16.33  |\n| Fine-tuning(w/ context) | 16.75 | 17.08 | 17.97 | 13.93 |  16.43  |\n|           Ours          | 19.46 | 19.87 | 20.34 | 18.47 |  19.54  |\n\nThe experimental results in above table suggest that simply concatenating context sentences does not provide significant benefits (an average BLEU score increase of merely 0.10) on average. Specifically, the with-context fine-tuning yielded a worsen performance (16.75) in Drama domain compared to the no-context fine-tuning (17.29).\n\nWe hope this addresses your concern and provides a fair comparison of our proposed method with existing methods. Thank you for your valuable feedback.\n\n[1] Voita et al. 2018. Context-Aware Neural Machine Translation Learns Anaphora Resolution.\n\n***\n\nThank you for all the great suggestion! Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides a document-level Chinese-English translation dataset to address the ellipsis in translation. The dataset is useful for the machine translation research community.", "reasons_to_reject": "I cannot agree that the proposed method is effective yet. In my understanding, several methods such as Base can use only single sentence as an input but the authors should use concatenated two sentences as an input like the baseline of [Voita et al., 18] for a fair comparison. \n\nVoita et al., 18: Context-Aware Neural Machine Translation Learns Anaphora Resolution.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In some languages such as Chinese and Japanese, we can remove pronoun from an utterance. Translation from a sentence with the ellipsis on pronouns to a language without such ellipsis is a challenging issue in the machine translation field. To address this issue, this paper provides a new dataset for document-level Chinese-English spoken language translation because such ellipsis frequently occurs during speaking. In addition, the authors applied contrastive learning (and mixup) to a mention encoder which encodes a context sentence. Experimental results show that the proposed method achieved better performance than several previous methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the issue of "pronoun-dropping" in Chinese-English translation. 
A dataset with careful pro-drop annotation is created, consisting of four domains: talk, drama, movie, and vlog. 
The paper also proposes a novel method to address the pro-drop problem, and the results show that it outperforms existing approaches. 

A few revisions are necessary, which are already provided in authors' response.
Decision: Accept-Main