SlowGuess/ACL_2020_OCR huggingface dataset:
Junnan Zhu $^{1,2}$ , Yu Zhou $^{1,2,3}$ , Jiajun Zhang $^{1,2}$ , Chengqing Zong $^{1,2}$
$^{1}$  National Laboratory of Pattern Recognition, Institute of Automation, CAS
$^{2}$  School of Artificial Intelligence, University of Chinese Academy of Sciences
<sup>3</sup> Beijing Fanyu Technology Co., Ltd
{junnan.zhu, yzhou, jjzhang, cqzong}@nlpr.ia.ac.cn
Cross-lingual summarization aims at summarizing a document in one language (e.g., Chinese) into another language (e.g., English). In this paper, we propose a novel method inspired by the translation pattern in the process of obtaining a cross-lingual summary. We first attend to some words in the source text, then translate them into the target language, and summarize to get the final summary. Specifically, we first employ the encoder-decoder attention distribution to attend to the source words. Second, we present three strategies to acquire the translation probability, which helps obtain the translation candidates for each source word. Finally, each summary word is generated either from the neural distribution or from the translation candidates of source words. Experimental results on Chinese-to-English and English-to-Chinese summarization tasks have shown that our proposed method can significantly outperform the baselines, achieving comparable performance with the state-of-the-art.
Cross-lingual summarization is to produce a summary in a target language (e.g., English) from a document in a different source language (e.g., Chinese). Cross-lingual summarization can help people efficiently understand the gist of an article written in an unfamiliar foreign language.
Traditional cross-lingual summarization methods are pipeline-based. These methods either adopt summarize-then-translate (Orasan and Chiorean, 2008; Wan et al., 2010) or employ translate-thensummarize (Leuski et al., 2003; Ouyang et al., 2019). The pipeline-based approach is intuitive and straightforward, but it suffers from error propagation. Due to the difficulty of acquiring cross-lingual summarization dataset, some previous researches focus on zero-shot methods (Ayana et al., 2018;
Figure 1: An example of the translation pattern in a sample extracted from Zh2EnSum (Zhu et al., 2019) which is a Chinese-to-English cross-lingual summarization dataset. It shows that some words in the summary are translated from the source words (in the same color). The translation table also gives the corresponding relation to these words. Best viewed in color.
Duan et al., 2019), i.e., using machine translation or monolingual summarization or both to teach the cross-lingual system.
Recently, Zhu et al. (2019) propose to use round-trip translation strategy to obtain large-scale cross-lingual summarization datasets. They incorporate machine translation and monolingual summarization into the training of cross-lingual summarization using multi-task learning to improve the summary quality with a quite promising performance. However, we find that there exist the following problems: (1) The multi-task methods adopt extra large-scale parallel data from other related tasks, such as monolingual summarization or machine translation. These methods are heavily dependent on data, making it difficult to migrate to languages with low resources. (2) The multi-task methods either simultaneously train cross-lingual summarization and monolingual summarization or alternately train cross-lingual summarization and machine translation, resulting in a quite time-consuming
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1309-1321 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
training process.
To alleviate the above problems, we observe some examples extracted from the cross-lingual summarization dataset. We find that there exists a translation pattern in the cross-lingual summaries, as shown in Figure 1. Inspired by the translation pattern, we can first attend to some specific segments in the input sequence, then translate them into the target language, and integrate this bilingual information into the final summary. Therefore, in this paper, we explore an efficient method consistent with the translation pattern.
To achieve that goal, we propose a novel method (Figure 2) that allows either generating words from the vocabulary or selecting words from the translation candidates of the words in the source article. Specifically, we first employ the encoder-decoder attention distribution to help determine which source word should be translated. Then we present three strategies, i.e., Naive, Equal, and Adapt, to obtain the translation probability from a probabilistic bilingual lexicon. The translation distribution can be acquired based on the encoder-decoder attention distribution and the translation probability. Next, we add an extra translation layer to calculate a translating probability. The final distribution is the weighted sum (weighed by the translating probability) of the translation distribution and the neural distribution.
- We introduce a novel and efficient method which integrates the operation of attending, translating, and summarizing.
- We present three effective strategies to acquire the translation probability. It has shown that all these strategies can significantly improve the performance over the baseline.
- Experimental results demonstrate that our method can achieve remarkable improvements over baselines and achieve comparable performance with the state-of-the-art on both English-to-Chinese and Chinese-to-English cross-lingual summarization tasks.
- Our method has two advantages over the state-of-the-art: (1) We only adopt an additional probabilistic bilingual lexicon in
stead of a large-scale parallel machine translation dataset, which significantly relaxes the model's dependence on data. (2) Our model has a much smaller model size and a much faster training speed.
In this paper, we implement our method based on Transformer (Vaswani et al., 2017) encoder-decoder framework, where the encoder first maps the input sequence  $X = (x_{1}, x_{2}, \dots, x_{n})$  into a sequence of continuous representations  $z = (z_{1}, z_{2}, \dots, z_{n})$  and the decoder generates an output sequence  $Y = (y_{1}, y_{2}, \dots, y_{m})$  from the continuous representations. The encoder and decoder are trained jointly to maximize the conditional probability of target sequence given a source sequence:
$$
L _ {\theta} = \sum_ {t = 1} ^ {N} \log \mathrm {P} \left(\mathrm {y} _ {\mathrm {t}} \mid \mathrm {y} _ {<   \mathrm {t}}, \mathrm {X}; \theta\right) \tag {1}
$$
Transformer is composed of stacked encoder and decoder layers. The encoder layer is a self-attention block followed by a position-wise feed-forward block. Compared with the encoder layer, the decoder layer has an extra encoder-decoder attention block. For self-attention and encoder-decoder attention, a multi-head attention block is used to obtain information from different representation subspaces at different positions. Each head corresponds to a scaled dot-product attention, which operates on query  $Q$ , key  $K$ , and value  $V$ :
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V \tag {2}
$$
where  $d_{k}$  is the dimension of the key.
Finally, the output values are concatenated and projected by a feed-forward layer to get final values:
$$
\begin{array}{l} \operatorname {M u l t i H e a d} (Q, K, V) = \operatorname {C o n c a t} \left(\operatorname {h e a d} _ {1}, \dots , \operatorname {h e a d} _ {\mathrm {h}}\right) W ^ {O} \\ \text {w h e r e} \quad \text {h e a d} _ {i} = \text {A t t e n t i o n} \left(Q W _ {i} ^ {Q}, K W _ {i} ^ {K}, V W _ {i} ^ {V}\right) \tag {3} \\ \end{array}
$$
where  $W^{O}, W_{i}^{Q}, W_{i}^{K}$ , and  $W_{i}^{V}$  are learnable matrices, and  $h$  is the number of heads.
Inspired by the phenomenon that some words contained in a cross-lingual summary can be obtained by translating some source words (Figure 1), we introduce a novel cross-lingual summarization
<sup>1</sup>A multi-task method (Zhu et al., 2019) which trains cross-lingual summarization and machine translation using alternating training strategy.
Figure 2: Overview of our method. We first use encoder-decoder attention distribution to attend to some words and obtain the translation candidates from a probabilistic bilingual lexicon. Then a translating probability  $p_{\mathrm{trans}}$  is calculated, which balances the probability of generating words from the neural distribution with that of selecting words from the translation candidates of the source text. The final distribution is obtained by the weighted sum (weighed by  $p_{\mathrm{trans}}$ ) of the neural distribution  $P_{\mathrm{N}}$  and the translation distribution  $P_{\mathrm{T}}$ . Best viewed in color.
method. It first attends to some source words, then obtains the translation candidates of them, and finally generates words from the translation candidates or the neural distribution. Our proposed method is a hybrid between Transformer and an additional translation layer, which is depicted in Figure 2 and described as follows.
Attend. Inspired by the pointer-generator network (See et al., 2017), we employ the encoder-decoder attention distribution  $\alpha_{t}^{h}$  (the last layer) to help focus on some salient words in the source text. Since  $\alpha_{t}^{h}$  is a multi-head attention, we take the mean value over the heads as follow:
$$
\alpha_ {t} = \frac {1}{h} \sum_ {h} \alpha_ {t} ^ {h} \tag {4}
$$
Translate. With the attention distribution on the source words, we also need to know what should each source word be translated into. To achieve that, we obtain a probabilistic bilingual lexicon  $P^{L}(w_{1} \Rightarrow w_{2})$  from existing machine translation corpora and then acquire the translation probability  $P_{\mathrm{T}}$  based on  $P^{L}(w_{1} \Rightarrow w_{2})$ .
Acquisition of the probabilistic bilingual lexicon. There are many different ways to get the probabilistic bilingual lexicon, such as learning from bilingual corpora (Dyer et al., 2013; Chandar A P et al., 2014; Artetxe et al., 2016) and learning from
monolingual corpora (Conneau et al., 2018; Zhang et al., 2017; Artetxe et al., 2018). To facilitate access to the high-quality probabilistic bilingual lexicon, we apply the method described in Dyer et al. (2013). Specifically, we first extract word alignments L using the fast-align tool (Dyer et al., 2013) on the bilingual parallel corpus² for machine translation in both source-to-target and target-to-source directions. To improve the quality of the word alignments, we only keep the alignments existing in both directions. Next, the lexicon translation probability  $P^L(w_1 \Rightarrow w_2)$  is the average of source-to-target and target-to-source probabilities calculated through maximum likelihood estimation on word alignments L. We filter the lexicon pairs  $(w_1, w_2)$ , where  $P^L(w_1 \Rightarrow w_2) < 0.05$ , and renormalize the lexicon translation probabilities to get the final probabilistic bilingual lexicon.
We propose the following three different strategies (Figure 3) to obtain the translation probability:
(1) Naive. We directly use the probability in the probabilistic bilingual lexicon as the translation probability. We limit the number of translation candidates of the word  $w_{1}$  to at most  $m$ . Specifically,
2We employ the 2.08M sentence pairs from the LDC corpora which includes LDC2000T50, LDC2002L27, LDC2002T01, LDC2002E18, LDC2003E07, LDC2003E14, LDC2003T17, LDC2004T07.
Figure 3: Overview of our three strategies to obtain the translation probability from the probabilistic bilingual lexicon. We take  $m = 3$  for example.
we sort the translation candidates of word  $w_{1}$  in descending order according to the lexicon translation probability and then take the top- $m$ . Finally, the lexicon translation probability will be normalized to get the translation probability:
$$
P _ {\mathrm {T}} \left(w _ {1} \Rightarrow w _ {2}\right) = \frac {P ^ {L} \left(w _ {1} \Rightarrow w _ {2}\right)}{\sum_ {w _ {j}} P ^ {L} \left(w _ {1} \Rightarrow w _ {j}\right)} \tag {5}
$$
(2) Equal. The Naive strategy will bring about a problem that the decoder tends to select the words with the high probability from the translation candidates of source word  $w_{1}$ , and those with low translation probability will hardly be selected. To alleviate this, we set the translation probability of  $w_{1}$ 's translation candidates to be equal. Therefore, which translation candidate will eventually be selected depends on the probability of these translation candidates in the neural distribution. This strategy can be considered to achieve the goal of small vocabulary with the help of translation knowledge.
(3) Adapt. This strategy aims to select the correct translation candidates by source-side context. Specifically, we first limit the number of translation candidates to at most  $m$ , which is consistent with the two strategies above. Then we propose a translation-attention which is a multi-head attention block, where the hidden state of the source word  $w_{1}$  is fed as the query and the target-side embedding of the corresponding translation candidates will be treated as the keys and values.
$$
P _ {\mathrm {T}} \left(w _ {1} \Rightarrow w _ {2}\right) = \text {A t t e n t i o n} \left(w _ {1}, w _ {2} ^ {\mathrm {t g t}}, w _ {2} ^ {\mathrm {t g t}}\right) \tag {6}
$$
where  $w_{2}^{\mathrm{tgt}}$  is the target-side embedding of word  $w_{2}$ . We also take the mean value of the multihead translation-attention as the final translation probability. Since the hidden state of the source word  $w_{1}$  is obtained by the self-attention on the source-side, this context-aware strategy can help the model learn to choose the correct translation adaptively with the help of the source-side context.
Summarize. We use  $\mathrm{H_{dec}}$  to represent the decoder hidden state at timestep  $t$  and  $d_{\mathrm{model}}$  to denote the dimension of the hidden states. We employ a translation layer to determine the translating probability  $p_{\mathrm{trans}} \in [0,1]$  via a dynamic gate:
$$
p _ {\text {t r a n s}} = \sigma \left(\mathbf {W} _ {2} \left(\mathbf {W} _ {1} \mathrm {H} _ {\mathrm {d e c}} + b _ {1}\right) + b _ {2}\right) \tag {7}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d_{\mathrm{model}}\times d_{\mathrm{model}}}$  and  $\mathbf{W}_2\in \mathbb{R}^{1\times d_{\mathrm{model}}}$  are learnable matrices,  $b_{1}\in \mathbb{R}^{d_{\mathrm{model}}}$  and  $b_{2}\in \mathbb{R}^{1}$  are bias vectors,  $\sigma$  is the sigmoid function. Then  $p_{\mathrm{trans}}$  is regarded as a soft switch to determine whether to generate a word  $w$  by sampling from the neural distribution or directly select a word from the translation candidates of the source words. Therefore, the final probability distribution can be calculated as follow:
$$
\begin{array}{l} P (w) = p _ {\text {t r a n s}} \sum_ {i: w _ {i} = w _ {\text {s r c}}} \alpha_ {t, i} P _ {\mathrm {T}} \left(w _ {\text {s r c}} \Rightarrow w\right) \tag {8} \\ + (1 - p _ {\mathrm {t r a n s}}) P _ {\mathrm {N}} (w) \\ \end{array}
$$
where  $P_{\mathrm{T}}(w_{\mathrm{src}} \Rightarrow w)$  denotes the translation probability of word  $w_{\mathrm{src}}$  to word  $w$  and  $P_{\mathrm{N}}$  means the neural distribution.
In this study, we focus on Chinese-to-English and English-to-Chinese cross-lingual summarization. We test our proposed method on En2ZhSum and Zh2EnSum datasets $^3$  released by Zhu et al. (2019). En2ZhSum is an English-to-Chinese summarization dataset, which contains 370,687 English documents (755 tokens on average) paired with multisentence English (55 tokens on average) and Chinese summaries (96 Chinese characters on average). The dataset is split into 364,687 training pairs, 3,000 validation pairs, and 3,000 test pairs. Zh2EnSum is a Chinese-to-English summarization dataset, which contains 1,699,713 Chinese short texts (104 Chinese characters on average) paired with Chinese (18 Chinese characters on average) and English short summaries (14 tokens on average). The dataset is split into 1,693,713 training pairs, 3,000 validation pairs, and 3,000 test pairs. Both the English-to-Chinese and Chinese-to-English test sets are manually corrected.
We follow the setting of the vocabularies described in Zhu et al. (2019). In En2ZhSum, we surround each target sentence with tags “<t>” and “</t>. If there is no special explanation, the limit on the number of translation candidate  $m$  in our models is set to 10. All the parameters are initialized via Xavier initialization method (Glorot and Bengio, 2010). We train our models using configuration transformer_base (Vaswani et al., 2017), which contains a 6-layer encoder and a 6-layer decoder with 512-dimensional hidden representations. Each mini-batch contains a set of document-summary pairs with roughly 3,072 source and 3,072 target tokens. We apply Adam optimizer (Kingma and Ba, 2015) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.998$ , and  $\epsilon = 10^{-9}$ . For evaluation, we use beam search with a beam size 4 and length penalty 0.6. All our methods are trained and tested on a single NVIDIA TITAN XP.
4.3 Comparative Methods
We compare our method with the following relevant methods (Zhu et al., 2019):
- GETran: It first translates the original article into the target language by Google Translator
and then summarizes the translated text via LexRank (Erkan and Radev, 2004).
- GLTran: It first summarizes the original article via a Transformer-based monolingual summarization model and then translates the summary into the target language by Google Translator.
- TNCLS: It denotes the Transformer-based neural cross-lingual summarization system.
The above methods only employ the cross-lingual summarization dataset, and we also compare our method with the following two methods (Zhu et al., 2019) that use extra datasets in other tasks.
- CLSMS: It refers to the multi-task method, which simultaneously trains cross-lingual summarization and monolingual summarization.
- CLSMT: It is the multi-task method which adopts the alternating training strategy (Dong et al., 2015) to train cross-lingual summarization and machine translation jointly.
We denote our method as ATS:
- ATS: It refers to our method with three different strategies (Naive, Equal, and Adapt).
We evaluate all models with the standard ROUGE metric (Lin, 2004), reporting the F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L. All ROUGE scores are reported by the  $95\%$  confidence interval measured by the official script. Besides, we evaluate the equality of English summaries in Zh2EnSum with MoverScore (Zhao et al., 2019) which compares system output against references based on their semantics rather than surface forms. Zhao et al. (2019) have shown that MoverScore has a higher correlation with human judgment than ROUGE on evaluating English summaries.
Results on Zh2EnSum and En2ZhSum. Table 1 shows the results of different models on Zh2EnSum test set, while Table 2 gives the results on En2ZhSum test set. We use "subword-subword" and "word-character" segmentation granularities in Zh2EnSum and En2ZhSum, respectively.
We find that ATS can significantly outperform the baseline TNCLS on both Zh2EnSum and
4The parameter for ROUGE script here is "-c 95 -r 1000 -n 2 -a".
3http://www.nlpr.ia.ac.cn/cip/dataset.htm
Model	RG-1	RG-2	RG-L	MVS
Baseline	GETran	24.34	9.14	20.13	0.64
GLTran	35.45	16.86	31.28	16.90
TNCLS	38.85	21.93	35.05	19.43
Baseline +Extra Data	CLSMS	40.34	22.65	36.39	21.09
CLSMT	40.25	22.58	36.21	21.06
ATS	Naive	40.40	23.82†	36.63	21.86*
Equal	40.10	23.36*	36.22	21.41
Adapt	40.68	24.12†	36.97	22.15
Table 1: ROUGE F1 scores  $(\%)$  and MoverScore scores  $(\%)$  on Zh2EnSum test set. RG and MVS refer to ROUGE and MoverScore, respectively. We adopt "subword-subword" segmentation granularity here. The improvement of all ATS models over the baseline TNCLS is statistically significant  $(p < 0.01)$ .  $* (\dagger)$  indicates that the improvement over CLSMS is statistically significant where  $p < 0.05$  (0.01).
Model	RG-1	RG-2	RG-L
Baseline	GETran	28.19	11.40	25.77
GLTran	32.17	13.85	29.43
TNCLS	36.82	18.72	33.20
Baseline +Extra Data	CLSMS	38.25	20.20	34.76
CLSMT	40.23	22.32	36.59
ATS	Naive	40.19	21.84	36.46
Equal	39.98	21.63	36.29
Adapt	40.47	22.21	36.89
En2ZhSum. Furthermore, ATS can significantly outperform CLSMS and CLSMT on Zh2EnSum while achieving comparable performance with CLSMS and CLSMT on En2ZhSum. However, both CLSMS and CLSMT employ large-scale parallel datasets of other tasks during the training process, limiting the generality of the models. In contrast, our method only requires an extra probabilistic bilingual lexicon, which significantly reduces the dependence on data. Among the variants of ATS, the ATS with Adapt strategy has the best performance. The reason is quite straightforward since the Adapt strategy helps to choose the right translation with the help of the source-side context. The Equal strategy performs worst, but its advantage over the Naive strategy is that it is not affected by the prior probability in probabilistic bilingual lexicon. In other words, the Equal strategy only makes use of the corresponding relationship be
Table 2: ROUGE F1 scores  $(\%)$  on En2ZhSum test set. RG refers to ROUGE for short. We adopt "word-character" segmentation granularity here. The improvement of all ATS models over both TNCLS and CLSMS is statistically significant  $(p < 0.01)$ .
Src-Tgt	Model	Size (M)	Train (S)
Zh-En	TNCLS	134.92	21
CLSMS	211.41	48
CLSMT	208.84	63
ATS-NE	136.55	27
ATS-A	137.60	30
En-Zh	TNCLS	113.74	24
CLSMS	190.23	65
CLSMT	148.16	72
ATS-NE	114.00	24
ATS-A	115.05	25
Table 3: Model size (number of trainable parameters and M denotes mega) and training time of various models. Train (S) denotes how many seconds required for each model to train the 100-batch cross-lingual summarization task of the same batch size (3072). ATS-NE refers to our method with the Naive or Equal strategy. ATS-A is the one with Adapt strategy.
tween source language words and target language words, making it effective even if there is only a bilingual vocabulary dictionary. In summary, all three of our strategies can bring about significant improvement, which demonstrates that our method is robust to the acquisition method of translation candidates.
Model size and training time. The model size and training time of various models are given in Table 3. As it is shown, ATS is comparable with Transformer from both model size and training time. For model size, ATS is significantly less than the multi-task methods CLSMS and CLSMT. Especially on the Zh2En task, the size of multi-task models is nearly twice that of ATS. For training time, ATS is roughly half of the multi-task methods on both Zh2En and En2Zh tasks. Therefore, compared with the multi-task methods, ATS can significantly reduce the model size and improve the training efficiency.
In conclusion, our ATS models have achieved significant improvements over the baseline TNCLS on both Zh2EnSum and En2ZhSum, which can demonstrate the effectiveness of our approach. Furthermore, ATS achieves comparable performance with the state-of-the-art. Compared with the state-of-the-art, ATS can not only relax model's dependence on datasets but also reduce model size and improve training efficiency.
The impact of  $m$ . To study the impact of  $m$  (the limit on the number of translation candidates), we conduct an experiment on how the model performance changes when  $m$  varies from 10 to 5 or a
Model	m	Zh2En	En2Zh
RG-1	RG-2	RG-L	MVS	RG-1	RG-2	RG-L
ATS-A	1	40.93	24.17	37.11	22.31	39.85	21.45	36.12
5	41.05	24.31	37.28	22.77	40.27	21.96	36.60
10	40.68	24.12	36.97	22.15	40.47	22.21	36.89
Table 4: Results of ATS on Zh2EnSum and En2ZhSum under different hyperparameters, where  $m$  is the limit on the number of translation candidates. RG and MVS refer to ROUGE and MoverScore, respectively. We adopt "subword-subword" and "word-character" segmentation granularities in Zh2En and En2Zh models, respectively.
Model	Unit	RG-1	RG-2	RG-L	MVS
TNCLS	w-w	37.70	21.15	34.05	19.43
sw-sw	38.85	21.93	35.05	19.07
ATS-A	w-w	39.65	23.79	36.05	22.06
sw-sw	40.68	24.12	36.97	22.15