SlowGuess/ACL_2020_OCR huggingface dataset:
more aggressive value 1. The results are presented in Table 4. In Zh2En experiment, the ATS-A  $(m = 5)$  performs best while ATS-A  $(m = 1)$  performs comparably with ATS-A  $(m = 10)$ . In En2Zh experiment, the ATS-A  $(m = 5)$  performs comparably with ATS-A  $(m = 10)$  while the performance drops a bit when  $m = 1$ . The above results illustrate that (1) A slightly larger  $m$  enables the model to learn when to search for translation candidates from the source words and which ones to choose, leading to improve the quality of the final summaries. (2) When  $m = 1$ , the translation probability will contain some noise, but our method is still significantly better than the baseline, which further demonstrates the effectiveness and robustness of our method.
The impact of segmentation granularity. To study the effect of different segmentation granularities on the performance, we compare the performance of the model trained with "word-word" and "subword-subword" segmentation granularities on Zh2EnSum dataset. The results are given in Table 5. From ROUGE, our method brings about a similar degree of improvement over the baseline when using these two segmentation granularities. From MoverScore, it can be found that our method brings slightly greater improvement over the baseline when using the "subword-subword" segmentation granularity than using the "word-word" segmentation granularity. MoverScore metric com
Table 5: Results of models on Zh2EnSum with different segmentation granularities. Unit represents the granularity combination of text units.  $w$  and  $sw$  denote "word" and "subword" (Sennrich et al., 2016), respectively. The improvement of all ATS models over TNCLS is statistically significant  $(p < 0.01)$ .
Task	Unit	\( p_{trans}^{\text{macro}} \)	\( p_{trans}^{\text{micro}} \)	\( r^{\text{macro}} \)	\( r^{\text{micro}} \)
Zh2En	sw-sw	21.41	20.71	21.86	21.00
Zh2En	w-w	21.17	20.46	21.90	21.05
En2Zh	w-c	14.91	14.84	14.27	14.05
Table 6: Statistics on  $p_{\mathrm{trans}}$  in ATS-A models.  $p_{\mathrm{trans}}^{\mathrm{macro}}$  (\%) and  $p_{\mathrm{trans}}^{\mathrm{micro}}$  (\%) respectively represent the macro-average and micro-average translating probability during decoding.  $\mathbf{r}^{\mathrm{macro}}(\%)$  and  $\mathbf{r}^{\mathrm{micro}}(\%)$  respectively represent the ratio of words where  $p_{\mathrm{trans}} > 0.5$  during decoding.
pares system output against references based on their semantics, thus we believe ATS-A (sw-sw) can improve the semantic accuracy of the generated summary to a greater extent than ATS-A (w-w). Although the obtained probabilistic bilingual lexicon is of lower quality when using a smaller segmentation granularity, the source side covers more units, thus more translation candidates are exposed, making up for the noise in the probabilistic bilingual dictionary. In summary, our method can improve the performance under the above two different segmentation granularities, which illustrates that our method is robust to the segmentation granularity.
Translating Probability. Table 6 gives the statistics of translating probability in different ATS-A models. As it is shown, there is little difference in average translating probability under different segmentation granularities. However, the translation probabilities in tasks with different language directions are quite different. It is worth noting that the ration of words with translating probability greater than 0.5 does not mean that so many words are generated from translation operations, since the final distribution of summary words is jointly determined by translating probability, translation probability, encoder-decoder attention distribution, and neural distribution.
Human Evaluation. We conduct the human evaluation on 25 random samples extracted from each of Zh2EnSum and En2ZhSum, respectively.
Model	Zh2En	En2Zh
IF	CC	FL	IF	CC	FL
TNCLS	3.34	4.00	3.78	3.08	3.28	3.12
CLSMS	3.56	4.12	3.92	3.28	3.40	3.36
CLSMT	3.44	4.08	4.04	3.38	3.56	3.48
ATS-A	3.64	4.16	4.18	3.36	3.54	3.52
We compare the summaries generated by ATS (Adapt strategy) with other methods (including TNCLS, CLSMS, and CLSMT). Three graduate students are recruited to rate the generated summaries according to the references. Each summary is assessed from the three independent aspects: (1) How informative is the summary? (2) How concise is the summary? (3) How fluent and grammatical is the summary? Each aspect is scored from 1 (worst) to 5 (Best). The average results are given in Table 7.
We can find that the informativeness score, conciseness score, and fluency score of ATS-A are significantly better than those of the baseline TNCLS, which further demonstrates the effectiveness of our method. In Zh2En task, ATS-A outperforms CLSMT from all three aspects. The conciseness score of ATS-A is comparable with that of CLSMS, but ATS can generate more informative and fluent summaries. In En2Zh task, ATS-A outperforms CLSMS from all three aspects as well. The informativeness score and conciseness score of CLSMT are comparable with those of ATS-A, but ATS-A can generate more fluent summaries. To sum up, ATS-A can outperform CLSMS and CLSMT in Zh2En task, and ATS-A can outperform CLSMS while performing comparably with CLSMT in En2Zh task.
We show a case study of a sample from Zh2EnSum test set. The summaries generated by each model are presented in Figure 4.
Although the summary generated by the TNCLS captures the critical character "the former director of zengcheng health" and the crime of "received bribes" committed by the character, it mistakenly expresses "sentenced" as "arrested" and fails to identify the prison term. Both CLSMT-generated summary and CLSMS-generated summary are fluent and grammatically correct. How
Table 7: Human evaluation results. IF, CC, and FL represent informativeness, conciseness, and fluency, respectively.
Input (Chinese): 从广州市中级人民法院了解到，广东省增城市卫生局原局长郭铁军收受下属医疗单位20名负责人赠送的节日礼金近34万元一案，该院二审维持一审原判，驳回郭铁军上诉，以受贿罪判处其有期徒刑5年半。According to the Guangzhou Intermediate People's Court, Guo Tiejun, former director of the Zengcheng Municipal Health Bureau in Guangdong Province, received bribes nearly 340,000 yuan in holiday gifts from 20 persons in charge of subordinate medical units. The court upheld the original judgment in the first instance, rejected Guo Tiejun's appeal and sentenced him to five and a half years in prison for bribery. (The English Translation of Source Text)
Reference: zengcheng 's former director of health received bribes and was sentenced to five and a half years' imprisonment
TNCLS: the former director of zengcheng health bureau was arrested on suspicion of accepting bribes
CLSMT: guo tiejun , former director of zengcheng health bureau , was sentenced to five and a half years 'imprisonment for accepting bribes of 340,000 yuan
CLSMS: the former director of zengcheng health bureau was sentenced to five and a half years 'imprisonment for accepting bribes of nearly 340,000 yuan
ATS-A: the former director of zengcheng health bureau was sentenced to five and a half years for bribery
Figure 4: Examples of generated summaries. The English translation of source text is also given for better reading. The blue shading intensity denotes the value of the translating probability  $p_{\mathrm{trans}}$ .
ever, the amount in the source article is an approximate value "nearly 340,000 yuan", while CLSMT-generated summary directly expresses the exact value, which is inappropriate. The downside to both CLSMT-generated summary and CLSMS-generated summary is that they contain redundant information, since they are relatively lengthy. The summary generated by our ATS-A method matches the reference best and nearly captures all the key points in the source article. In conclusion, our method can generate summaries with more accurate semantics than baselines.
Cross-Lingual Summarization. The traditional cross-lingual summarization approaches are based on the pipelined paradigm and can be categorized into translate-then-summarize (Leuski et al., 2003; Ouyang et al., 2019) and summarize-then-translate (Orasan and Chiorean, 2008; Wan et al., 2010). Leuski et al. (2003) translate the Hindi document into English and then generate the English headline. Ouyang et al. (2019) train a robust abstractive summarization system on noisy English documents and clean English reference summaries. Then the system can learn to produce fluent summaries from disfluent inputs, which enables the system to summarize translated documents.
Orasan and Chiorean (2008) summarize the Romanian news and then translate the summary into English. Wan et al. (2010) apply the summarizethen-translate scheme to English-to-Chinese crosslingual summarization, which extracts English sentences considering both the informativeness and translation quality of sentences and automatically translates the English summary into Chinese. They also argue that summarize-then-translate is better, since it can alleviate both the computational expense of translating sentences and sentence extraction errors caused by incorrect translations.
There have been some researches focusing on improving cross-lingual summarization with bilingual information. Wan (2011) translates the English document into Chinese and extracts sentences based on the original English sentences and Chinese translation. Yao et al. (2015) propose a compressive method which calculates the sentence scores based on the aligned bilingual phrases obtained by machine translation service and performs compression via deleting redundant or poorly translated phrases. Zhang et al. (2016) introduce an abstractive method that constructs a pool of bilingual concepts represented by the bilingual elements of the source-side predicate-argument structures and the target-side counterparts.
Recently, end-to-end methods have been applied to cross-lingual summarization. Due to the lack of supervised training data, Ayana et al. (2018) and Duan et al. (2019) focus on zero-shot training methods that use machine translation or monolingual summarization or both to teach the cross-lingual system. Zhu et al. (2019) propose to acquire large-scale datasets via a round-trip translation strategy. They incorporate monolingual summarization or machine translation into cross-lingual summarization training using multi-task learning.
Neural Abstractive Summarization. Rush et al. (2015) present the first neural abstractive summarization model, an attentive convolutional encoder and a neural network language model decoder, which learns to generate news headlines from the lead sentences of news articles. Their approach has been further improved with recurrent decoders (Chopra et al., 2016), abstractive meaning representations (Takase et al., 2016), hierarchical networks (Nallapati et al., 2016), variational autoencoders (Miao and Blunsom, 2016), hybrid strategy (Zhu et al., 2017), selective mechanism (Zhou et al., 2017), and entailment knowledge. See et al.
(2017) propose a pointer-generator network, which allows copying words from the source text with the copying mechanism (Gu et al., 2016). Li et al. (2018) incorporate entailment knowledge into summarization model to improve the correctness of the generated summaries. Li et al. (2020) apply guidance signals of keywords to both the encoder and decoder in the abstractive summarization model.
Inspired by the pointer-generator network and the translation pattern in obtaining cross-lingual summaries, we introduce a novel model in this paper, which integrates the operation of attending, translating, and summarizing.
In this paper, we present a novel method consistent with the translation pattern in the process of obtaining a cross-lingual summary. This method first attends to the source words, then obtains the translation candidates, and incorporates them into the generation of the final summary. Experimental results have shown that our method can significantly outperform the baseline and achieve comparable performance with the state-of-the-art. Furthermore, our method has two advantages over the state-of-the-art: (1) Our model requires only an additional probabilistic bilingual lexicon rather than large-scale parallel datasets of other tasks, thus reducing the model's dependence on data and making it easier for the model to migrate to other domains or other language pairs. (2) Our model has a much smaller size and a much faster training efficiency.
In our future work, we consider incorporating our method into the multi-task method. Besides, we will also explore the influence of probabilistic bilingual lexicon obtained by learning only from monolingual data on our method.
The research work described in this paper has been supported by the National Key Research and Development Program of China under Grant No. 2017YFC0820700. We thank the three anonymous reviewers for their insightful comments and suggestions. We would like to thank Haitao Lin, Yu Lu, Yining Wang, Lu Xiang, and Yang Zhao for their invaluable contributions in shaping the early stage of this work. We thank Jinliang Lu, Xina Fu, and Meng Zhang for conducting human evaluation. We make our code publicly available here:
https://github.com/ZNLP/ATSum.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2289-2294. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 789-798. Association for Computational Linguistics.
Ayana, shi-qi Shen, Yun Chen, Cheng Yang, Zhi-yuan Liu, and Maosong Sun. 2018. Zero-shot crosslingual neural headline generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 26(12):2319-2327.
Sarath Chandar A P, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems 27 (NIPS), pages 1853-1861. Curran Associates, Inc.
Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 93-98. Association for Computational Linguistics.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations (ICLR).
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 1723-1732. Association for Computational Linguistics.
Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019. Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3162-3172. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 644-648. Association for Computational Linguistics.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research (JAIR), 22:457-479.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics, pages 249-256.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1631-1640. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Anton Leuski, Chin-Yew Lin, Liang Zhou, Ulrich German, Franz Josef Och, and Eduard Hovy. 2003. Cross-lingual c* st* rd: English access to Hindi information. ACM Transactions on Asian Language Information Processing (TALIP), 2(3):245-269.
Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 1430-1441. Association for Computational Linguistics.
Haoran Li, Junnan Zhu, Jiajun Zhang, Chengqing Zong, and Xiaodong He. 2020. Keywords-guided abstractive sentence summarization. In Proceedings of the Thirty-Forth AAAI Conference on Artificial Intelligence (AAAI).
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81. Association for Computational Linguistics.
Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 319-328. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero Dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning (CONLL), pages 280-290. Association for Computational Linguistics.
Constantin Orasan and Oana Andreea Chiorean. 2008. Evaluation of a cross-lingual Romanian-English multi-document summariser. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08). European Language Resources Association (ELRA).
Jessica Ouyang, Boya Song, and Kathy McKeown. 2019. A robust abstractive system for cross-lingual summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2025–2031. Association for Computational Linguistics.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 379-389. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073-1083. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1715-1725. Association for Computational Linguistics.
Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata. 2016. Neural headline generation on abstract meaning representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1054-1059. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS), pages 5998-6008. Curran Associates, Inc.
Xiaojun Wan. 2011. Using bilingual information for cross-language document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1546-1555. Association for Computational Linguistics.
Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010. Cross-language document summarization based on machine translation quality prediction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 917-926. Association for Computational Linguistics.
Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015. Phrase-based compressive cross-language summarization. In Proceedings of the 2015 conference on empirical methods in natural language processing (EMNLP), pages 118-127. Association for Computational Linguistics.
Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2016. Abstractive cross-language summarization via translation model enhanced predicate argument structure fusing. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 24(10):1842-1853.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1959-1970. Association for Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578. Association for Computational Linguistics.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1095-1104. Association for Computational Linguistics.
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3045–3055. Association for Computational Linguistics.
Junnan Zhu, Long Zhou, Haoran Li, Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2017. Augmenting neural sentence summarization through extractive summarization. In Proceedings of the 6th Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 16-28. Springer.
Zh2EnSum	train	valid	test
#Documents	1,693,713	3,000	3,000
#AvgChars (S)	103.59	103.56	140.06
#AvgWords (R)	13.70	13.74	13.84
#AvgSentChars	52.73	52.41	53.38
#AvgSents	2.32	2.33	2.30
Table 8: Corpus statistics of Zh2EnSum. #AvgChars (S) is the average number of Chinese characters in the source document. #AvgWords (R) means the average number of English words in the reference. #AvgSentChars refers to the average number of characters in a sentence in the source document. #AvgSents denotes the average number of sentences in the source document.
En2ZhSum	train	valid	test
#Documents	364,687	3,000	3,000
#AvgWords (S)	755.09	759.55	744.84
#AvgChars (R)	55.21	55.28	54.76
#AvgSentWords	19.62	19.63	19.61
#AvgSents	40.62	41.08	40.25
Datasets. Table 8 and Table 9 show the statistics of Zh2EnSum dataset and En2ZhSum dataset, respectively.
Table 9: Corpus statistics of En2ZhSum. #AvgWords (S) is the average number of English words in the source document. #AvgChars (R) means the average number of Chinese characters in the reference. #AvgSentWords refers to the average number of Words in a sentence in the source document. #AvgSents denotes the average number of sentences in the source document.
Task	Unit	Source	Target
Zh2En	sw-sw	100,000	40,000
Zh2En	w-w	100,000	40,000
En2Zh	w-c	100,000	18,000
Table 10: The vocabulary size of models with different segmentation granularities.