SlowGuess/ACL_2020_OCR huggingface dataset:
DUC2004 DUC2004 corpus only contains test sets. We use the model trained on gigaword corpus to generate summaries on DUC2004 test sets. We use the 500 human-translated test samples provided by (Duan et al., 2019) to do Zh-to-En tests.
LCSTS LCSTS (Hu et al., 2015) is a Chinese summarization corpus, which contains 2.40M training pairs, 10,666 validation pairs, and 725 test pairs. We use 3K cross-lingual test samples provided by Zhu et al. (2019) to do Zh-to-En tests.
CNN/DM CNN/DM (Hermann et al., 2015) contains 287.2K training pairs, 13.3K validation pairs, and 11.5K test pairs. We use the 3K cross-lingual test samples provided by Zhu et al. (2019) to do En-to-Zh cross-lingual tests.
We use ROUGE-1 (unigram), ROUGE-2 (bigram), and ROUGE-L (LCS) F1 scores as the evaluation metrics, which are most commonly used evaluation metrics in the summarization task.
6.4 Competitive Models
For unsupervised cross-lingual summarization, we set the following baselines:
- Unified It jointly trains English and Chinese monolingual summarizations in a unified model and uses the first token of the decoder to control whether it generates Chinese or English summaries.
- Unified+CLWE It builds a unified model and adopts pre-trained unsupervised cross-lingual word embeddings. The cross-lingual word embeddings are obtained via projecting embeddings from source language to target language. We use Vecmap² to learn the cross-lingual word embeddings.
For supervised cross-lingual summarization, we compare our model with (Shen et al., 2018), (Duan et al., 2019), and Zhu et al. (2019). We also consider the following baselines for comparison:
- Pipe-TS The Pipe-TS baseline first uses a Transformer-based translation model to translate the source text to the other language, then uses a monolingual summarization model to generate summaries. To make this baseline stronger, we replace the translation model with the Google translation system and name it as Pipe-TS*.
- Pipe-ST The Pipe-ST baseline first uses a monolingual summarization model to generate the summaries, then uses a translation model to translate the summaries to the other language. We replace the translation model with the Google translation system as Pipe-ST*.
- Pseudo The Pseudo baseline directly trains a cross-lingual summarization model by using the pseudo parallel cross-lingual summarization data.
- XLM Pretraining This method is proposed by Lample and Conneau (2019), where they pretrain the encoder and decoder on large-scale multilingual text using causal language modeling (CLM), masked language modeling (MLM), and translation language modeling (TLM) tasks.
6.5 Implementation Details
For transformer architectures, we use the same configuration as Vaswani et al. (2017), where the number of layers, model hidden size, feed-forward hidden size, and the number of heads are 6, 512, 1024, and 8, respectively. We set  $g = \sqrt{d_{\mathrm{model}}} = \sqrt{512}$  in ScaleNorm. The mapper is a linear layer with a hidden size of 512, and the discriminator is a two-layer linear layer with a hidden size of 2048.
We use the NLTK $^4$  tool to process English texts and use jieba $^5$  tool to process Chinese texts. The vocabulary size of English words and Chinese words are 50,000 and 80,000 respectively. We set  $\lambda_1 = 1, \lambda_2 = 5, \lambda_3 = 2$  in unsupervised training and  $\lambda_1 = 0.5, \lambda_2 = 5$  in supervised training according to the performance of the validation set. We set dis_iters = 5 in Alg. 1.
3This baseline was suggested by the reviewers, and the results are only for reference since it additionally uses a lot of pre-training text.
4https://github.com/pltk/pltk
<sup>5</sup>https://github.com/fxsjy/jieba
$^{2}$ https://github.com/artetxem/vecmap
6225
Method	Zh-to-En	En-to-Zh
Gigaword	DUC2004	LCSTS	CNN/DM
R1	R2	RL	R1	R2	RL	R1	R2	RL	R1	R2	RL
Pipe-TS	22.27	6.58	20.53	21.29	5.96	17.99	27.26	10.41	21.72	-	-	-
Pipe-ST	28.27	11.90	26.50	25.73	8.19	21.60	36.48	18.87	31.44	25.95	11.01	23.29
Pipe-TS*	22.52	6.67	20.76	21.83	6.11	18.42	29.29	11.09	23.18	-	-	-
Pipe-ST*	29.56	12.50	26.42	26.66	8.51	22.37	38.26	19.56	32.93	27.82	11.78	24.97
Pseudo*	30.93	13.25	27.29	27.03	8.49	23.08	38.61	19.76	34.63	35.81	14.96	32.07
(Shen et al., 2018)	21.5	6.6	19.6	19.3	4.3	17.0	-	-	-	-	-	-
(Duan et al., 2019)	30.1	12.2	27.7	26.0	8.0	23.1	-	-	-	-	-	-
(Zhu et al., 2019)	-	-	-	-	-	-	40.34	22.65	36.39	38.25	20.20	34.76
(Zhu et al., 2019) w/LDC	-	-	-	-	-	-	40.25	22.58	36.21	40.23	22.32	36.59
XLM Pretraining	32.28	14.03	28.19	28.27	9.40	23.78	42.75	22.80	38.73	39.11	17.57	34.14
Ours	32.04	13.60	27.91	27.25	8.71	23.36	40.97	23.20	36.96	38.12	16.76	33.86
We use Adam optimizer (Kingma and Ba, 2014) with  $\beta = (0.9, 0.98)$  for optimization. We set the learning rate to  $3e - 4$  and adopt the warm-up learning rate (Goyal et al., 2017) for the first 2,000 steps, the initial warm-up learning is set to  $1e - 7$ . We adopt the dropout technique and set the dropout rate to 0.2.
7 Results and Analysis
7.1 Unsupervised Cross-Lingual Summarization
The experiment results of unsupervised crosslingual summarization are shown in Table 2, and it can be seen that our model significantly outperforms all baselines by a large margin. By training a unified model of all languages, the model's crosslingual transferability is still poor, especially for the gigaword dataset. Incorporating cross-lingual word embeddings into the unified model can improve the performance, but the improvement is limited. We think this is due to that the cross-lingual word embeddings learned by Vecmap cannot leverage the contextual information. Due to space limitations, we present case studies in the Appendix.
After checking the generated summaries of the two baseline models, we find that they can generate readable texts, but the generated texts are far away from the theme of the source text. This indicates that the encoder and decoder of these baselines have a large gap, such that the decoder cannot understand the output of the encoder. We also find that summaries generated by our model are obviously more relevant, demonstrating that aligned representations between languages are helpful.
But we can also see that there is still a gap be
Table 1: Rouge F1 scores (\%) on cross-lingual summarization tests. "XLM Pretraining" and "Zhu et al. (2019) w/ LDC" use additional training data. Our model significantly  $(p < 0.01)$  outperforms all pipeline methods and pseudo-based methods.
Method	LCSTS	Gigaword
R1	R2	RL	R1	R2	RL
Unified	13.52	1.35	10.02	5.25	0.87	2.09
Unified+CLWE	14.02	1.49	12.10	6.51	1.07	2.92
Ours	20.11	5.46	16.07	13.75	4.29	11.82
Table 2: Rouge F1 scores  $(\%)$  on unsupervised crosslingual summarization tests. Our model outperforms all baselines significantly  $(p < 0.01)$ .
tween our unsupervised results (Table 2) and supervised results (Table 1), indicating that our model has room for improvement.
7.2 Supervised Cross-Lingual Summarization
The experiment results of supervised cross-lingual summarization are shown in Table 1. Due to the lack of corpus for training Chinese long document summarization model, we do not experiment with the Pipe-TS model on the CNN/DM dataset.
By comparing our results with pipeline-based or pseudo baselines, we can find that our model outperforms all these baselines in all cases. Our model achieves an improvement of  $0\sim 3$  Rouge scores over the Pseudo model trained directly with translated parallel cross-lingual corpus, and  $1.5\sim 4$  Rouge-1 scores over those pipeline models. We also observe that models using the Google translation system all perform better than models using the Transformer-based translation system. This may because the Transformer-based translation system will bring some "UNK" tokens, and the transformer-based translation system trained by ourselves does not perform as well as the Google translation system. In addition, Pipe-ST models perform better than Pipe-TS models, which is con
6226
Method	Info. ↑	Con. ↑	Flu. ↑
Reference	3.60	3.50	3.80
PipeST*	3.56	3.51	4.00
PipeTS*	3.37	3.80	3.81
Pseudo	3.27	3.81	3.89
Ours (supervised)	3.56	3.93	3.94
Ours (unsupervised)	2.18	3.34	2.87
Table 3: Results of the human evaluation on the gigaw-word dataset.
Method	Info. ↑	Con. ↑	Flu. ↑
Reference	3.58	3.57	4.21
PipeST*	3.38	3.45	4.13
PipeTS*	3.38	3.93	3.78
Pseudo	3.46	3.90	4.05
Ours (supervised)	3.55	4.03	4.13
sistent with the conclusions of previous work. This is because (1) the translation process may discard some informative clauses, (2) the domain of the translation corpus is different from the domain of summarization corpus, which will bring the domain discrepancy problem to the translation process, and (3) the translated texts are often "translationese" (Graham et al., 2019). The Pseudo model performs better than Pipe-TS models but performs similarly as Pipe-ST models.
By comparing our results with others, we can find that our model outperforms Shen et al. (2018) and Duan et al. (2019) on both gigaword and DUC2004 test sets, and it outperforms Zhu et al. (2019) on the LCSTS dataset. But our Rouge scores are lower than Zhu et al. (2019) on the CNN/DM dataset, especially the Rouge-2 score. However, our model performs worse than pretrained models.
7.3 Human Evaluation
The human evaluation was also performed. Since we cannot get the summaries generated by other models, we only compare with our baselines in the human evaluation. We randomly sample 50 examples from the gigaword (Zh-to-En) test set and 20 examples from the CNN/DM (En-to-Zh) test set. We ask five volunteers to evaluate the quality of the generated summaries from the following three aspects: (1) Informative: how much does the generated summaries cover the key content of the source text? (2) Conciseness: how concise are the generated summaries? (3) Fluency: how fluent are the generated summaries? The scores are
Table 4: Results of the human evaluation on the CNN/DM dataset.
Method	Gigaword	CNN/DM
R1	R2	RL	R1	R2	RL
Ours (supervised)	32.04	13.60	27.91	38.12	16.76	33.86
w/o summ. loss	30.36*	12.84*	26.41*	36.37*	15.97*	32.11*
w/o mappers	31.95	13.46	27.88	38.28	16.73	33.93
w/o ScaleNorm	31.27*	13.29	27.22*	37.01*	16.30*	32.87*
w/o pre. steps	31.33*	13.30	27.35*	37.23*	16.39	33.01*
Unshare enc/dec	30.10*	12.71*	26.28*	35.93*	15.86*	31.82*
Table 5: Results of ablation tests in supervised setting. Statistically significant improvement  $(p < 0.01)$  over the complete model are marked with  $*$ .
between 1-5, with 5 being the best. We average the scores and show the results in Table 3 and Table 4.
Our model exceeds all baselines in informative and conciseness scores, but get a slightly lower fluency score than Pipe-ST*. We think this is because the Google translation system has the ability to identify grammatical errors and generate fluent sentences.
7.4 Ablation Tests
To study the importance of different components of our model, we also test some variants of our model. For supervised training, we set variants of (1) without (monolingual) summarization loss, (2) without mappers $^6$ , (3) replace ScaleNorm with LayerNorm, (4) without pre-trained monolingual steps, and (5) unshare the encoder and decoder. For unsupervised training, we additionally set variants without cyc-reconstruction loss or back-translation loss. The results of ablation tests of supervised and unsupervised cross-lingual summarization are shown in Table 5 and Table 6, respectively.
It seems that the role of mappers does not seem obvious in the case of supervised training. We speculate that this may be due to the joint training of monolingual and cross-lingual summarizations, and directly constraining the context representations before mapping can also yield shared (aligned) representations. But mappers are crucial for unsupervised cross-lingual summarization. For supervised cross-lingual summarization, except for mappers, all components contribute to the improvement of the performance. The performance decreases after removing any of the components. For unsupervised cross-lingual summarization, all components contribute to the improvement of the performance and the mappers and shared encoder/decoder are key components.
In this case, we directly constrain the parallel  $z_{x}$  and  $z_{y}$  to be the same.
6227
Method	LCSTS	Gigaword
R1	R2	RL	R1	R2	RL
Ours (unsupervised)	20.10	5.46	16.07	13.75	4.29	11.82
w/o mappers	14.79*	2.29*	12.36*	6.26*	1.02*	3.11*
w/o cyc. loss	17.51*	4.70*	13.95*	7.21*	1.31*	4.04*
w/o back. loss	19.37	5.23	15.44	13.20	4.11	11.27
w/o ScaleNorm	19.24*	5.21	15.37*	13.15*	4.08	11.21
w/o pre. steps	19.70	5.24	15.72	13.13	4.10	10.91
Unshare enc/dec	12.28*	0.97*	10.37*	4.88*	0.82*	1.91*
Table 6: Results of the ablation tests of unsupervised cross-lingual summarization. Statistically significant improvement  $(p < 0.01)$  over the complete model are marked with  $*$ .
8.1 Cross-Lingual Summarization
Early researches on cross-lingual abstractive summarization are mainly based on the monolingual summarization methods and adopt different strategies to incorporate bilingual information into the pipeline model (Leuski et al., 2003; Orasan and Chiorean, 2008; Wan et al., 2010; Wan, 2011; Yao et al., 2015).
Recently, some neural cross-lingual summarization systems have been proposed for cross-lingual summarization (Shen et al., 2018; Duan et al., 2019; Zhu et al., 2019). The first neural-based cross-lingual summarization system was proposed by Shen et al. (2018), where they first translate the source texts from the source language to the target language to form the pseudo training samples. A teacher-student framework is adopted to achieve end-to-end cross-lingual summarization. Duan et al. (2019) adopt a similar framework to train the cross-lingual summarization model, but they translate the summaries rather than source texts to strengthen the teacher network. Zhu et al. (2019) propose a multi-task learning framework by jointly training cross-lingual summarization and monolingual summarization (or machine translation). They also released an English-Chinese cross-lingual summarization corpus with the aid of online translation services.
8.2 Learning Cross-Linguual Representations
Learning cross-lingual representations is a beneficial method for cross-lingual transfer.
Conneau et al. (2017) use adversarial networks to learn mappings between languages without supervision. They show that their method works very well for word translation, even for some distant language pairs like English-Chinese. Lample
et al. (2018) learn word mappings between languages to build an initial unsupervised machine translation model, and then perform iterative back-translation to fine-tune the model. Aldarmaki and Diab (2019) propose to directly map the averaged embeddings of aligned sentences in a parallel corpus, and achieve better performances than word-level mapping in some cases.
In this paper, we propose a framework that jointly learns to align and summarize for neural crosslingual summarization. We design training objectives for supervised and unsupervised cross-lingual summarizations, respectively. We also propose methods to enhance the isomorphism and crosslingual transfer between languages. Experimental results show that our model outperforms supervised baselines in most cases and outperforms unsupervised baselines in all cases.
This work was supported by National Natural Science Foundation of China (61772036), Tencent AI Lab Rhino-Bird Focused Research Program (No.JR201953), and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019. On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2440-2452.
Hanan Aldarmaki and Mona Diab. 2019. Context-aware cross-lingual mapping. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3906-3911.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations.
6228
Xilun Chen, Ahmed Hassan, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098-3112.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087.
Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019. Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3162-3172.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Priya Goyal, Piotr Dálár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. arXiv preprint arXiv:1906.09833.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. Lcsts: A large scale chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1967-1972.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Alexandre Klementiev, Ivan Titov, and Binod Bhattacharai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459-1474.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS).
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049.
Anton Leuski, Chin-Yew Lin, Liang Zhou, Ulrich German, Franz Josef Och, and Eduard Hovy. 2003. Cross-lingual c* st* rd: English access to Hindi information. ACM Transactions on Asian Language Information Processing (TALIP), 2(3):245-269.
Courtney Naples, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, pages 95-100. Association for Computational Linguistics.
Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895.
Constantin Orasan and Oana Andreea Chiorean. 2008. Evaluation of a cross-lingual Romanian-english multi-document summariser. In LREC 2008.
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. 2019. Analyzing the limitations of cross-lingual word embedding mappings. arXiv preprint arXiv:1906.05407.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. arXiv preprint arXiv:1908.06625.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1599–1613.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
Shi-qi Shen, Yun Chen, Cheng Yang, Zhi-yuan Liu, and Mao-song Sun. 2018. Zero-shot cross-lingual neural headline generation. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(12):2319-2327.
Anders Søgaard, Sebastian Ruder, and Ivan Vulic. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778-788.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
6229
Xiaojun Wan. 2011. Using bilingual information for cross-language document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1546-1555.
Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010. Cross-language document summarization based on machine translation quality prediction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 917-926.
Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011.
Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015. Phrase-based compressive cross-language summarization. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 118-127.
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. Ncls: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3045-3055.
A Visualization
We use the PCA (Wold et al., 1987) algorithm to visualize the pre- and post-aligned context representations of our model in Figure 2. The left picture shows the original distribution of two languages, and the right picture shows the distribution after we map Chinese representations to English.
Figure 2 reveals that the representations of the two languages are originally separated but become aligned after our proposed procedure, which demonstrates that our proposed alignment procedure is effective.
We show four cases of Chinese-to-English summarization in Table 7. Since most of the summaries generated by other unsupervised baselines are meaningless (e.g., far away from the theme of the source text, all tokens are "UNK" and so on), we don't show their results here.
(a) Pre-alignment
Figure 2: Visualization of the pre- and post-aligned context representations. The blue dots are English context representations and the red dots are Chinese context representations.
(b) Post-alignment