SlowGuess/NAACL_2022_OCR huggingface dataset:
SUS (Zhang et al., 2020) models. It needs to be acknowledged that PEGASUS performances are much better since it is pre-trained specifically for summarization tasks. We think it is not fair to compare our model with it. Different from the unilateral improvement in (Zhang and Tetreault, 2019), our method not only obtains about  $1\%$  (abs.) improvements on ROUGE-1/L scores but also significantly surpasses the performance on the METEOR score with around  $2\%$  (abs.). Previous works generate titles that only focus on limited sentences or static sentences' embeddings more often produce trivial words, instead, our method generates each target word of the title meticulously via TSE to re-locate the related words and update the sentences' embeddings. Thus, with the increase of ROUGE score, our method also obtains improvement of METEOR score. Identically, when the ground-truth is the human-annotated subjects, our method outperforms competitive baselines remarkably, the results are summarized in Table 3. The improvement on ROUGE-1/L is over  $3\%$  (abs.) and about  $2\%$  (abs.) on METEOR score. We also achieved the best Rouge-2 score in this setting with about  $2\%$  improvements. Besides, we also replace the GRU layers with Transformer layers to test the usefulness of TSE with different backbone and
the performance is a little better. It demonstrates that the two novel mechanisms, TSE and AC are effective for different kinds of neural structures.
methods	R-1	R-2	R-L	METEOR
PG-net	50.25	43.36	49.84	27.46
PALM	58.30	49.49	57.69	32.45
SimCLS	59.29	50.12	58.01	32.86
TSE-VC	58.02	49.30	57.15	31.51
TSE-AC	62.17	53.09	61.21	36.24
TSE-VC-Trans	59.11	50.06	58.21	32.44
TSE-AC-Trans	63.09	54.13	61.97	36.17
It is worthy to indicate that our small model achieves much better results compared with the classical pre-trained model T5 (Raffel et al., 2020) and the SimCLS which use the results generated by T5 as candidates to further training the model with contrastive learning (Liu and Liu, 2021). We do not compared to the customized pre-trained model for summarization in (Zhang et al., 2020) since it is unfair. The improvements are higher when using human annotation as references. Through in-depth analysis of the generated results, we found that the pre-trained model suffers from aligning with longer inputs and the short titles with words scattered among the whole content. This information aggregation process has a large gap compared to the pre-training tasks. And the pre-trained T5 without coverage mechanism is poor at tackling the repetition problem. In addition, the pre-trained model with much more parameters and general knowledge learns more easily what it observed, so the performance in Table 3 degrades heavily when the ground-truth is different from the learning target.
Moreover, we also implement experiments over Gigaword in Chinese with longer source articles and titles, the results demonstrate similar conclusions. Specifically, we adopt PALM (Bi et al., 2020b) as a baseline on this dataset. PALM is a pre-trained generation model and we obtained the pre-trained model with a large Chinese corpus from the author, it performs better on many natural language processing tasks in Chinese than T5. Our model achieves over  $10\%$  (abs.) higher score on Gigaword with PG-net as baseline and about  $5\%$  (abs.) improvements over PALM. It indicates the versatility of our approach that is capable for different scenarios of title generation.
Human Evaluation We only conduct human evaluation on Chinese Gigaword dataset since the
Table 4: The performance comparison on Gigaword. TSE-VC means using vanilla coverage with TSE.
method	Fluency	Relevance	Usability
SimCLS	3.07	3.38	65.32%
w/o Cov	2.78	3.16	50.12%
D-VC	3.15	3.56	68.31%
D-AC	3.54	3.72	80.27%
Table 5: Human evaluation on fluency, relevance and usability with different coverage settings.
three human annotators are with Chinese as native language. 200 news texts are sampled from the corpus randomly. The comparison among SimCLS and different settings of the coverage mechanism with our proposed approach are shown in Table 5, from which we can conclude that the general performance of our method with TSE and coverage mechanism can provide titles with high quality. Though our model without coverage mechanism performs worse than SimCLS, two kinds of coverage mechanisms can improve the performance effectively. Especially the AC can raise the titles' quality to higher level and is acceptable to be used in practical scenarios with over  $80\%$  of the generated titles being usable for real news.
Analysis of Computation Complexity We further analyze the computation complexity of our model and compared it to traditional seq2seq models. Assuming that the input content has  $M$  sentences, and each sentence has  $N$  words averagely, the target title contains  $T$  tokens. The time complexity of vanilla seq2seq models like PG-net is  $O(M*N) + O(T)$  since they have to encode the input content sequentially and generate the title word by word. For our TSE-AC model, the encoder with hierarchical architecture can parallel encode words in each sentence, and we need to update the embedding of each sentence for every target word, so the time complexity is  $O(N) + O(T*M)$ . As we mentioned before, we focus on short title generation tasks in this paper which means  $T < N$ , and it can be derived that  $O(M*N) + O(T) >= O(M*(T+1)) + O(T) = O(T*M) + O(M) + O(T) > O(M) + O(T*M)$ . As a result, if we split the input content into  $M$  sentences with  $N$  words in each sentence and make sure  $M < N$ , our model's computation complexity is less than traditional seq2seq models. When change the backbone model from GRU to Transformers or pure hierarchical encoder, our model is a little complex since transformer layers based encoder is parallel naturally, but the cost of re-computation of sentences' embeddings are accept
Ref.	Condition	DEV	TEST
R-1	R-2	R-L	METEOR	R-1	R-2	R-L	METEOR
Ori	Our Best	26.35	12.07	25.99	11.08	25.18	11.86	24.59	11.25
w/o TSE	24.25	9.47	23.64	10.03	22.76	8.68	22.30	10.05
w/o WG	25.34	10.26	24.92	10.95	24.87	8.82	23.99	11.21
w/o SG	24.62	8.79	24.18	9.42	23.17	7.93	22.80	13.14
w/o POS	23.30	8.54	22.82	9.48	23.02	8.76	22.69	9.72
w/o Cov	25.02	9.64	24.39	10.35	23.38	8.68	22.83	9.77
w/ VC	25.62	10.36	25.18	11.03	24.11	10.17	23.55	10.45
HA	Our Best	30.01	13.29	29.13	16.55	30.20	13.32	29.07	16.17
w/o TSE	23.63	9.05	22.82	12.78	24.69	9.29	23.90	13.24
w/o WG	25.20	10.31	24.60	13.42	26.42	9.76	25.41	14.21
w/o SG	25.53	9.73	24.75	9.09	26.62	10.05	25.96	13.49
w/o POS	24.74	9.33	23.99	13.13	25.55	9.70	24.90	13.34
w/o Cov	26.71	9.91	25.70	13.24	26.90	9.82	25.98	13.93
w/ VC	27.40	11.02	26.51	14.23	27.28	11.12	26.50	14.25
Table 6: The ablation study of our method on AESLC. Ori and HA indicate original references and human annotations. w/o TSE represents the model without TSE which means the sentences' embeddings are frozen for all decoder steps, w/o WG and w/o SG mean that the inner relations among words or sentences are not used. POS means part of speech information and Cov indicates coverage mechanism. w/ VC means using vanilla coverage.
Coverage	PG-net	TSE
R-1	R-2	R-L	METEOR	R-1	R-2	R-L	METEOR
w/o Cov	21.65	6.71	19.47	10.35	26.90	9.82	25.98	13.93
w/ VC	24.09	7.11	22.52	13.96	27.28	11.12	26.50	14.25
w/ AC	26.50	10.08	25.08	15.46	29.44	12.41	28.20	15.53
Table 7: The performance comparison among different settings of the coverage mechanism for title generation by PG-net and our TSE-AC on AESLC test set with annotated subjects as reference. w/o Cov means coverage mechanism not used. The results of PG-net are reproduced by ourselves using publicly available code.
able and meaningful since the titles are always brief.
Since there are several components in our proposed model, including TSE mechanism, GCN layer for inner relations encoding, POS information, and coverage mechanism, we conduct comprehensive experiments to discover whether the specific part gives positive influence to the model's performance or negative. The results are shown in Table 6.
With the original email subjects as references, the performances declined heavily on ROUGE score when the TSE is discarded and the value of METEOR reduced most without the POS information as model's inputs. The performances on human annotations have the same phenomenon. Overall, the absence of these two components has the greatest negative impact on the performance of our model. The results demonstrate that the TSE mechanism is essential for title generation since it can re-locate the key information and update the sentences' embeddings at each decoding timestep. The POS information is also useful because words with different POS play a different role in the whole texts, which facilitates the TSE to
focus on the corresponding critical words. For the effectiveness of GCN for inner relations encoding, the results indicate that GCN encoder layers have positive influence on ROUGE obviously but inconsistent impact on METEOR metric while the value is much higher when sentence level GCN is abandoned with original subject as a reference. This may result from the inconsistency of the original subject and the annotations. The coverage mechanism is also useful especially our novel AC. The ablation study demonstrates that the TSE and AC is effective for brief title generation tasks.
To compare the effectiveness of our AC mechanism with the vanilla one directly, we conduct experiments with different coverage settings on our proposed model and PG-net. The results shown in Table 4,6 and 7 indicate that our novel AC can achieve better performances not only with TSE but also other title generation methods or backbones. And it means that AC is more general and can make up for the shortcoming of VC. From the second term of Equation 15, the coverage loss will make the model ignore the words which have been pay more attention before as much as possible. If the word with high attention in previous timestep and has not been generated actually, the models with
VC mechanism are easier to ignore it. And our AC mechanism can well avoid this wrong punishment by only considering the actually generated words. The experiments also indicate that the combination of TSE and AC results in higher improvements compared to changing the backbone from GRU to Transformer.
In Table 8 (see Appendix A), some examples with subjects and corresponding email contents are given to demonstrate the ability of our novel model. From the first three samples, we find that our model can generate more accurate subjects compared to the baseline method proposed in (See et al., 2017), of which the subjects need to be concluded based on the entire source content, not just the first or last few sentences. And the related words are different within the same sentences for each target word according to their location and inner relations. The traditional seq2seq model ignores the inner relationship among words and sentences but only encodes their sequential information. The work in (Zhang and Tetreault, 2019) first extracts some key sentences and generated subjects merely base on those sentences will lose many useful clues to generate more valuable keywords. Moreover, although our model is trained with original subjects, it can generate more reasonable targets compared to the original ones in most cases.
The last two examples indicate that our model can obtain more suitable subjects even if the key information is contained in the first few sentences. As the title is a highly condensed summary of the article, each word in the title needs to be confirmed based on different parts of the original content. Our model with TSE and AC mechanisms can finely relocate the valuable words scattered in the original text, and the selected words are used to update the sentences' embeddings for each decoding timestep. The context vector is further modified so as to better summarize the input article.
All the examples show that our novel AC mechanism can obtain subjects with more information while the VC lost some key information more or less. Our model TSE-AC can generate titles with more complete and accurate information compared to PG-net, and the repetition problem is mitigated while preserving more critical words compared with T5 and vanilla coverage.
In this paper, we proposed a novel title generation framework with Timestep aware Sentence Embedding, which re-locates critical words dynamically for each target word from the source content based on the decoding states to update the corresponding sentence's embedding. Moreover, we present Acme Coverage that can accurately penalize the probability of the word which has been generated actually. The experiments demonstrate that our approach achieves the state of the art performance on different kinds of title generation scenarios.
This work is supported by the Postdoctoral Program of Zhejiang Province. Thanks for the guidance and help from my advisors Prof. Wei Chu (weichu.cw@antgroup.com) and Prof. Huajun Chen (huajunsir@zju.edu.cn). Thanks to my supervisor Mr. Taifen Wang (taifeng.wang@antgroup.com) of the invaluable suggestions for this work.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020a. PALM: pre-training an autoencoding&autoregressive language model for context-conditioned generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8681-8691. Association for Computational Linguistics.
Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020b. PALM: pre-training an autoencoding&autoregressive language model for context-conditioned generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8681-8691. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734. ACL.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:1804.05685.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1409-1418. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrialstrength Natural Language Processing in Python.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5082-5093. Association for Computational Linguistics.
Rong Jin and Alexander G. Hauptmann. 2001. Automatic title generation for spoken broadcast news. In Proceedings of the First International Conference on Human Language Technology Research, HLT 2001, San Diego, California, USA, March 18-21, 2001. Morgan Kaufmann.
Rong Jin and Alexander G. Hauptmann. 2002. A new probabilistic model for title generation. In 19th International Conference on Computational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24 - September 1, 2002.
Paul E. Kennedy and Alexander G. Hauptmann. 2000. Automatic title generation for EM. In Proceedings of the Fifth ACM Conference on Digital Libraries, June 2-7, 2000, San Antonio, TX, USA, pages 230-231. ACM.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Chenliang Li, Weiran Xu, Si Li, and Sheng Gao. 2018. Guiding generation for abstractive text summarization based on key information guide network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 55-60. Association for Computational Linguistics.
Haoran Li, Junnan Zhu, Jiajun Zhang, Chengqing Zong, and Xiaodong He. 2020a. Keywords-guided abstractive sentence summarization. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8196-8203. AAAI Press.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020b. Leveraging graph to improve abstractive multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6232-6243. Association for Computational Linguistics.
Chin-Yew Lin and FJ Och. 2004. Looking for a few good metrics: Rouge and its evaluation. In Ntcir Workshop.
Wenfeng Liu, Peiyu Liu, Yuzhen Yang, Jing Yi, and Zhenfang Zhu. 2019. A< word, part of speech> embedding model for text classification. Expert Systems, 36(6):e12460.
Yixin Liu and Pengfei Liu. 2021. Simcls: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021,
(Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 1065-1072. Association for Computational Linguistics.
Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. arXiv preprint arXiv:1605.03148.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
Robert Parker, D Graff, K Chen, J Kong, and K Maeda. 2011. Chinese gigaword fifth edition (ldc2011t13). Linguistic Data Consortium.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018a. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018b. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685.
Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, and Abe Ittycheriah. 2016. Temporal attention model for neural machine translation. arXiv preprint arXiv:1608.02927.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936. PMLR.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.