SlowGuess/ACL_2020_OCR huggingface dataset:
Then we introduce our generation model. Firstly we change the pointer-generator network to a multi-source pointer-generator network. The multi-source pointer-generator network has two encoders and one decoder. The two encoders encode the
citing paper's context and cited paper's abstract separately. The input context of citing paper is seen as a sequence of words  $\{cw_1, cw_2, \ldots, cw_n\}$  and the input cited paper's abstract is seen as a sequence of words  $\{aw_1, aw_2, \ldots, aw_m\}$ . We use the same notation to represent both a word and its embedding vector. The context is encoded by corresponding encoder to a sequence of encoder hidden states  $\{ch_i\}$  and the cited paper's abstract is encoded to a sequence of encoder hidden states  $\{ah_j\}$ . At each decoding step  $t$ , we calculate attention vectors  $\{ac_i^t\}$ ,  $\{as_i^t\}$  and corresponding context vectors  $c_t^1$ ,  $c_t^2$  separately as described in equations (1), (2) and (3). To make the model copy words from both two encoders, we change equation (5) to:
$$
\begin{array}{l} [ p _ {g e n}, p _ {c o p y 1}, p _ {c o p y 2} ] = s o f t m a x (W _ {c 1} ^ {T} c _ {t} ^ {1} + W _ {c 2} ^ {T} c _ {t} ^ {2} \\ + W _ {s} ^ {T} s _ {t} + W _ {x} ^ {T} x _ {t} + b _ {p t r}) \tag {7} \\ \end{array}
$$
where  $p_{gen}$  is the probability of generating words,  $p_{copy1}$  is the probability of copying words from citing paper's context and  $p_{copy2}$  is the probability of copying words from cited paper's abstract. And equation (6) needs to be changed to:
$$
\begin{array}{l} P (w) = p _ {g e n} P _ {v} (w) + p _ {c o p y 1} \Sigma_ {i: c w _ {i} = w} a c _ {i} ^ {t} \tag {8} \\ + p _ {c o p y 2} \Sigma_ {i: a w _ {i} = w} a s _ {i} ^ {t} \\ \end{array}
$$
Then we add the cross attention mechanism to the multi-source pointer-generator network. By making citing paper's context and cited paper's abstract attend to each other, we capture the relationships between them. First, we calculate a match matrix  $M$  between the sequence of context's states  $\{ch_i\}$  and the sequence of cited paper's abstract's states  $\{ah_j\}$ . The element of the match matrix
6186
$M_{i,j}$  is:
$$
M _ {i, j} = c h _ {i} \cdot a h _ {j} \tag {9}
$$
Then we apply softmax function on the row vectors of the matrix and get an attention matrix  $A^{row}$ . The row vector  $A_{i}^{row}$  of the attention matrix is:
$$
A _ {i} ^ {\text {r o w}} = \operatorname {s o t m a x} \left(\left[ M _ {i, 1}, M _ {i, 2}, \dots , M _ {i, m} \right]\right) \tag {10}
$$
The vector  $A_{i}^{row}$  represents the attention of word  $cw_{i}$  to the sequence of words  $\{aw_{1}, aw_{2}, \dots, aw_{m}\}$ . We also apply softmax function on the column vectors of the matrix and get another attention matrix  $A_{i}^{column}$ . The column vector of the attention matrix  $A_{i}^{column}$  represents the attention of word  $aw_{i}$  to the sequence of words  $\{cw_{1}, cw_{2}, \dots, cw_{n}\}$ . With the two attention matrices, we calculate two special sequences of vectors. The first sequence of vectors  $\{r_{1}, r_{2}, \dots, r_{n}\}$  is calculated as:
$$
r _ {i} = \sum_ {j = 1} ^ {m} A _ {i, j} ^ {r o w} * a w _ {j} \tag {11}
$$
The second sequence  $\{q_1, q_2, \dots, q_m\}$  is calculated as:
$$
q _ {j} = \Sigma_ {i = 1} ^ {n} A _ {i, j} ^ {\text {c o l u m n}} * c w _ {i} \tag {12}
$$
The vector  $r_i$  represent what the word  $cw_i$  thinks about the sequence of words  $\{aw_1, aw_2, \ldots, aw_m\}$ , while the vector  $q_j$  represents what the word  $aw_j$  thinks about the sequence of words  $\{cw_1, cw_2, \ldots, cw_n\}$ . We believe that the two sequences of vectors can model the relationship between the input citing paper's context and cited paper's abstract, so we call them relationship vectors. With these two sequences of relationship vectors, we calculate two new context vectors  $c_t^3$  and  $c_t^4$  separately at each decoding step  $t$ , by replacing the encoder hidden state  $h_i$  with the relationship vector  $r_i$  or  $q_j$  in equations (1) (2) and (3). Finally, we calculate the vocabulary distribution with all four context vectors. We just need to change equation (4) to:
$$
P _ {v} = \operatorname {s o f t m a x} \left(V _ {2} \left(V _ {1} \left[ s _ {t}, c _ {t} ^ {1}, c _ {t} ^ {2}, c _ {t} ^ {3}, c _ {t} ^ {4} \right] + b\right) + b ^ {\prime}\right) \tag {13}
$$
The final probability distribution over the extended vocabulary is still calculated as equation (8).
7.1 Experimental Setup
The baseline models include:
RandomSen: It randomly selects a sentence from the abstract of paper B.
Models	ROUGE-1	ROUGE-2	ROUGE-L
RandomSen	15.18	1.37	11.35
MaxSimSen	15.65	1.64	11.45
EXT-ORACLE	22.60	4.21	16.83
COPY-CIT	20.54	3.25	14.79
PTGEN	24.60	6.16	19.19
PTGEN-Cross	26.28	7.50	20.49
Table 5: Comparison results on Explicit dataset
ROUGE-1	ROUGE-2	ROUGE-L
RandomSen	15.65	1.36	10.98
MaxSimSen	17.70	1.80	12.20
Ext-ORACLE	22.59	3.88	15.97
COPY-CIT	19.32	2.71	13.02
PTGEN	22.83	5.17	18.37
PTGEN-Cross	24.54	5.44	19.21
Table 6: Comparison results on HP dataset
ROUGE-1	ROUGE-2	ROUGE-L
RandomSen	15.65	1.36	10.98
MaxSimSen	17.70	1.80	12.20
Ext-ORACLE	22.59	3.88	15.97
COPY-CIT	20.08	2.67	13.01
PTGEN	23.26	5.12	18.83
PTGEN-Cross	24.22	6.04	19.38
Table 7: Comparison results on HR dataset
MaxSimSen: It selects a sentence from the abstract of paper B, which has the largest similarity with the context of A.
EXT-ORACLE: It can be viewed as an upper bound for extractive models. It creates an oracle citation text by selecting the best possible sentence from the abstract of paper B that gives the highest ROUGE with respect to the gold text.
COPY-CIT: It randomly copies one citation text from the papers in the training dataset which also cite the paper B.
PTGEN: It is a pointer-generator network which allows both copying words via pointing and generating words from a fixed vocabulary. When using this model, we concatenate the citing paper's context and the cited paper's abstract as the input sequence.
Our proposed model is called PTGEN-Cross. Both our model and the PTGEN has 256-dimensional hidden states and 128-dimensional word embeddings. The vocabulary size is set to  $50\mathrm{k}$ . At test time the citation texts are produced using beam search with beam size 4.
7.2.1 Automatic Evaluation
We evaluate our models with ROUGE (Lin, 2004), reporting the  $F_{1}$  scores for ROUGE-1, ROUGE-2
6187
Context	...They include entity approaches for local coherence which track the repetition and syntactic realization of entities in adjacent sentences [otherrefer] and content approaches for global coherence which view texts as a sequence of topics, each characterized by a particular distribution of lexical items [otherrefer]. [cit] Early theories [otherrefer] posited that there are three factors which collectively contribute to coherence: intentional structure (purpose of discourse), attentional structure (what items are discussed) and the organization of discourse segments...
Abstract	We combine lexical, syntactic, and discourse features to produce a highly predictive model of human readers judgments of text readability ... Our experiments indicate that discourse relations are the one class of features that exhibits robustness across these two tasks.
Gold	Other work has shown that co-occurrence of words [otherrefer] and discourse relations [refer] also predict coherence.
PTGEN	Recently, approaches [refer] have been suggested to predict the quality of discourse relations.
PTGEN-Cross	Other work has shown that co-occurrence of sentences [otherrefer]; [refer] and discourse relations [otherrefer] discourse can be used to predict the coherence of sentences in texts.
Table 8: Example output citation texts
Gold	PTGEN	PTGEN-Cross
Readability	4.89	3.77	3.79
Content	4.42	2.76	2.77
Coherence	4.41	2.70	2.85
Overall	4.55	2.84	2.91
Table 9: Human evaluation results
and ROUGE-L. The test results on three datasets are shown in Tables 5, 6 and 7, respectively.
On all three datasets, extractive models perform poorly. Our baseline generation model PTGEN outperforms EXT-ORACLE which can be seen as a 'perfect' extractive system. This is completely different from how these models preform on other summarization tasks like news document summarization. We believe it shows the particularity of this task. It not only requires the model to capture the important content of the cited paper, but also requires the model to capture the attitude of the citing paper to the cited paper. The model not only needs to generate fluent and informative text, but also needs to ensure the contextual coherence.
Our proposed model PTGEN-Cross obviously outperforms the baseline model PTGEN. This proves the effectiveness of the cross attention mechanism. We think the cross attention mechanism helps the model capture the relationship between the citing paper's context and the cited paper's abstract. The results on explicit citation text generation dataset are all higher than the results on the other two datasets, which means the task of explicit citation text generation is easier than the task of full citation text generation. We think it is because the context of explicit citation sometimes contains some implicit citation sentences and these sentences can be very helpful to the generation of explicit citation text. Another possible reason is that the quality of the training dataset for explica
it citation generation is higher than the other two training datasets. Because the test data of the two full citation text generation datasets is the same, we can compare the results of our model training on the two datasets. The model trained on the highrecall dataset performs slightly better. This tells us the coverage ability of the implicit citation extraction model is more important when constructing training dataset for citation generation.
7.2.2 Human Evaluation
We randomly sample 50 instances from the highrecall test set and perform human evaluation on them. Three graduate students are employed to rate the citation text produced by each method in four aspects: readability (whether the citation text is fluent), content (whether the citation text is relevant to the cited paper's abstract), coherence (whether the citation text is coherent with the citing paper's context) and overall quality. The rating score ranges from 1 to 5, and 1 means very bad and 5 means very good. Note that every text is scored by three judges and we take the average of three scores. The results are shown in Table 9.
As is shown in the table, our model outperforms the baseline model, especially with respect to the coherence and overall aspects. This further demonstrates the efficacy of our proposed model. We show an example of generation in Table 8. Note that all reference signs to the cited paper are masked as '[refer]' and all reference signs to other papers are masked as '[otherrefer]'. The '[cit]' in bold in context indicates the position the citation text should be. We can see that the citation text generated by our model is more contextual coherent because it can capture the relationship between context and the cited paper's abstract better.
6188
In this paper we investigate the challenging task of automatic generation of citation texts in scholarly papers. We annotate a dataset and train an implicit citation extraction model to automatically enlarge the training data. we then propose the multi-source pointer-generation network with cross attention mechanism to deal with this task. Empirical evaluation results on three datasets verify the efficacy of our proposed method. In future work, we will consider introducing more information like the citation texts to the cited paper in other papers to help the generation.
This work was supported by National Natural Science Foundation of China (61772036), Tencent AI Lab Rhino-Bird Focused Research Program (No.JR201953) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Awais Athar and Simone Teufel. 2012. Detection of implicit citations for sentiment detection. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, pages 18-26, Jeju Island, Korea. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Phyllis B Baxendale. 1958. Machine-made index for technical literaturean experiment. IBM Journal of research and development, 2(4):354-361.
Jingqiang Chen and Hai Zhuge. 2014. Summarization of scientific documents by detecting common facts in citations. Future Generation Computer Systems, 32:246-252.
Jingqiang Chen and Hai Zhuge. 2019. Automatic generation of related work through summarizing citations. *Concurrency and Computation: Practice and Experience*, 31(3):e4261.
Arman Cohan and Nazli Goharian. 2018. Scientific document summarization via citation contextualization and scientific discourse. International Journal on Digital Libraries, 19(2-3):287-303.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Harold P Edmundson. 1969. New methods in automatic extracting. Journal of the ACM (JACM), 16(2):264-285.
Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 427-435. Association for Computational Linguistics.
Yue Hu and Xiaojun Wan. 2014. Automatic generation of related work sections in scientific papers: an optimization approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1624-1633.
Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2016. Overview of the cl-scisumm 2016 shared task. In Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL), pages 93-102.
Chaker Jebari, Manuel Jesús Cobo, and Enrique Herrera-Viedma. 2018. A new approach for implicit citation extraction. In International Conference on Intelligent Data Engineering and Automated Learning, pages 121-129. Springer.
Dain Kaplan, Ryu Iida, and Takenobu Tokunaga. 2009. Automatic extraction of citation contexts for research paper summarization: A coreference-chain based approach. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pages 88-95.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Wei Lu, Hwee Tou Ng, Wee Sun Lee, and Luke S Zettlemoyer. 2008. A generative model for parsing natural language to meaning representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 783-792. Association for Computational Linguistics.
Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159-165.
Qiao zhu Mei and Cheng Xiang Zhai. 2008. Generating impact-based summaries for scientific literature. In Proceedings of ACL-08: HLT, pages 816-824.
Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009. Using citations to generate surveys of scientific paradigms. In Proceedings of human language technologies: The 2009 annual conference of the North American
6189
chapter of the association for computational linguistics, pages 584-592. Association for Computational Linguistics.
Vahed Qazvinian and Dragomir R Radev. 2008. Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 689-696. Association for Computational Linguistics.
Vahed Qazvinian and Dragomir R Radev. 2010. Identifying non-explicit citing sentences for citation-based summarization. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 555-564. Association for Computational Linguistics.
Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. 2013. The acl anthology network corpus. Language Resources and Evaluation, 47(4):919-944.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Henry Small. 2011. Interpreting maps of science using citation context sentiments: a preliminary investigation. Scientometrics, 87(2):373-388.
Parikshit Sondhi and ChengXiang Zhai. 2014. A constrained hidden markov model approach for nonexplicit citation context extraction. In Proceedings of the 2014 SIAM International Conference on Data Mining, pages 361-369. SIAM.
Yuk Wah Wong and Raymond J Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 439-446. Association for Computational Linguistics.
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7386-7393.
Ozge Yeloglu, Evangelos Milios, and Nur Zincir-Heywood. 2011. Multi-document summarization of scientific corpora. In Proceedings of the 2011 ACM Symposium on Applied Computing, pages 252-258. ACM.