SlowGuess/NAACL_2022_OCR huggingface dataset:
1 Compute the cost matrix C, Compute document's semantic distribution  $\mathrm{TF_D}$
2 Initialise  $\mathbf{w}\in \mathbb{R}^n$
3 for iteration  $t \in [1, \dots, T]$  do
4 Convert w to probability value pr with Sigmoid function;
5 Convert  $\mathbf{pr}$  to  $\mathbf{b} = [b_i,..,b_n]$  by hard sampling from the Gumbel-Softmax distribution;
6 Construct summary's semantic distribution  $TF_{\mathbf{S}}$
7 Compute the Wasserstein distance  $d_W(TF_{\mathbf{D}}, TF_{\mathbf{S}}|\mathbf{C})$
8 Compute the  $L_{1}$  regularisation of  $\mathbf{b}$ ;
9 Compute loss by weighted sum of the Wasserstein distance and the squared difference of  $B$  and  $\mathbf{b}$
10 Compute gradients and update  $\mathbf{w}$
11 end
12 Compute  $\mathbf{m}^*$  by soft sampling Sigmoid(w) from the Gumbel-Softmax distribution;
13 Obtain  $\mathbf{S}^*$  by extracting top-  $B$  sentences with the highest  $m_{i}$  values for  $i = 1,\dots,n$ ;
3.5.2 Binary Integer Programming Strategy
Some prior works showed that integer linear programming is an efficient solution to summarisation problem (McDonald, 2007; Gillick and Favre,
2009). The Binary Integer Programming (BIP) strategy therefore is utilised to search for the optimal extraction vector  $\mathbf{m}^*$  with  $T$  iterations. Based on the extraction vector, we obtain the optimal extractive summary  $\mathbf{S}^*$ . Algorithm 2 presents the optimisation steps to obtain the optimal summary with OTExtSum using the BIP strategy. The time complexity is  $O(T(p^3\log (p)))$ .
As  $\mathbf{m}^*$  is a multi-hot vector and is not differentiable, to make the backpropagation work, we optimise a proxy continuous vector  $\mathbf{w} \in \mathbb{R}^n$ , which is differentiable. Then we hard sample from the Gumbel-Softmax distribution (Maddison et al., 2016) to discretise and compute a multi-hot vector  $\mathbf{b}$  during the iterations, and soft sample to compute  $\mathbf{m}^*$  at the end.
The BIP strategy optimises the following loss function w.r.t.  $\mathbf{w}$ , which is a weighted sum of the Wasserstein distance  $d_W(\mathrm{TF_D},\mathrm{TF_S})$  and the  $L_{1}$  regularisation of  $\mathbf{b}^{2}$ :
$$
d _ {W} \left(\mathrm {T F} _ {\mathbf {D}}, \mathrm {T F} _ {\mathbf {S}} | \mathbf {C}\right) + \alpha | B - \sum_ {i = 1} ^ {n} b _ {i} |, \tag {10}
$$
where  $\alpha$  denotes the weight of  $L_{1}$  regularisation.
4 Experimental Results and Discussions
To validate the effectiveness of the proposed OTExtSum on the documents with various writing styles and its ability to achieve improved summarisation performance, we perform experiments on four widely used challenging datasets collected from different domains.
Dataset	Multi-News	BillSum	PubMed	CNN/DM
Domain	News	Law	Science	News
#Sent./Doc.	80	46	102	33
B	9	7	6	3
Test Set Size	5,622	3,269	6,658	11,490
Table 1: Overview of the datasets. #Sent./Doc. denotes the average number of sentences in the documents,  $B$  denotes the budget of number of extracted sentences.
CNN/DailyMail (CNN/DM) (Hermann et al., 2015) is the standard single-document datasets with manually-written summaries. Multi-News (Fabbri et al., 2019) is a multi-document dataset which summarises multiple news articles. We concatenate the multiple articles as a single input. BillSum
2We choose  $L_{1}$  regularisation for sparsity (Ng, 2004).
Method	Multi-News	BillSum	PubMed	CNN/DM
ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L
LEAD	42.3	14.2	22.4	43.5	25.6	37.8	34.0	8.6	27.1	40.0	17.5	32.9
ORACLE	45.4	20.6	28.1	43.7	25.7	38.0	37.1	15.5	30.4	43.1	23.7	37.5
Non-learning based Methods
LSA (Gong and Liu, 2001)	-	-	-	32.6	15.7	26.3	33.9	9.9	29.7	-	-	-
LexRank (Erkan and Radev, 2004)	38.3	12.7	13.2	-	-	-	39.2	13.9	34.6	-	-	-
TextRank (Mihalcea and Tarau, 2004)	38.4	13.1	13.5	34.4	17.8	27.8	-	-	-	34.1	12.8	22.5
OTExtSum-BIP (GPT2)	40.6	12.1	20.7	36.6	15.6	30.6	35.4	10.8	28.8	34.1	12.6	28.1
OTExtSum-BIP (BERT)	40.6	12.1	20.7	36.6	15.6	30.6	35.4	10.8	28.8	34.1	12.6	28.1
OTExtSum-BS (Word2Vec)	42.3	12.8	21.9	40.1	19.4	34.3	38.2	11.7	30.8	32.3	10.8	25.9
OTExtSum-BS (GPT2)	42.4	14.2	23.2	36.5	19.7	32.0	39.7	13.8	32.3	33.5	12.0	26.7
OTExtSum-BS (BERT)	43.1	13.9	22.5	37.5	19.7	32.6	39.8	13.6	32.3	34.5	12.8	27.8
Unsupervised Deep Learning based Methods
PacSum (Zheng and Lapata, 2019)	43.2	14.3	28.5	-	-	-	-	-	-	40.3	17.6	24.9
PMI (Padmakumar and He, 2021)	40.5	13.2	19.8	-	-	-	37.8	13.4	29.9	36.7	14.5	23.3
Supervised Deep Learning based Method
MatchSum (Zhong et al., 2020)	46.2	16.5	41.9	-	-	-	41.2	14.9	36.8	44.2	20.6	40.4
PEGASUS (Zhang et al., 2020)	47.5	18.7	24.9	57.3	40.2	45.8	45.1	19.6	27.4	44.2	21.5	41.1
Table 2: Comparisons between our OTExtSum and the state-of-the-art methods across different categories. The highest scores are bold, and the second highest ones are underlined.
(Kornilova and Eidelman, 2019) is a dataset for law document summarization, which contains long state bill documents. PubMed (Cohan et al., 2018) is a scientific article dataset that uses the abstract section as the ground-truth summary and the long body section as the document. Table 1 shows an overview of the four datasets. The dataset details are in Appendix A.
While CNN/DM contains shorter documents and summaries, the other three datasets are more challenging because they have more extended documents and summaries, thus have a higher chance to extract sentences containing redundant contents or having limited relevance to the document.
In terms of the pre-trained token embedding model, we compare the static embedding model Word2Vec and the contextual embedding models BERT and GPT2. The details of hyperparameter settings and software used are in Appendix B and C.
Our OTExtSum is compared against LEAD (See et al., 2017), ORACLE (Nallapati et al., 2017), the state-of-the-art non-learning based methods and the recent unsupervised learning-based methods. LEAD and ORACLE are standard baselines in the summarisation task. LEAD baseline extracts the first several sentences of a document as a summary. ORACLE baseline greedily extracts the sentences that maximise the ROUGE-L score based on the reference summary. We compare with the results of strong non-learning-based methods, including LSA (Gong and Liu, 2001), TextRank (Mihalcea and Tauru, 2004), and LexRank (Erkan and Radev, 2004).
Their results on MultiNews, BillSum, PubMed, and CNN/DM are from (Fabbri et al., 2019), (Kornilova and Eidelman, 2019), (Cohan et al., 2018), and (Padmakumar and He, 2021) respectively. For an informative reference, we report recent unsupervised learning-based methods, including PacSum (Zheng and Lapata, 2019), which its released model was trained on the news domain, and PMI (Padmakumar and He, 2021), of which the released models were trained on the news and science domains. Their results on CNN/DM are from (Padmakumar and He, 2021). Their results on MultiNews, BillSum, and PubMed are evaluated on the datasets with the corresponding released models from the same domains. And we include the results of the state-of-the-art supervised learning-based methods with extractive approach MatchSum from (Zhong et al., 2020), and those with abstractive approach PEGASUS from (Zhang et al., 2020).
The commonly used ROUGE metric (Lin, 2004) is also adopted for our quantitative analysis. It evaluates the content consistency between the generated summary and the reference summary. In detail, ROUGE-n scores measure the number of overlapping n-grams between the generated summary and the reference summary. A ROUGE-L score considers the longest common subsequence between the generated summary and the reference summary.
Performance Overview. The experimental results of OTExtSum on the four datasets are listed in Table 2 in terms of ROUGE-1, ROUGE-2 and ROUGE-L F-scores. We observed that the BS strat
egy could generally achieve better optimisation results than the BIP strategy. It is in line with our design understanding that beam search can better reach the global optimum. Whereas, the two strategies achieve similar results in CNN/DM, which could be because CNN/DM has fewer document sentences and lower budget, thus fewer possible solutions and easier to find the optimum.
OTExtSum outperforms the state-of-the-art nonlearning based methods and is comparable to the learning-based methods. Note that the state-of-the-art methods usually optimise at the sentence level, whilst OTExtSum is based on the summary level OT evaluation, by which the quality of the resulting summaries is improved.
We observed that OTExtSum obtains significantly better ROUGE scores than the baseline methods on Multi-News, BillSum and PubMed, while the improvement is not that significant on CNN/DM. When the summary is more extended, such as these three more challenging datasets, the summary sentences are more likely to have redundant content. That is, even summary-level optimisation is more difficult to achieve, our OTExtSum demonstrates higher improvements.
OTExtSum is a non-learning based method, and training is not required. Unlike learning-based methods, it is not limited by the training data domain and can be used for different domains. Experimental results demonstrate generalisation ability of OTExtSum over news, law, and science domains.
Effects of Token Embeddings Models. OTExtSum is dependent on a pre-trained token embedding method. Specifically, the token embedding model affects the cost matrix  $C$  and the tokenisation, thus the frequency vector, of the document. We examine how different token embedding models would affect the performance of OTExtSum by comparing static embedding model Word2Vec, and contextual embedding models BERT and GPT2.
The results on most of the datasets indicate that a more advanced contextual embedding model such as BERT and GPT2 is more effective than a static embedding model Word2Vec. It is in line with the intuitive understanding that a more representative model with adequate training samples often approximates better token embeddings and representation. Despite that, the performance of OTExtSum with Word2Vec is surprisingly competitive.
Effects on Stop-words. We investigate the
impact of stop-words on the performance of OTExtSum. As shown in Table 3 in Appendix E, the effect varies slightly across the datasets, and may not much influence the ROUGE scores. It could be because text summarisation does not generally depend on stop-words. A side benefit of removing the stop-words is reducing the vocabulary size and thus the computation time of OT.
Effects on Distance Measurement. We examine how the distance measurement of the cost matrix would impact the performance of OTExtSum. As shown in Table 3 in Appendix E, cost matrix based on the cosine distance and the Euclidean distance usually yield similar ROUGE scores.
4.4 Interpretable Visualisation
OTExtSum is able to provide an interpretable visualisation of the summarisation procedure. Figure 2 in Appendix D illustrates the transport plan heatmap, which indicates the transportation of semantic contents between tokens in the document and its resulting summary. The higher the intensity, the more the semantic content of a particular document token is covered by a summary token.
Figure 3, 4, 5, and 6 in Appendix F compare the summaries produced by OTExtSum and TextRank. TextRank extracted sentences that are salient on their own yet redundant when combined to form a summary. In comparison, OTExtSum is able to compose summaries that have higher semantic coverage and less redundant content.
In this paper, we have presented OTExtSum, the first optimal transport-based optimisation method for extractive text summarisation. It aims to identify an optimal subset of sentences for producing a summary that achieves high semantic coverage of the document by minimising the Wasserstein distance between the semantic distributions of the document and the summary. It helps obtain a summary from a global perspective and provides an interpretable visualisation of extraction results. In addition, OTExtSum does not require computationally expensive training. The comprehensive experiments demonstrate the effectiveness of OTExtSum, which is generalisable over various document domains. In our future work, we will explore other OT solvers for extractive summarisation.
Jason Altschuler, Jonathan Weed, and Philippe Rigollet. 2017. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In International Conference on Neural Information Processing Systems(NeurIPS).
Liquun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. 2018. Adversarial text generation via feature-mover's distance. In International Conference on Neural Information Processing Systems (NeurIPS).
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
H. P. Edmundson. 1969. New methods in automatic extracting. Journal of the ACM, 16:264-285.
G. Erkan and D. R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Annual Meeting of the Association for Computational Linguistics (ACL).
Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. 2019. Interpolating between optimal transport and mmd using sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics.
Dan Gillick and Benoit Favre. 2009. A scalable global model for summarization. In Workshop on Integer Linear Programming for Natural Language Processing.
Yihong Gong and Xin Liu. 2001. Generic text summarization using relevance measure and latent semantic analysis. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR).
Karl Moritz Hermann, Tomáš Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In International Conference on Neural Information Processing Systems (NeurIPS).
Eduard Hovy and Chin-Yew Lin. 1998. Automated text summarization and the Summarist system. In TIP-STER Text Program Phase III.
Anastassia Kornilova and Vladimir Eidelman. 2019. BillSum: A corpus for automatic summarization of
US legislation. In Workshop on New Frontiers in Summarization (NFiS).
Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR).
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International Conference on Machine Learning (ICML).
Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Annual Meeting of the Association for Computational Linguistics (ACL).
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Chin-Yew Lin and Eduard Hovy. 2000. The automated acquisition of topic signatures for text summarization. In International Conference on Computational Linguistics (COLING).
Ling Luo, Xiang Ao, Yan Song, Feiyang Pan, Min Yang, and Qing He. 2019. Reading like HER: Human reading inspired extractive summarization. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR).
Inderjeet Mani. 2001. Automatic summarization. John Benjamins Publishing.
Daniel Marcu. 1999. Discourse trees are good indicators of importance in text. In Advances in Automatic Text Summarization.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Annual Meeting of the Association for Computational Linguistics (ACL).
Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In *European Conference on IR Research*.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question? In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In *Conference on Empirical Methods in Natural Language Processing* (EMNLP).
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI Conference on Artificial Intelligence.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Shashi Narayan, Joshua Maynez, Jakub Adamek, Daniele Pighin, Blaz Bratanic, and Ryan McDonald. 2020. Stepwise extractive summarization and planning with structured transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends® in Information Retrieval, 5:103-233.
Andrew Y Ng. 2004. Feature selection, 1 1 vs. 1 2 regularization, and rotational invariance. In International Conference on Machine Learning (ICML).
Chikashi Nobata and Satoshi Sekine. 2004. Crl/nyu summarization system. In Document Understanding Conference (DUC).
Vishakh Padmakumar and He He. 2021. Unsupervised extractive summarization using pointwise mutual information. In Conference of the European Chapter of the Association for Computational Linguistics (EACL).
Gabriel Peyre, Marco Cuturi, et al. 2019. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning.
Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. 2000. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In Conference of the North American Chapter of the Association for Computational Linguistics - Applied Natural Language Processing Conference Workshop: Automatic Summarization (NAACL-ANLP-AutoSum).
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. Annual Meeting of the Association for Computational Linguistics (ACL).
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning (ICML).
Kyle Swanson, Lili Yu, and Tao Lei. 2020. Rationalizing text matching: Learning sparse alignments via optimal transport. In Annual Meeting of the Association for Computational Linguistics (ACL).
Christoph Tillmann and Hermann Ney. 2003. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Annual Meeting of the Association for Computational Linguistics (ACL).
Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021. Vocabulary learning via optimal transport for neural machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL).
Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2017. Recent advances in document summarization. Knowledge and Information Systems, 53(2):297-336.
Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin Solomon. 2019. Hierarchical optimal transport for document representation. In International Conference on Neural Information Processing Systems (NeurIPS).
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning (ICML).
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Annual Meeting of the Association for Computational Linguistics (ACL).
Hao Zheng and Mirella Lapata. 2019. Sentence centrality revisited for unsupervised summarization. In Annual Meeting of the Association for Computational Linguistics (ACL).
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Annual Meeting of the Association for Computational Linguistics (ACL).
We followed (Zhong et al., 2020) to set  $B$  for CNN/DM, PubMed and Multi-News, and used the average number of sentences in the summaries to set  $B$  for BillSum since this is a common practice in the literatures (Narayan et al., 2018). These datasets were obtained from a source, namely HuggingFace Datasets<sup>3</sup>.
Since OTExtSum does not require training, for a fair comparison, all experimental results are reported on the test splits of the four datasets only.
For the hyperparameter settings of the BIP strategy, the number of iteration  $T$  was set to 200,  $\alpha$  was set to 1, and it used the SGD optimiser (Sutskever et al., 2013) with learning rate 0.1. For the BS strategy, the beam width  $K$  was set to  $5^4$ .
C Software and Hardware Used
We obtained the pre-trained Word2vec (Google News 300 dimension) from GENSIM  $^{5}$ , and the contextual embedding models BERT (base version) and GPT2 from HuggingFace  $^{6}$ . To compute the Wasserstein distances, we adopted GENSIM, the POT  $^{7}$  and GeomLoss (Feydy et al., 2019) libraries. List of stop-words was from NLTK library  $^{8}$ . Our experiments were run on a GeForce GTX 1080 GPU card. We obtain our ROUGE scores by using the pyrouge package  $^{9}$ .
D Example of Interpretable Visualisation
Figure 2: Interpretable visualisation of the OT plan from a source document to a resulting summary on the CNN/DM dataset. The higher the intensity, the more the semantic content of a particular document token is covered by a summary token. Purple line highlights the transportation from the document to the summary of semantic content of token "month", which appears in both the document and the summary. Red line highlights how the semantic content of token "sponsor", which appears in the document only but not the summary, are transported to token "tour" and "extension", which are semantically closer and have lower transport cost, and thus achieve a minimum transportation cost in the OT plan.
<sup>3</sup>https://huggingface.co/docs/datasets/
4We chose the beam width in line with a common practice in the literature (Meister et al., 2020)
<sup>5</sup>https://radimrehurek.com/gensim/index.html
$^{6}$ https://huggingface.co
<sup>7</sup>https://pythonot.github.io
$^{8}$ https://www.nltk.org
$^{9}$ https://pypi.org/project/pyrouge/
Method	Multi-News	BillSum	PubMed	CNN/DM
ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-1	ROUGE-2	ROUGE-L
Euc. \w s.w.	43.1	13.9	22.5	37.5	19.7	32.6	39.8	13.6	32.2	34.5	12.8	27.8
Cos. \w s.w.	43.1	13.9	22.5	39.0	19.5	33.6	39.8	13.6	32.3	34.4	12.4	27.7
Euc. \w s.w.	43.4	14.4	23.4	36.9	19.6	32.2	40.6	13.8	33.0	34.1	12.1	27.1
Cos. \w s.w.	43.9	14.2	23.1	38.1	19.6	33.0	40.6	13.6	32.9	34.1	12.1	27.1
Table 3: Ablation studies of OTExtSum based on the BS optimisation strategy and pre-trained BERT tokeniser. Euc. denotes the Euclidean distance and Cos. denotes the cosine distance. s.w. denotes stop-words.
F Generation Samples
Below are the generation samples of OTExtSum and TextRank. In general, OTExtSum based summary contains less redundant content and provides higher semantic coverage with the same number of extracted sentences.