SlowGuess/NAACL_2022_OCR huggingface dataset:
Peggy Tang†, Kun Hu†, Rui Yan*, Lei Zhang*, Junbin Gao‡, Zhiyong Wang†
†School of Computer Science, The University of Sydney
$^{\star}$ Gaoling School of Artificial Intelligence, Renmin University of China  $^{\diamond}$ International Digital Economy Academy
$^{\ddagger}$ The University of Sydney Business School, The University of Sydney
{peggy.tang, junbin. gao, zhiyong.wang}@sydney.edu.au,kuhu6123@uni.sydney.edu.au,ruiyan@ruc.edu.cn,leizhang@idea.edu.cn
Extractive text summarisation aims to select salient sentences from a document to form a short yet informative summary. While learning-based methods have achieved promising results, they have several limitations, such as dependence on expensive training and lack of interpretability. Therefore, in this paper, we propose a novel non-learning-based method by for the first time formulating text summarisation as an Optimal Transport (OT) problem, namely Optimal Transport Extractive Summariser (OTExtSum). Optimal sentence extraction is conceptualised as obtaining an optimal summary that minimises the transportation cost to a given document regarding their semantic distributions. Such a cost is defined by the Wasserstein distance and used to measure the summary's semantic coverage of the original document. Comprehensive experiments on four challenging and widely used datasets - MultiNews, PubMed, BillSum, and CNN/DM demonstrate that our proposed method outperforms the state-of-the-art non-learning-based methods and several recent learning-based methods in terms of the ROUGE metric.
Text summarisation aims to condense a given document into a short and succinct summary that best covers the semantics of the document with the least redundancy. It helps users quickly browse and understand long documents by focusing on their most important sections (Mani, 2001; Nenkova and McKeown, 2011). A common practice for text summarisation is extractive summarisation which aims to select the salient sentences of a given document to form its summary. Extractive summarisation ensures the production of grammatically and factually
Figure 1: Illustration of Optimal Transport Extractive Summariser (OTExtSum): the formulation of extractive summarisation as an optimal transport (OT) problem. Optimal sentence extraction is conceptualised as obtaining the optimal extraction vector  $\mathbf{m}^*$ , which achieves an OT plan from a document  $\mathbf{D}$  to its optimal summary  $\mathbf{S}^*$  that has the minimum transportation cost. Such a cost is defined as the Wasserstein distance between the document's semantic distribution  $\mathrm{TF}_{\mathbf{D}}$  and the summary's semantic distribution  $\mathrm{TF}_{\mathbf{S}}$  and is used to measure the summary's semantic coverage.
correct summaries, though the output summaries could be inflexible. Since abstractive summaries are highly prone to contain contents that are unfaithful and nonfactual to the original document (Maynez et al., 2020), extractive summaries are more practical for real-world scenarios, especially for the domains requiring formal writing such as legal, science, and journalism documents.
Existing methods (Yao et al., 2017) often first score the importance of individual sentences of a given document and then combine the top-ranked ones to form a summary. However, the sentences with high importance scores may not well represent the document from a global perspective, which results in a sub-optimal summary. Recently, learning-
<sup>1</sup>Our code is publicly available for research purpose in https://github.com/peggypytang/OTExtSum/
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1128 - 1141 July 10-15, 2022 ©2022 Association for Computational Linguistics
based methods, especially those based on supervised and unsupervised deep learning techniques (Narayan et al., 2018; Zheng and Lapata, 2019; Zhang et al., 2019; Narayan et al., 2020; Xu et al., 2020; Zhong et al., 2020; Padmakumar and He, 2021) can significantly improve summarisation performance. However, training deep learning models is computationally expensive, and it can be difficult to apply those models learned from a particular domain to other domains with different distributions. Moreover, deep learning methods generally lack interpretability for the summarisation process.
Motivated by these issues, we propose a novel non-learning based extractive summarisation method, namely Optimal Transport Extractive Summariser (OTExtSum). As illustrated in Figure 1, we formulate extractive summarisation based on the optimal transport (OT) theory (Peyre et al., 2019). A candidate summary can be evaluated by an OT plan regarding the optimal cost to transport between the semantic distributions of the summary and its original document. Then a Wasserstein distance can be obtained with this optimal plan to measure the discrepancy between the two distributions. To this end, it can be expected that a summary of high quality minimizes this Wasserstein distance. Moreover, a common assumption in the formulations of the OT problem is that the source and target distributions are fixed. In OTExtSum problem formulation, we relax this assumption by adding an extraction vector  $\mathbf{m}^*$  to indicate which document sentences would be extracted to form the summary's semantic distribution, thus making the target distribution variable.
The semantic distributions of a given document and its candidate summary can be formulated in line with the frequency of their tokens. Inspired by Word Mover's Distance (Kusner et al., 2015), summarisation can be conceptualized as moving the "semantics" of a given document to its summary, and the ideal summary is obtained at the minimal transportation cost. This ensures the highest semantic coverage of the given document and the least redundancy in the summary without explicitly modelling conventional criteria such as relevance and redundancy. Thus, under the OT plan, the Wasserstein distance indicates the candidate summary's semantic coverage of the given document.
We design two optimisation strategies to approximate the extraction vector  $\mathbf{m}^*$ , namely beam search strategy (Tillmann and Ney, 2003), which iter-
tively evaluates the semantic coverage scores of a set of candidate summaries to obtain the optimal extraction, and binary integer programming strategy, which approximates the optimal extraction given the constraints of the Wasserstein distance and extraction budget. As a non-learning based method, OTExtSum does not require any training and is applicable to different document domains. Furthermore, it provides explainable results in terms of the semantic coverage of the summary.
There have been some studies on OT in NLP, such as document distance (Kusner et al., 2015; Yurochkin et al., 2019), text generation (Chen et al., 2018), text matching (Swanson et al., 2020), and machine translation (Xu et al., 2021). These methods generally focus on deriving similarities between words, sentences, and documents. On the contrary, we for the first time formulate text summarisation as an OT problem that optimally transports the semantic distributions between two texts (e.g., source document and summary candidate).
Overall, the key contributions of this paper are:
- We propose a non-learning based extractive summarisation method - OTExtSum by treating the text summarisation task as an optimal transport problem for the first time.
- We design two optimisation strategies for OTExtSum: beam search strategy and binary integer programming strategy.
- We present an interpretable visualisation of the semantic coverage of a generated summary by visualising the transport plan between summary tokens and document tokens.
- Comprehensive experimental results on four widely used datasets, including CNN/DM, MultiNews, BillSum and PubMed, demonstrate that OTExtSum outperforms the state-of-the-art non-learning based methods.
Generally, text summarisation methods can be categorized as extractive, abstractive, and hybrid ones. While abstractive and hybrid summarisation methods (Lebanoff et al., 2019; Zhang et al., 2020) aim to mimic human beings in summarisation by paraphrasing a given document, extractive summarisation generally produces more factual summaries. In this section, we review existing extractive sum
marisation methods in two categories: non-learning based and learning-based methods.
2.1 Non-learning based Methods
Most of the non-learning based methods conceptualise text summarisation as a sentence ranking task. Each sentence in a given document is scored in terms of various sentence importance criteria, which measure how well the sentence could represent the document. The top-ranked sentences are combined to form a summary. These methods often heavily rely on handcrafted features in regards to linguistic knowledge by focusing on local and/or global contexts.
Local Context based Methods. Local context-based methods rank a sentence based on the features obtained from the sentence itself. Sentence features such as frequency-based and topic-based were studied. Frequency-based features (Edmundson, 1969; Hovy and Lin, 1998) assume that the occurrence of high-frequency terms in a sentence is associated with their importance. Topic-based features (Kupiec et al., 1995; Nobata and Sekine, 2004; Lin and Hovy, 2000) assume that the density of a set of topic terms is highly correlated to the topic of a document.
Global Context based Methods. As local context features could overlook the correlations between sentences and lead to redundant summaries involving similar sentences, global context-based methods rank individual sentences from the perspective of the entire document. Discourse-based methods (Marcu, 1999) construct a document's rhetorical structure and extract the sentences on the longest chain of the semantic structure, i.e. the main topic. Centroid-based methods (Radev et al., 2000) cluster the sentences of a document through similarity measures and rank the sentences based on their distances to the cluster centroids. TextRank (Mihalcea and Tarau, 2004), as a graph-based method, is the state-of-the-art non-learning based method. A graph among document sentences is first formed by connecting sentences using sentence similarity scores, then the sentence connectivity can be used to score the importance of a sentence. Nonetheless, the nature of these sentence based scoring methods could miss summary-level or document-level patterns.
2.2 Learning-based Methods
Instead of utilising handcrafted features, due to the great success of deep learning in many natural lan
guage processing tasks, recent studies on extractive summarisation aim to learn sentence features from the corpus in a data-driven manner.
Supervised Methods. Most of these methods follow the sentence ranking conceptualisation, and an encoder-decoder scheme is generally adopted (Nallapati et al., 2017; Zhang et al., 2019; Narayan et al., 2020; Xu et al., 2020). An encoder formulates document or sentence representations, and a decoder predicts a sequence of sentence importance scores with the supervision of ground-truth sentence labels.
Reinforcement Learning based Methods. Reinforcement learning (RL) can be utilised for extractive summarisation by directly optimising the ROUGE metric, which is used as the training reward. The RL based summarisation task can be treated as a sentence ranking problem similar to the aforementioned methods (Narayan et al., 2018) or as a contextual-bandit problem (Luo et al., 2019).
Unsupervised Methods Various unsupervised methods have also been proposed to leverage pretrained language models to compute sentence similarities and select important sentences. Some methods (Zheng and Lapata, 2019) use these similarities to construct a sentence graph and select sentences based on their centrality. Some methods (Padmakumar and He, 2021) use these to score relevance and redundancy of sentences as selection criteria.
Although these learning-based methods have significantly improved summarisation performance, computationally expensive training costs are inevitable, and it is challenging to generalise the trained models to documents from other domains that have distributions different from the training dataset. In addition, it is difficult to explain the correspondence and the coverage between a summary and a source document using these deep models. Therefore, to address these limitations, we revisit the non-learning based approach and propose a novel summarisation method by exploring the optimal transport theory for the first time.
As shown in Figure 1, OTExtSum utilizes a text OT approximation to obtain the optimal extraction vector  $\mathbf{m}^{*} = [m_{1},\dots,m_{n}]^{T}$ , where  $m_{i}\in \{0,1\}$  denotes whether the  $i$ -th sentence is to be extracted (denoted by 1) or not (denoted by 0). The optimal extraction vector  $\mathbf{m}^{*}$  achieves an OT plan from the semantic distribution of the document to that
of its optimal candidate summary which has the minimum total transportation cost.
The OT approximation consists of four components: 1) a tokeniser & embedding procedure that formulates token level representations and a semantic distribution estimation that computes the frequency of each token within a summary or a document; 2) a transportation cost matrix that measures the cost using one token to represent another based on their Euclidean distances; 3) an OT solver that approximates Wasserstein distance and semantic coverage of the candidate summaries; and 4) an optimisation strategy that obtains the optimal extraction vector by choosing the summary with the minimum Wasserstein distance, and thus with the highest semantic coverage of the source document.
3.1 Optimal Transport
Consider a transportation problem that transports goods from a collection of suppliers  $\mathbf{D} = \{d_i | i = 1, \dots, N\}$  to a collection of customers  $\mathbf{S} = \{s_j | j = 1, \dots, N\}$ , where  $d_i$  and  $s_j$  indicate the supply quantity of the  $i$ -th supplier and the order quantity of the  $j$ -th customer, respectively. Note that, in this study, we consider the number of suppliers to be the same as the customers. By defining  $t_{ij}$  as the quantity transported from the  $i$ -th supplier to the  $j$ -th customer, a transport plan  $\mathbf{T} = \{t_{ij}\} \in \mathbf{R}^{N \times N}$  can be obtained. Given a cost matrix  $\mathbf{C} = \{c_{ij}\} \in \mathbf{R}^{N \times N}$ , where  $c_{ij}$  is the cost to deliver a unit of goods from the  $i$ -th supplier to the  $j$ -th supplier, the cost of the transport plan  $\mathbf{T}$  can be calculated. Particularly, an OT plan  $\mathbf{T}^* = \{t_{i,j}^*\} \in \mathbf{R}^{N \times N}$  in pursuit of minimising the transportation cost can be obtained by solving the following optimisation problem:
$$
\mathbf {T} ^ {*} = \underset {\mathbf {T}} {\operatorname {a r g m i n}} \sum_ {i, j = 1} ^ {N} t _ {i j} c _ {i j},
$$
$$
s. t. \sum_ {j = 1} ^ {N} t _ {i j} = d _ {i}, \quad \forall i \in \{1, \dots , N \}, \tag {1}
$$
$$
\sum_ {i = 1} ^ {N} t _ {i j} = s _ {j}, \quad \forall j \in \{1, \dots , N \},
$$
$$
t _ {i j} \geq 0, \forall i, j \in \{1, \dots , N \},
$$
where the first two constraints indicate the quantity requirements for both suppliers and customers and the last constraint proves a non-negative order quantity. Mathematically, this OT problem is to find a joint distribution  $\mathbf{T}$  with respect to a cost  $\mathbf{C}$ ,
of which the marginal distribution is  $\mathbf{D}$  and  $\mathbf{S}$ . In particular, Wasserstein distance can be defined as:
$$
d _ {W} (\mathbf {D}, \mathbf {S} | \mathbf {C}) = \sum_ {i, j} t _ {i, j} ^ {*} c _ {i, j}. \tag {2}
$$
It can be viewed as the distance between the two probability distributions  $\mathbf{D}$  and  $\mathbf{S}$ , if they are normalized, in line with the cost  $\mathbf{C}$ .
3.2 Semantic Distribution
In the context of text summarisation, denote  $\mathbf{D} = \{\mathbf{s}_1,\dots ,\mathbf{s}_n\}$  to represent a document, where  $\mathbf{s}_i$  denotes the  $i$ -th sentence contained in the document. The sentence  $\mathbf{s}_i$  has a semantic distribution  $\mathrm{TF}_i\in \mathbb{R}^N$  computed by the normalised bag-of-tokens with removal of stop-words:
$$
\mathrm {T F} _ {i} = \left[ T F _ {i 1},..., T F _ {i N} \right] ^ {T},
$$
$$
T F _ {i j} = \frac {d _ {j}}{\sum_ {k = 1} ^ {N} d _ {k}}, \tag {3}
$$
where  $d_{j}$  indicates the count of the  $j$ -th token in a vocabulary of size  $N$ .
A document  $\mathbf{D}$  has a semantic distribution  $\mathrm{TF_D}$ :
$$
\mathrm {T F} _ {\mathbf {D}} = \frac {\mathrm {T F} _ {1} + \dots + \mathrm {T F} _ {n}}{n}. \tag {4}
$$
For a summary  $\mathbf{S} \subset \mathbf{D}$  with its corresponding extraction vector  $\mathbf{m}$ , of which the  $i$ -th element  $m_i$  is an indicator ( $m_i = 1$  if  $\mathbf{s}_i \in \mathbf{S}$ ,  $m_i = 0$  otherwise), it has a semantic distribution TFs:
$$
\mathrm {T F} _ {\mathbf {S}} = \frac {m _ {1} \times \mathrm {T F} _ {1} + \dots + m _ {n} \times \mathrm {T F} _ {n}}{m _ {1} + \dots + m _ {n}}. \tag {5}
$$
In our proposed method, a normalization step is introduced to approximate the semantic distributions of  $\mathbf{D}$  and  $\mathbf{S}$  with term frequency. Note that after the normalization  $\mathrm{TF_D}$  and  $\mathrm{TF_S}$  have an equal total good quantities of 1 and can be completely transported from one to the other. In addition,  $\mathrm{TF_D}$  and  $\mathrm{TF_S}$  satisfy the property of discrete probability distributions, of which the sum should be 1.
3.3 Transport Cost between Tokens
We define the unit transportation cost between two tokens by measuring their semantic similarity. Intuitively, the more semantically dissimilar a pair of tokens are, the higher the "transport cost" of transporting one token to another. Given a pre-trained tokeniser and token embedding model with  $N$  tokens, define  $\mathbf{v}_i$  to represent the feature embedding
of the  $i$ -th token. The transport cost from the  $i$ -th token to the  $j$ -th token  $c_{ij}$  in  $\mathbf{C}$  can be written as:
$$
c _ {i j} = \left\| \mathbf {v} _ {i} - \mathbf {v} _ {j} \right\| _ {2}, \tag {6}
$$
which is based on the Euclidean distance.
3.4 Semantic Coverage of Candidate Summaries
Intuitively, a good summary S is supposed to be close to the document D in terms of their semantic distributions. OTExtSum utilizes the Wasserstein distance to measure the distance between the two associated semantic distributions  $\mathrm{TF_D}$  and  $\mathrm{TF_S}$  with the OT cost. The computation of the Wasserstein distance has time complexity of  $O(p^3\log (p))$  (Altschuler et al., 2017), where  $p$  denotes the number of unique words in the document.
In detail, it can be obtained with Eq. (2) as  $d_W(\mathrm{TF_D},\mathrm{TF_S}|\mathbf{C})$  with a pre-defined cost matrix  $\mathbf{C}$ . Then a semantic coverage score of the summary  $\mathbf{S}$  in respect to the document  $\mathbf{D}$  can be further defined based on the Wasserstein distance:
$$
g (\mathbf {D}, \mathbf {S}) = 1 - d _ {W} \left(\operatorname {T F} _ {\mathbf {D}}, \operatorname {T F} _ {\mathbf {S}} | \mathbf {C}\right). \tag {7}
$$
Therefore, OTExtSum aims to search for an extraction vector  $\mathbf{m}$ , of which the corresponding summary  $\mathbf{S}$  minimises the Wasserstein distance, i.e. maximising the semantic coverage score for the given document  $\mathbf{D}$  by solving OT problems.
3.5 Optimisation Strategy
The remaining problem for OTExtSum is to search for the optimal extraction vector  $\mathbf{m}^*$  which achieves the minimum total transportation cost from the semantic distribution of the document  $\mathrm{TF_D}$  to that of the optimal summary  $\mathrm{TF_S}$ , given a budget  $B$  which is the number of sentences can be extracted to create a summary:
$$
\mathbf {m} ^ {*} = \underset {\mathbf {m}} {\operatorname {a r g m i n}} d _ {W} \left(\mathrm {T F} _ {\mathbf {D}}, \mathrm {T F} _ {\mathbf {S}} | \mathbf {C}\right), \tag {8}
$$
$$
\begin{array}{l} \text {s . t .} m _ {1} + \ldots + m _ {n} \leq B. \end{array}
$$
In search of optimal extraction vector  $\mathbf{m}^*$ , we design two optimisation strategies, namely beam search strategy to achieve better coverage approximation, and binary integer programming strategy to achieve better computational efficiency.
Algorithm 1: Optimisation of OTExtSum with Beam Search Strategy
Input :D the document,  $B$  the budget of the number of extracted sentences,  $K$  the beam width.
Output:  $\mathbf{S}^*$  the optimal extractive summary.
1 Compute the cost matrix  $\mathbf{C}$ , and the document's semantic distribution  $\mathrm{TF}_{\mathbf{D}}$ ;
2 Initialise  $\mathbf{m} = \mathbf{0}$ , i.e. the candidate summary set  $\mathbb{S} = \varnothing$ ;
3 while # of sentences in candidate summary  $\leq B$  ; do // Beam search
4 | for  $k = 1, \dots, |\mathbb{S}|$  do
Generate the successor set  $\mathbb{S}_b^k$  for  $\mathbf{S}^k\in \mathbb{S}$
$$
g \left(T F _ {\mathbf {D}}, T F _ {\mathbf {S} ^ {k}} | \mathbf {C}\right);
$$
6 end
7  $\mathbb{S}\gets \bigcup_{k}\mathbb{S}_{b}^{k};$
8 | for  $k = 1, \dots, |\mathbb{S}|$  do
9 Compute the semantic distribution  $TF_{\mathbf{S}^k}$  of  $\mathbf{S}^k \in \mathbb{S}$ ;
10 Compute the Wasserstein distance  $d_W(TF_{\mathbf{D}},TF_{\mathbf{S}^k}|\mathbf{C})$  and the semantic coverage
11 end
12 Keep the top  $K$  candidate summaries with the highest  $g(\mathrm{TF}_{\mathbf{D}},\mathrm{TF}_{\mathbf{S}^k}|\mathbf{C}))$  and prune the rest in  $\mathbb{S}$
13 end
14  $\mathbf{S}^{*} = \underset {\mathbf{S}^{k}\in \mathbb{S}}{\mathrm{argmax}}g(\mathrm{TF}_{\mathbf{D}},\mathrm{TF}_{\mathbf{S}^{k}}|\mathbf{C}))$
3.5.1 Beam Search Strategy
The Beam Search (BS) strategy with the beam width  $K$  maintains the candidate summary set  $\mathbb{S}$  and searches for the optimal extraction vector  $\mathbf{m}^*$ , thus the optimal extractive summary  $\mathbf{S}^*$ . Algorithm 1 presents the steps to obtain the optimal summary with OTExtSum using the BS strategy. The time complexity is  $O(BKn(p^{3}\log (p)))$ .
Initially, we have  $\mathbf{m} = \mathbf{0}$ , where none of the sentences are extracted. Then, each sentence in the document  $\mathbf{D}$  is selected as a candidate summary, which derives a set of candidate extraction vectors corresponding to a set of candidate summaries, and its semantic coverage score can be evaluated. The top  $K$  candidate summaries in terms of the semantic coverage are kept in the set  $\mathbb{S}$  and the rest are pruned. During the  $b$ -th iteration of the beam search, by appending each possible sentence
<sup>1</sup>We investigated the effect of different distance measurements. As discussed in Section 4.3, cost matrix based on the Euclidean distance and the cosine distance yield similar ROUGE scores.
to an existing candidate summary  $\mathbf{S}^k\in \mathbb{S}$ , where the sentence is not in  $\mathbf{S}^k$ , a set of new candidate summaries  $\mathbb{S}_b^k$  can be obtained. Then  $\mathbb{S}$  is updated by combining all these sets of new candidate summaries in regards to  $k$ :
$$
\mathbb {S} \leftarrow \bigcup_ {k} \mathbb {S} _ {b} ^ {k}. \tag {9}
$$
At the end of beam search, a set of final  $K$  summary candidates within the budget  $B$  is obtained.
Among the  $K$  final candidates from the beam search, OTExtSum obtains the optimal extraction vector and thus the optimal summary by choosing the candidate with the highest semantic coverage of the document  $\mathbf{D}$ .
Algorithm 2: Optimisation of OTExtSum with Binary Integer Programming Strategy
Input :D the document,  $B$  the budget of the number of extracted sentences,  $T$  the number of iterations.