Text Summarization with Oracle Expectation
Authors: Yumo Xu
Mirella Lapata
Venue: ICLR 2023 poster (2023)
Keywords: Text Summarization
NLP

Abstract: Extractive summarization produces summaries by identifying and concatenating the most important sentences in a document. Since most summarization datasets do not come with gold labels indicating whether document sentences are summary-worthy, different labeling algorithms have been proposed to extrapolate oracle extracts for model training. In this work, we identify two flaws with the widely used greedy labeling approach: it delivers suboptimal and deterministic oracles. To alleviate both issues, we propose a simple yet effective labeling algorithm that creates soft, expectation-based sentence labels. We define a new learning objective for extractive summarization which incorporates learning signals from multiple oracle summaries and prove it is equivalent to estimating the oracle expectation for each document sentence. Without any architectural modifications, the proposed labeling scheme achieves superior performance on a variety of summarization benchmarks across domains and languages, in both supervised and zero-shot settings.

Published as a conference paper at ICLR 2023
TEXT SUMMARIZATION WITH ORACLE EXPECTATION
Yumo Xu & Mirella Lapata
Institute for Language. Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB yumo. xuêed. ac. uk,
ed. ac.uk
ABSTRACT
Extractive summarization produces summaries by identifying and concatenating the most important sentences in document. Since most summarization datasets do not come with gold labels indicating whether document sentences are summary-worthy, different labeling algorithms have been proposed extrapolate oracle extracts for model training. In this work, we identify two flaws with the widely used greedy labeling approach: delivers suboptimal and deterministic oracles. To alleviate both issues, we propose a simple yet effective labeling algorithm that creates soft, expectation-based sentence labels. We define a new learning objective for extractive summarization which incorporates learning signals from multiple oracle summaries and prove it is equivalent to estimating the oracle expectation for each document sentence. Without any architectural modifications, the proposed labeling scheme achieves superior performance on variety of summarization benchmarks across domains and languages, in both supervised and zero-shot settings
INTRODUCTION
Summarization is the process condensing a source text into a shorter version while preserving its information content. Thanks to neural encoder-decoder models (Bahdanau al. 2013 [Sutskever architectures (Vaswani et al. 2017), and large-scale pretraining (Liu & Lapata 20199] Zhang et al. 2020a Lewis et al. 2020), the past few years have witnessed huge leap forward summarization technology. Abstractive methods fluently paraphrase the main content of the input, using vocabulary different from the original document, while extractive approaches are less creative - they produce summaries by identifying and subsequently concatenating the most important sentences in document but manage to avoid ballucinations, false statements and inconsistencies.
cet al. 4), Transformer-basse
Neural extractive summarization is typically formulated sequence labeling problem (Cheng & Lapata 2016), assuming access to (binary) labels indicating whether a document sentence should be in the summary. In contrast to the plethora of datasets (see Section [ examples) available for abstractive summarization (typically thousands of document-absttrat pairs), there are no large-scale datasets with gold sentence labels for extractive summarization. Oracle labels are thus extrapolated from abstracts via heuristics, amongst which greedy search (Nallapati et al. (201)) the most popular by far (Liu & Lapata 2019 Xu et al. 2020 Dou et al. 2021 Jia et al. 2022).
In this work we challenge received wisdom and rethink whether greedy search the best way to create sentence labels for extractive summarization. Specifically, we highlight two flaws with greedy labeling: (1) the search procedure is suboptimal, i.e., it does not guarantee global optimum for the search objective, and (2) greedy oracles are deterministic, i.e., they yield a single reference extract for any given input by associating sentences the document to its corresponding abstract.
Perhaps an obvious solution to the suboptimality problem would be to look for oracle summaries following a procedure based on beam search. Although beam search finds better oracles, we empirically observe that summarization models trained on these do not consistently improve over greedy oracles, possibly due to the higher risk of under-fitting (Narayan al. 2018a) there are too few positive labels. Moreover, beam search would also create deterministic oracles. A summarization
Our code and models can be found http::://gtttbbb.///////////////

---

Published as a conference paper at ICLR 2023
Table 1: Sentence labels for a CNN/DM article according different labeling schemes. Only the first 10 document sentences are shown. Greedy and Beam create oracle summaries (i.c., sentences with label 1) with greedy and beam search, respectively. OREO, our labeling algorithm, incorporates information from multiple summary hypotheses shown in the bar chart (R is the mean of ROUGE- and ROUGE-2). OREO assigns high scores (> 0.5) sentences and which contain important named entity, Jasmine Coleman, and location, Croydon, South East London. In comparison, greedy and beam labeling consider only one oracle summary. and assign zero to sentences or 4, failing to capture that these are informative and should be probably included in the summary.
Document Seutence
Greedy
OREO
1 Jasmine Coleman, 12, has been found safe and well some 50 miles from her home.
0
0.568
2 A 12-year-old girl who went missing from her family home at 2 AM amid fears she was has found safe well.
1
1.000
driven away by "older
3 Jasmine Coleman was reported as missing this morning after disappearing from her home in lancling west Sussex.
0
0.429
4 The child was found this afteroon following police appeal 50miles away 1 Croydon, South East London.
0.778
5 Police feared she may have driven London by an man when they launched
appeal for information this morning.
6 schoolgirl not heen
PM Friday night.
7 Sussex police said she may have boen talking with somcone Facetime before disappearing 0 at around am.
0,555
8 The force launched public uppeal for information her whereahouts Saturday 0 ing.
0.000
9 In she deseribed as fair with long. blonde hair and having possibly been wearing black riding trousers and polo shirt or paisley pattern dress.
0
0.000
Saturday altemoon force confirmed had could not confirm the cireumstances under which police located her.
in Croydon
Reference Summary
60
Jasmine Coleman disappeared from her home around 2 AM this morning.
40
20
Police believed she may have been driven to-
wards London by an older man.
She has been found safe and well in Croydon, South East London today.
2 245 5
147
247 7
124 4
2 5
233
2.3
2
4
127 7
7
7
3 345
35 10
Top-16 Beams (Beam Size k =256)
system trained on either greedy or beam oracles is optimized by maximizing the likelihood of single oracle summary. This ignores the fact that there can be multiple valid summaries for an article, in other words, the summary hypothesis space is naturally a multi-modal probability distribution. We illustrate this point in Table
In this paper define new learning objective for extractive summarization which promotes non- deterministic learning in the summary hypothesis space, and introduce OREO, ORacle ExpectatiOn labeling, simple yet effective sentence labeling scheme. We prove the equivalence between estimating OREO labels and optimizing the proposed learning objective. As result, it is sufficient for current models be trained OREO labels without requiring any architectural changes.
Extensive experiments on summarization benchmarks show that OREO outperforms comparison labeling schemes in both supervised and zero-shot settings, including cross-domain and cross-lingual tasks. Additionally, we showcase that extracts created by OREO can better guide the learning and inference of generative system, facilitating the generation of higher-quality abstracts. We further analyze ORFO's 's behavior by measuring attainable summary knowledge at inference time, and demonstrate it is superior to related deterministic and soft labeling schemes, which argue contributes to consistent performance gain across summarization tasks.
2 RELATED WORK
Narayan et al. (011a) were among the first discuss problematic aspects of sentence labeling schemes for extractive summarization. They argue that labeling sentences individually as in Cheng & Lapata 2016) often generates too many positive labels which leads to overfitting, while model
2

---

Published as a conference paper at ICLR 2023
trained on greedy labels (allapati et al. 2017) underfits the data. Although extractive performance can be boosted via finetuning pretrained encoders with greedy labels (Liu & Lapata 2019). Zhong et al.] (2020) show that reranking summary candidates constructed from greedy predictions can further improve summary quality. This demonstrates that the underfitting problem caused by greedy labels still exists even when pretrained models are used. Issues with greedy labeling have also been discussed from the perspective of data bias, including lead bias (Nenkoval 2005 Kedzic et al. (2018) Grenander al. 2019)) greedy labels display a bias towards lead sentences in news text and systems trained on them do not easily transfer to other domains - and monolingual bias (Jia et al. 2022) greedy labels created for one language (e.g., English) do not transfer to different language.
The idea of learning from multiple references has found application various tasks including dialog response generation (Gao et al. 2019), machine translation (Khayrallah et al. 2020), and question answering (Zhu et al., 2020). Summarization datasets with multiple references are not generally available for model training. but a few have been manually created for system evaluation (Dang) 2005). extractive summarization, gold references in the form of sentence labels do not usually exist, and learning from multiple references has not been yet explored. In this work, we use beam search to create multiple high-quality oracle summaries, from which summary-level supervision is aggregated into sentence labels to promote multi-reference learning for extractive summarization.
3 PROBLEM FORMULATION
Let D= {- denote a document consisting of sentences z;, An extractive summarizer produces summary hypothesis that represents salient information via identifying subset of sentences y {0, Jf, << m within D. In practice, ROUGE (Lin & Hovy 2003). an automatic metric based on lexical overlap is commonly adopted to evaluate RYY, .S), the quality of Y against gold reference summary S.
Following previous work (Cheng & Lapata 2016 Nallapati et al.) 2017 Narayan al. 201
& Lapata 2019), we conceptualize extractive summarization as sequence labeling task, and aim to build a system that estimates the summary-worthhe worthiiess of each sentence in a non aon-autorerresivee manner. As mentioned carlier, sentence labels need to be first extrapolated to train an extractive system, since existing datasets are label-free, they only contain document-abstin abstract pairs. BERTSUM (Liu & Lapata 2019) is popular extractive model and representative of the approach sketched above. Built on top of BERT (Devlin et 2019), adds two-layer Transformer (Vaswani et al. 2017) for sentence representation and a s sigmoid d layer for summary prediction. During inference, document sentences {Tr 1m are ranked based on their estimated scores, and summary {M, Jn is identified. The number of n sentences to be included in the summary is often pre-defined and fixed.
4 FROM EXISTING LABELING SCHEMES TO OREO
Early labeling methods create sentence labels & by evaluating the similarity of x; against reference
summary S through various heuristics ft.), e def h(z,, S), including ROUGE (Lin & Hovy 2003 and rule-based features such as sentence and paragraph position information, and the number of mentioned entities (Woodsend & Lapata 2010 Cheng & Lapata 2016). These methods obtain local labels as they assume a sentence can be classified as summary-worlee on its own, without taking the summary context into account.
However, model evaluation does not operate at the sentence-level, as a sentence part a summary hypothesis Y together with other candidates (Narayan et al. 2018a). The aim of extractive summa- rization to deliver a high-quality summary hypothesis, i.e., good set of sentences rather than a set of good sentences. A sentence might achieve high score on its own but contribute little to summary hypothesis, e.g., due to redundancy. An alternative is obtain global labels, based on whether sentence occurs within the optimal set of sentences which collectively achieve the highest score according to some evaluation metric like ROUGE:
def 1(I E Y*) where Y = arg max R(Y,)). YEC(D)
(I)
where |C(D)] = ((m) is the hypothesis combinatorial space. As Equation is computationally intractable, in practice, is approximated by further conditioning heuristic search space & such
3

---

Published as a conference paper at ICLR 2023
that Y* arg ES(D) S),, and the approximated Y is usually called an oracle summary. A widely adopted approximation is greedy labeling (Nallapati et al.) 2017 Narayan et al. 220 8a Liu & Lapata 2019; Zhong al. (020), which uses greedy search to maximize R at each step of sentence selection (the algorithm stops when R can no longer increase or the maximum number of steps is reached; see Appendix [ for the algorithm).
While significantly reducing complexity, greedy labeling does not guarantee a global optimum. To find better summaries to serve as oracles, we propose replace greedy search with beam search which we refer to as beam labeling. We empirically find that around 8%-20% of (greedy) labels can be potentially improved with beam search (when setting the bean size to 256; see Appendix for details). However, having better labels does not necessarily translate to performance improvements, and we next discuss why this is the case.
4.1 OREO: ESTIMATING ORACLE EXPECTATION
Extractive summarization models are typically trained to optimize max mnaxpa(Y (Y*)) where the best hypothesis Y* can be approximated with greedy beam search. This learning objective maximizes the probability at single point Y*, and assigns zero probability to other summary hypotheses Y, regardless of their quality. We note that this formulation leads to a discrepancy between how the model is optimized and how the labels against which this optimization takes place are obtained. Given an input document, sequence labeling summarization models assume conditional independence at sentence-level inference, while in greedy labeling, each step in the process of maximizing R(Y*, S) conditions on the outcomes of previous steps. From an optimization perspective, this mismatch renders fitting a non-autoregressive sequence labeler difficult for two reasons: (1) learning to search and maximizing the likelihood at Y is challenging, and so the model tends to underfit Y* Narayan al., 2018a), and (2) probabilities at other Y with high evaluation scores remain under-estimatee and uncalibrated due to supervision sparsity. Simply replacing greedy search with beam search does not resolve these optimization challenges point estimation still performed.
solution is to evaluate summary hypotheses during training and reward the model accordingly (Narayan et al. 2018a). However, this is non-trivial as the the metric R is usually non-differentiable, and is computational expensive to sample from large combinatorial hypothesis space, e.g., with Reinforcement Learning (Sutton Barto 1998). Rather than changing the training of the model, in this work, we study how to derive a better sentence labeling algorithm that leads to a better optimization objective.
Specifically, we wish to incorporate multiple high-quality hypotheses as oracle summaries into the learning objective. Our key assumption is that extractive oracles non-deterministic, but drawn from a distribution p(Y' D,S)) S). We thus formulate the objective for extractive summarization as: R(Y", (S)oo
E Y Y*>p(Y
Under this formulation, an optimized model is expected assign high probability p D)) when there exists an oracle summary with high probability and high score according to some quality evaluation metric.
From the perspective of sentence labeling, we note that candidate x; relates to the summarization task through the oracle summary space W. As Y is a combinatorial space, the mapping x; Y* is one-to-many. Therefore, we can compute the probability for each candidate to be selected via marginalization:
(s, (.n,))=<(a,, Y'I,.)
To connect the marginalization in Equation (3) with the summarization objective in Equation 2 we further incorporate hypothesis evaluation RY S), and define the summary-worthinee sentence 2; as the expectation of its associated oracle evaluation:
def
((Y",S)((zlY"" DD)(("" D) y (D.)
E
R(Y",S) p(x:(Y",))
(4)
prucle evaluation oracle membership
reade
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 Labeling with Oracle Expectation
1: function OREO(n, k, p)
1: function STEP(j. B)
Step and beam
2:
Max number of sentences in summary k: beam size; p.: oracle distribution
2: Initialize visited paths V
3: Initialize beam B
3:
for b, r <- B do
[b] < j then
5: for j 4 n do
5:
continue
P Skip early stopping
6: B < STEP(. 15)
6:
+ [D] do
7: Initialize f to O, Vi. b Pre-scaled expectation
7:
8' = SORT(b + (2))
8:
for b, < B do
8:
if not in V then
9: for i f b do
9:
r' ROUGE())
10:
4 + + p(b)
10:
r' then
11: L MAXMINSAALE(C)
11:
B B ((Kr))
12: return 8
12:
V V {b)
13:
end function
13: return ToP-k(B)
Pruned beam
14: end function
where the oracle membership p(r, V D) is identical to y = (x, Y*) and the oracle distribu-
tion will be discussed in Section 4.3 Given sequence labeling model 0. maximizing the oracle
expectation for all input sentences is equivalent to the objective Equation 2 We present the proof
in Appendix
To be compatible with the standard sequence labeling architecture for extractive summarization, we perform MLE with cross-entropy loss:
min C(0) min) Σ
CrossEntropy (o:
(5)
where the scaled expectation ((x,) (e: mminn/ mmn) constitutes the final sentence labels. The details of oracle expectation labeling are given in Algorithm
4.2 COMPARISON WITH EXISTING LABELING ALGORITHMS
OREO creates soft (continuous) sentence labels, i.e., it incorporates summary-leved evaluation while maintaining low sparsity. A detailed comparison with other labeling algorithms is provided in Table 3 Equation (4) also bears resemblance to the RL objective used in Narayan al. (018a):
max E (R(Y,))) They evaluate summary hypotheses directly while Equation (4) es- timates sentence-level membership. This is consequence of the nature of the sequence laneler which does not explicitly represent the summary hypothesis space (which is combinatorial), and therefore supervision is delegated to sentences rather than summaries. By maximizing sentence-level likelihood, estimations for the associated hypotheses are updated, albeit indirectly.
Narayan et al. (2018) employ REINFORCE (Williams) 1992), an on-policy RL algorithm that samples from a model during training, while in Equation (4), samples are drawn from the non- parametrized oracle distribution MY*1D, D,.S). We provide offine supervision in the form of static sample labels. In contrast to the online reward in RL, offline labels can be reused during the course of training and therefore more sample efficient. Our labeling scheme
be seen a type of offline bandit learning (Nguyen-Tane Tang et al. 2022). While the latter has been recently applied to abstractive summarization (Pang & He 2021) remains under-explored in extractive summarization.
4.3 THE ORACLE DISTRIBUTION
We derive the oracle distribution p(Y*) heuristically bearing in mind that: (a) we have no prior knowledge to which hypothesis is more less likely as oracle summary and therefore assume the oracle distribution to uniform over a large hypothesis space; and (b) it is desirable for p(Y* (D) expect this correlation become stronger
to positively correlate with R(Y*, ,S) and course of optimization. In practice, we use beam search (with beam size << [V] to find potential
the
oracle summaries, and adopt uniform distribution over top-ranked beams: p p(Y*DD) where f < is a aaper--araaeter which we optimize on a development set. We also experimented with several weight annealing mechanisms over top beams determined by R, our summary quality evaluation metric (see Appendix D for details).
U1,,
5

---

Published as a conference paper at ICLR 2023
Table 2: Datasets for monolingual and cross-lingual (last column) summarization. Compression rate denotes the number of sentences extracted to form a summary; and f denotes that trigram blocking (Liu & Lapata 2019) was applied in sentence selection for redundancy removal.
Datasets
CNN/DM
XSum
Multi-News
Reddit
WikiHow
MLSum
Language
En
En
En
En
En
En/De/Es/Fr/Ru/Tr
Domain
Newswire
Newswire
Newswire
Social Media
Wikipedia
Newswire
#Train
287,084
203,028
44,972
41,675
168,126
287,227 (En)
#Validation
13,367
11,273
5,622
645
6,000
13,368 (En)
#Test
11,489
11,332 2
5,622
645
6,000
53,981 (Non-En)
#Compression Rate
3
9
2
4
21
Table 3: Sentence labeling schemes for extractive summarization. Sum refers to summary-level evaluation. m, n, and ls re- spectively denote document size, summary size, and beam size.
Extractive performance (test set, ROUGE-L) on CNN/DM (CD), XSum (XS), Multi-News (MN), Reddit (RD), and WikiHow (WH). We highlight best scores and scores out- side the 95% confidence interval of OREO (using bootstrap resampling; Davison & Hinkley 1997).
Table 4:
Scheme
Sum Sparsity Complexity
Local
Medium (())
Systems
CD
XS
MN
RD
WH
Global
/
High
aiim-n]!
mi
LEAD
36.67
14.79
38.97
23.24 20.13
14.34
Greedy
J
High
(nrn logm) w))
MATCHSUM
40.38
18.41
41.89
29.58
Beam (ours)
/ High
(nrnk log (m)))
ORACLE
OREO (ours)
/ Low
O (nmk (ogmm))
Greedy
48,87
23.57
44.27
28.98
32.68
Beam
52.86
23.71
46.40
29.11
36.51
OREO
50.08
20.07
46.14
24.55
34.28
BERTSUM
Greedy
39.56
17.16
41.53
19.11
28.24
Beam
39.66
17.66
41.50
19.81
25.71
EXPERIMENTS
OREO
39.96
17.81
41.71
20.02
28.46
5.1 SUPERVISED EXTRACTIVE SUMMARIZATION
We conducted all extractive summarization experiments with BERTSUM (Liu & Lapata 2019), the neural summarization architecture introduced in Section 3 We opted for BERTS UM due to its simplicity and popularity in wide range of summarization tasks (Zhong et al.) 2020 Xu & Lapata 2021;Jia et al., (2022). We nevertheless note that OREO is model-agnostiic and can be also applied to more complex architectures. For a fair comparison between different labeling schemes, we follow the standard training configuration used in Liu & Lapata (2019) without any additional optimization (c.g.. for our specific labeling scheme). We set R, the summary quality evaluation metric. to the mean of ROUGE-1 and ROUGE-2. We report experiments on variety of summarization datasets including CNN/DM (Hermann al. 2015), XSum (Narayan al. 20188), Multi-News (Fabbri et al. 2019), Reddit (Kim et al. 2019 and WikiHow (Koupaee & Wang 2018). Detailed statistics are shown Table 2
Our results are presented in Table 4 In the first block, we report the performance of the LEAD baseline which considers the first k sentences in a document as the summary (see last row in Table and MATCHSUM (Zhong et al. 2020), a tate-of-the-ar system which performs summary reranking with another BERT model. The second block reports ORACLE performance with greedy labels, beam labels (k 256). and OReO labels (k = 256, t = 16; we take the top-n sentences with non-zero scores). See Appendix E for the labeling hyperparameters
t for each dataset, and more detail on experimental settings. The third block reports BERTSUM performance with different labeling schemes.
Although beam ORACLE is superior to greedy ORACLE and raises the upper bound, the overall performance of BERTSUM optimized on beam labels does not significantly improve upon its greedy counterpart. In fact, performance drops drastically on WikiHow. OREO shows inferior ORACLE re- sults considers multiple top-ranked beams and is therefore not bound-preserving (see Section 5.5 for detailed analysis). However, BERTSUM trained with OREO labels consistently outperforms a model trained with beam labels, and achieves 08.89 ROUGE-I. improvement different

---

Published as a conference paper at ICLR 2023
15
Greedy Beam
Greedy Beam
Greedy Beam
Greedy Beam
Ratoo (9)
10
OREO
Ratio (%)
10
OREO
Ratio (3)
OREO
Ratio (%)
10
OREO
9 157799
113151771
9 133517119
3331777
(a)
CNN/DM
(b) XSum
(c)
Reddit
(d) WikiHow
Figure 1: Distribution of label values over sentence positions in documents (development set).
Table 5: Zero-shot cross- Table 6: domain performance on var- tion on MLSum (test set, ROUGE-L). Sys- stractive summarization ious test sets for mod- tems with are supervised. Systems with els trained on CNN/DM use XLM-R large. (ROUGE-L).
Zero-shot cross-lingual summariza- Table 7: Results for ab-
on CNN/DM (test set), R-1/2/L is a shorthand for ROUGE.
Systems
De
Es
Fr
Ru
Tr
AVG
CNN/DM XS
RD
WII
Xt.S NLS
41.28
21.99
24.12
10.44
33.29 26.22
Systems R-1 R-2 R-L
BERTSUM
34.95 21.20 23.59
10.13
31.49 24.27
BART 44.16
21.28 40.90
Greedy 15.62 17.06 25.39
XLS
GSUM
Beam
15.62 17.64 24.77
Greedy 28.75 20.83 23.10 9.43 29.52 22.33
Greedy 44.40 21.52 41.23
41.22
OREO
15.58 17.71 25.62
Beam 26.43 20.90 23.41
9.42
29.80 21.99
Beam 44.41 21.55 41.26
OREO 31.47 20.84 23.10
9.44 31.71 23.31
OREO 44.81 21.83 41.60
benchmarks compared to greedy labeling. Although BERTSUM with OREO still falls short of the state-of-the-art show that one-stage summarization modeling be enhanced with better labeling, which can serve as a foundation for more complex reranking methods (see Table in Appendix for details).
5.2 ZERO-SIKE SHOT CROSS-DOMAIN EXTRACTIVE SUMMARIZATION
We further examine the generalization capability of models trained with OREO labels in a zero-shot setting. Specifically, we evaluate a model trained on CNN/DM, against XSum, another news summarization dataset with shorter summaries (at most 2 sentences), and Reddit and WikiHow which represent entirely different domains and topics (discussion forums and instructional text).
Table 5) summarizes our results. Models trained with OREO perform on par with greedy labeling in-domain but display stronger generalization cross-domain. Greedy labels are more prone to lead bias, they deem as summary-worthe sentences mostly from the beginming of the document. Such bias present in news datasets like CNN/DM but does not transfer other domains like social media or Wikipedia. OREO alleviates this bias and performs better out-of-domain. As shown in Figures and bb), OREO is less concentrated on lead sentences in news text.
5.3 ZERO-SHOT CROSS-LINGUAL EXTRACTIVE SUMMARIZATION
We next investigate the generalization capabilities of our approach in cross-lingual setting. We use English data for model training and report zero-shot results on a variety of languages from the MLSum dataset (Scialom et al. 2020 see Tar detailed statistics). Following Jia al. (2022), we augment English articles with word replacement during training (Qin et al., 2021) using the MUSE (Lample et al. 2018) bilingual dictionary to align multilingual representations, We adopt word replacement rate of O.5. BERTSUM was initialized with XLM-R base (Conneau et al. 2020),
cross-lingual pretrained model (see XLS in Table
The first block in Table reports the results supervised X1.S model which has access to training
data for all languages; NL.S is the zero-shot state of the art system Jf et al. (2022); their approach creates multiple sets greedy labels with different machine translation methods and adopts a neural architecture to learn weights for the obtained label sets. The second block presents the results of
We also experimented with mBERT Devlin 2019 and achieved similar results. See Appendix

---

Published as a conference paper at ICLR 2023
zero-shot XLS model with different labeling schemes. As can be seen, OREO labels on Spanish, French, and Russian are on par with greedy labeling. Systems trained with greedy labels exhibit less cross-lingual generalization
German and Turkish, while OREO improves system performance on German by 2.72 ROUGE-L points and on Turkish by 2.19. Previous work (Jia et al., 2022) shows that cross- tross-lingual performance correlates with lead bias in the target language. For example, Turkish articles are less lead-biased than Russian in MLSum, and thus benefit more from better sentence labeling. OREO trails behind NLS which is not surprising the latter model benefits from more resources, i.., machine translation and XLM-R large, and a more complex network architecture.
5.4 SUPERVISED ABSTRACTIVE SUMMARIZATION
We further assessed whether the proposed labeling scheme is of benefit to abstractive summarization. We experimented with GSUM (Dou al.] 2021), state-of-the-art abstractive system that takes extractive summaries as additional input to guide the generation of document abstracts. During training. GSUM uses extractive oracles as guidance, while at inference time guidance is provided by summary hypotheses produced by a trained extractive system. We initialized GSUM with BART (Lewis et al. 2020), and used BERTSUM the guidance model optimized with different labeling schemes (i.c., greedy, beam and OREO).
Abstractive summarization results are shown in Table 7 The first block shows the performance of BART (Lewis et al. 2020) which serves as a baseline. In the second block, we report the performance of GSUM (Dou et al. 202T) with greedy labels (default) in addition to beam- and OREO-based variants. As we can see, while beam labeling performs on par with its greedy counterpart, OReO guidance boosts performance with 0.37 ROUGE-L points over vanilla GSUM. We conclude that abstractive systems can also benefit from our expectation-bas -based labeling algorithm, without any modeling changes or hyperparameter optimization. More results with varied guidance settings can be found in Appendix Examples of system output are shown in Appendix G
5.5 COMPARISON WITH BOUND- PRESERVING METHODS
Let {2t ",),, E {0, 1} denote a multi-hot representation of an oracle summary. We define a labeling function as bound-preserving (f(1;) > 7) 2,, Vi holds. Intuitively, the condition holds and only if the top-ranked sentences remain identical. Bound preservation of soft labels guarantees that the performance upper bound of summarization system trained on soft labels equals that of a system trained on their original hard labels, e.g., obtained via greedy and beam search. For instance, label smoothing (Szegedy al. 2016). a common technique for training deep neural networks, is bound- nuude-prssrrvigg.. In contrast, OREO is generally not bound- ponndd-resrriigg for either beam or greedy oracles To further analyse this property of soft labels, we propose ORMAX as a bound-preserving variant, by replacing the expectation with a max operator: e def mp ()((')"))(("', S))] Compared to OREO, ORMAX incorporates multiple oracles while additionally preserving the upper bound of beam labels 7
if there exists constant y E [0, 1] so that the condition
Table g shows the performance of label smoothing and ORMAX for extractive (first block) and abstractive (second block) summarization. Although label smoothing has been successfully applied to discriminative (Szegedy et al. 2016) and generative NLP tasks (Chen et al. 2018 Lewis et al.) 2020), the soft labels it creates do not yield better results than their original hard labels in extractive summarization. Label smoothing performs implicit model calibration (Müller et al.) 2019). which can potentially improve sentence ranking and selection at inference, however, it also imposes regularization in neural networks training (Szegedy et al., 2016), which may render it less effective for extractive summarization where there is a higher risk of underfitting (Narayan et al. 2018a). On the other hand, ORMAX performs on par with ORFO on abstractive summarization, while it underperforms on extractive summarization. Although bound preservation is, intuitively, desirable, our experimental results suggest that it is neither necessary nor sufficient condition to produce well-optimized summarization system.
Unnder two special cases our labeling scheme is bound-preserving: (1) with beam size k 1, Oreo equivalent to greedy labeling and (2) with top beam size
OREO is equivalent to beam labeling. *ORMAX is trivially bound- preserving since sentences selected by the top-ranked beam receive the highest score, and the top-ranked beam can be reconstructed by the top-ranked sentences. We illustrate the difference hetween OReO and bound- bound- preserving methods in Figure 5 in Appendix H
8

---

Published as a conference paper at ICLR 2023
Table 8: Comparison of OREO to bound- preserving labeling (CNN/DM test set). Results shown for extractive (BERTSUM) and abstrac- tive (GSLM) summarization. LS refers Label Smoothing (a optimized between {0.1, 0.2, 0.3)).
BOUNDD
60
8 ASk
R- Re-1//LL is shorthand for ROUGE.
40
Systems
R-1
R-2
R-L
BERTSUM
20
Greedy
43.18
20.16
39.56
Greedy+LS
10.03
10.01
t0.03
Beam
4322
20.14
39.66
Greedy+LS Beam+LS
ORMAX
OREO
Beam+LS
10.00
10.02
10.01
OOROO"
43338
20.43
39395
ORMAX
10.20
40.27
40.20
Figure 2: Upper bound and attainable sum- mary knowledge captured by sentence label- ing method (CNN/DM validation set) for La- bel Smoothing (+LS), ORMAX, and OREO.
GSUM
OREO
44.81
21.83
41.60
ORMAX
10.04
10.02
10.00
5.6 PERFORMANCE ANALYSIS
Previous experiments revealed that oracles are not necessarily indicative of model performance (see Tables 4 and
and (8. due to the discrepancy between model optimization and sentence labeling, as
discussed Section 4.1 To further understand how different labeling schemes (and the oracles based on them) influence model performance, quantify this discrepancy via a sampling-basee method which simulates sentence selection for a sequence labeling model at inference.
Bearing in mind that non-auvoregressive sequence labeling model performs conditionally indepen- dent predictions and selects a fixed size number of n sentences, we construct summary hypotheses
Y = { by drawing independent sentence samples, and measure the extent to which a model can Attain Summary relevant Knowlege (ASK) as:
ASK def
(R((5)))=1:S) where p(x;)D,S)
E 4
(05) P(ps D,S)
Note that p(z;) D. S) is shaped by soft labels, and thus results in varied sentence/summary samples for different labeling schemes. The comparison FigureZZexplains why we observe performance gains from Oreo despite obtaining the lowest upper bound performance. The latter considers only the best case scenario at inference, ignoring the fact that some summary knowledge encoded in sentence labels can be hard or impossible attain, e.g., when sentences in the oracle summary are highly co-dependent (and is therefore challenging to select them jointly with a model making independent predictions), or the oracle summary contains less than sentences (which again entails that information missing). Compared to other labeling schemes, OREO captures richer summary information that is attainable for sequence labeling models, narrowing the distance between upper bound performance and ASK. Consistent with our analysis, systems trained on ORFO perform robustly on wide variety of summarization tasks.
6 CONCLUSIONS AND FUTURE WORK
We provided a comprehensive analysis existing labeling schemes for extractive summarization, and identified two flaws in greedy labeling, namely it delivers suboptimal and deterministie labels. We proposed a novel optimization objective to learn from multiple oracle summaries, which can be instantiated by labeling scheme based oracle expectation. Experimental results show that the proposed scheme achieves substantial improvement across domains and languages, without any architectural modifications. Our framework is agnostic the labeling metric R. however, an important future direction is to incorporate different learning signals and provide sentence labels with more desirable properties, such query relevance (Xu & Lapata 2022) and faithfulness (Durmus) et al. 2020). We would also like to parametrize the oracle distribution and estimate it from data, so to derive even more accurate sentence labels.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their feedback and Shucong Zhang for useful discussions about the problem formulation. We gratefully acknowledge the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1).
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA, May 2015.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 76-86, Melbourne, Australia, July 2018.
Jianpeng Cheng and Mirella Lapata. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties, volume 1, pp. 484 494, Berlin, Germany, 2016.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 8440-8451, Online, July 2020.
Hoa Trang Dang. Overview of duc 2005. In Proceedings of the 2005 Document Understanding Conference, pp. 1-12, Vancouver, Canada, October 2005.
Anthony Christopher Davison and David Victor Hinkley. Bootstrap Methods and their Application. Number in Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1997.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171 11814116, Minneapolis, Minnesota, 2019.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4830 4842, Online, June 2021.
Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5055-5070, Online, July 2020.
Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-News: A large- scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, August 2019.
Xiang Gao, Sungjin Lee, Yizhe Zhang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. Jointly optimizing diversity and relevance in neural response generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 1229-1238, Minneapolis, Minnesota, June 2019.
10

---

Published as a conference paper at ICLR 2023
Matt Grenander, Yue Dong, Jackie Chi Kit Cheung. and Annie Louis. Countering the effects of lead bias in news summarization via stule training and auxiliary losses. In Proceedings of the Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNEL pp. 6014-6024, Hong Kong, China, November 2019.
2019 Conference
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 1693 1701, Cambridge, MA, USA, 2015.
Ruipeng Jia, Xingxing Zhang, Yanan Cao, Zheng Lin, Shi Wang, and Furu Wei. Neural label search for zero-shot multi-lingual extractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 561-570, Dublin, Ireland, May 2022.
Chris Kedzie, Kathleen McKeown, and Hal Daumé III. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1818-1828, Brussels, Belgium, October-November 2018.
Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. Simulated multiple reference training improves low-resource machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 82-89, Online, November 2020.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019.
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. arXiv preprint UrXiv:18100 810009305 2018.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In Proceedings of the 6th International Conference Learning Representations, Vancouver, BC, Canada, April May 2018
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. pre-training for
Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 17880, Online, July 2020.
Chin-Yes Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 71-78, Edmonton, Canada, 2003.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, pp. 3730-3740, Hong Kong, China, November 2019.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. Summarunner: recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference Artificial Intelligence, pp. 3075-3081, San Francisco, Califomia, USA, 2017.
11

---

Published as a conference paper at ICLR 2023
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1747-1759, New Orleans, Louisiana, 2018a.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! ! topic-aware convolutional neural networks for extreme summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018b.
Ani Nenkova. Automatic text summarization of newswire: Lessons learned from the document understanding conference. In Manuela M. Veloso and Subbarao Kambhampati (eds.). Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, pp. 1436-1441, Pittsburgh, Pennsylvania, USA,
July 2005.
Jun-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925-1930, Lisbon, Portugal, September 2015.
Thanh Nguyen-Tanne Sunil Gupta, A. Tuan Nguyen, and Svetha Venkatesh. Offline neural contextual bandits: Pessimism, optimization and generalization. In Proceedings of the 10th International Conference on Learning Representations, Online, April 2022.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. Proceedings of the 9th International Conference on Learning Representutions, Online, May 2021.
Pinelopi Papalampidi and Mirella Lapata. Hierarchical3D adapters for long video-to-text summariza- tion. arXiv preprini Nti:2220000422 2022.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. CoSDA-ML: multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences Artificial Intelligence, pp. 3853-3860, 2021.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 3246-3256, Hong Kong, China, November 2019.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. MLSUM: The multilingual summarization corpus. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 8051-8067, Online, November 2020.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pp. 3104-3112 Curran Associates, Inc., 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 8-2826, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998-59999999 6008. Curran Associates, Inc., 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-2555 1992.
Kristian Woodsend and Mirella Lapata. Automatic generation of story highlights. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 565-574, Uppsala, Sweden, July 2010.
12

---

Published as a conference paper at ICLR 2023
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. Discourse- aware neural extractive text sum- marization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5021-5031, Online, July 2020.
Yumo Xu and Mirella Lapata. Generating query focused summaries from query-free resources. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 6096-616 6109, Online, August 2021.
Yumo Xu and Mirella Lapata. Document summarization with latent queries. Transactions of the Association for Computational Linguistics, 10:1-16, 2022. doi: 10.1 2a1116/tacll_a aasl___00880
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11328- 11339, July 2020a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: evaluating text generation with BERT. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, April 2020b.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceed- ings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 563-578, Hong Kong, China, November 2019.
Ming Zhong. Pengfei Liu, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. Extractive summarization text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6197-6208, Online, July 2020.
Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy, Question answering with long multiple-span answers. Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3840-3849, Online, November 2020.
13

---

Published as a conference paper at ICLR 2023
A GREEDY SEARCH ALGORITHM
Algorithm Labeling with Greedy Search
function GREEDY(n)
Max number of sentences in summary
2: Initialize hypothesis to empty
3: for j 5- nt do
4: Initialize hypothesis score v to 0
5: for < [] do
6:
6 = b + {i]. v' ROUGE(b))
7:
if > V then
8:
b b- b', +- of
9:
if 0 then
Early stopping
10:
break
11: Initialize 4 to 0, Vi
Multi-hot labels
12: for Tj - do
13:
14:
return & end function
15:
B LABEL STATISTICS
100
90
Do
CNN/DM
100
Do XSum
R(Y*) (R (Yaa)) (%)
80
Multi-Nes
Rank of Greedy
70
99.9
60
50
99.8
40
30
99.7
CNN/DM XSum Multi-News
20
10
99.6
00
64
256
3
9 10
Beam Size k
Ratio (%)
Figure 4: Relative quality of best beams for beam labeling: results shown for different beam sizes across three validation sets.
Figure 3:
Distribution of greedy oracles over top beams across three validation sets.
Quality of Greedy Labels Figure 3 shows the distribution greedy oracles over position in beam search results, as ranked by R (k 256). As we can see, around 8-20% of greedy labels are not top-ranked, and could thus improve with beam labels. However, as shown our experimental results, this improvement does not lead to a summarization system that is consistently better across different tasks.
Effects of Beam Size As beam search does not guarantee a global optimum either, we further evaluate the relative quality of the top beam Y* (the top beam found by varied beam sizes), compared against Yz5s (the top beam found by beam size 256). Figure shows that the quality of the top beam converges when beam size increases to 64. However, oracles not necessarily indicative of actual model performance (see Section 5.6 for details), we view beam size hyperparameter for optimization.
calculate ((Y"))R(YEs))
14

---

Published as a conference paper at ICLR 2023
C EQUIVALENCE PROOF
Given input document D, sentence-level inference a non-autoregressivve summarization model conditionally independent, and the likelihood of oracle summary Y {v: mm (in its multi-hot representation over m document sentences) is calculated as:
(Y" (
II pa(a; =, D)
In this case, we show that maximizing the oracle expectation for all sentences equivalent to the objective in Equation (
m
17
max II 1=1
pa pa = GJD.S) =
E (Y D.S)
RRY"" (); Y D)]
D> Plug in OReO
m R(Y", S) II pa = y/ D)
E y'>p(Y' (D.))
D> Take out E
E (D,,S)
RRYY" S))po(Y
Apply Eq.
We note that objective in Equation serves lower bound of the classic extractive summa- rization objective max poppo (D) weighted by oracle evaluation:
max
(9)
V'mp(Y" D.S)
Y
P D, (y)o))
(10)
R(Y best S) (Y ID) where Y
arg max R(Y", Y-EY
(11)
x pepe(Y D)) The equality holds only if the oracle distribution D(Y))D,,5) is Dirac delta distribution 5(Y Y best)
(12)
D EFFECTS OF ORACLE DISTRIBUTION
Table 9: OReO results with different oracle distributions the CNN/DM validation set.
Systems
ROUGE-1
ROUGE-2
ROUGE-L
BERTSUM
Greedy
44.00
20.73
40.45
OREO, (L,
44.26
20.92
40.69
OREO, (,,
44.06
20.76
40.50
OREO, A,,
44.17
20.83
40,60
OREO, Ac
44.03
20.73
40.42
OREO, (1,
43.91
20.45
40.26
We devise and experiment with several oracle distributions that assign non-uniform probability to top / beams:
Annealing over Rank A, decreases the unnormalized weight from to over top beams, assuming that the oracle distribution positively correlates with hypothesis rank.
2. Annealing over Quality A, sets the unnormalized weight for a top beam Y* as R(Y"), assuming that the oracle distribution positively correlates with hypothesis evaluation score.
15

---

Published as a conference paper at ICLR 2023
3. Annealing over Locality Ag defines the locality of a hypothesis Y to be proportional the mean of sentence-level scores R), vf Y*. This is based on the assumption that hypothesis is more local if its sentences are, by themselves, high-scoring. Hypothetically, these sentences stand a higher chance to be selected by non-autoreereesive sequence labeling model which presumably focuses more their individual features rather than collective information (Zhong et al. 2020).
4. Annealing over Position Rank Ap decreases the unnormalized weight from I to over top beams which are reversely ranked by their position in the original document, assuming that oracle distribution positively correlates with document position.
Table 9 presents our results on CNN/DM validation set. As we can see, the above-mentioned distributions do not yield better results than simply adopting a uniform distribution (third row). We believe this is because hand-crafted distributions are all associated with strong assumptions, which may not be valid for real-world summarization data. However, we note that most of these distributions still manage to outperform greedy labels, showing consistent gains when considering information from multiple oracles.
Apart from heuristic oracle distributions, we could also learn a parametrized distribution from data. For instance, model with uniform oracle distribution could be trained to derive a potentially more accurate estimation from its predictions. A new set of sentence labels would be then calculated with Equation (5), and used to improve the optimization of a new model.
E
DETAILS OF EXPERIMENTAL SETTINGS
Table 10: Hyperparameters for supervised training of BERTSUM on five summarization datasets.
Monolingual
CNN/DM
XSum
Multi-News
Reddit
WikiHow
Beam size k
256
16
16
256
16
Oracle distribution t
16
16
16
32
16
Monolingual Extractive Summarization We used three GeForce RTX 2080 GPUs for model
training and bert. base in our experiments. We refer interested readers to Liu & Lapata (2019) for detailed training configurations which are identical to ours. Following Liu & Lapata (2019), ,
used the Python package pyrouge for calculating ROUGE. For our the proposed labeling methods, searched over the following (k, t) pairs: (256, 32), (256, 16), (256, 8), (32, 32), (16, 16), (8, 8).
We show the best best-performing hyperparameter combinations for each dataset Table I0 We used standard parameter settings for all experiments: ROUGE-1.555.pl -c 95 -m 1000 2 -a.
We used the datasets as preprocessed by Zhong et al. (2020) which can be accessed at: https: //githun com/ mas zww.gmingggg...........ooo.o...o http//aachhss//
Cross- Lingual Extractive Summarization In our cross-lingual experiments we used four GeForce RTX 2080 GPUs for model training with xlme. base and mbert. base. Particularly, interval embeddings (Liu & Lapata 2019) were used in MBERTSUM but not in XL.S since XLM-R removes segment embeddings from the input following RoBERTA (Liu et al. 2019). We refer readers to Jia et al. (2022) for details on training configuration; we made minimal adjustments to adapt to our training environment, i.e., no training hyperparameters were specifically optimized for our method. We set the batch size to 4, and accumulated gradients every 32 steps. Following Jia et al. (2022), we used word replacement rate of 0.5 learn cross-lingus representation alignment. We fine-tuned models on the English data with a learning rate of 2 x 10 for 50,000 optimization steps, and warm-step of 10,000. Following et al. (2022), we used the Python package spacy for non-English hypothesis/rrferencee tokenization, and pyrouge for ROUGE calculation.
Abstractive Summarization In our abstractive summarization experiments we used four GeForce RTX 2080 GPUs for model training with bart. large; the latter was also used in baseline BART system and to initialize GSUM. Due to GPU memory limitations, we set the maximum length of input document to 640 tokens (with the excess clipped) and used half float precision for efficient
16

---

Published as a conference paper at ICLR 2023
training. We used one sample for each GPU, and accumulated gradients every 32 steps. We fine-tuned all models on CNN/Daily Mail with a learning rate x 10 5 for 20,000 optimization steps, and a
warm--sep of 500. Following the evaluation steps in BART (Lewis al. 2020) and GSUM (Dou ct al. 2021), we used file2rouge to evaluate abstractive summaries.
F EXTENDED RESULTS
Table 11: Results for extractive summa-
rization on CNN/DM test set. Results for Greedy are taken from Zhong et (2020)
Table 12: Results BERTSUM with local labeling on CNN/DM test set.
Systems
R-1
R-2
R-L
Systems
R-1
R-2
R-L
MATCUSUM (Bert-based)
Local (soft)
42.45
19.55
38.75
Greedy 44.22
20.62
40.38
Local (top3)
42.59
19.67
38.87
OREO 44.32
20.66
40.51
Local (top5)
42.39
19.47
38.67
MATCHSUM (Roberta-based)
OREO
43.58
20.43
39.96
Greedy 44.41
20.86
40.55
OREO
44.49
20.84
40.63
Supervised Extractive Summarization We next investigate whether OREO can further improve performance when integrated with MATCHSUM (Zhong et al. 2020). To provide proof of concept, we further built MATCHSUM + OREO, using BERTSUM (OREO) as sentence ranker at inference. We experimented with two MATCHSUM versions (based on either BERT or RoBERTA), and show the results on CNN/DM in Table Note that MATCHSUM+ O ORFO uses the off-the-shelf summary-level reranker from MATCHSUM, i.e., not retrained on predictions from BERTSUM trained on ORFO. As we can see, ORFO improves upon vanilla MATCHSUM simply account of introducing higher- quality candidate sentences, and thus higher-quality candidate summaries for reranking. Retraining the summary-level reranker to avoid input distribution shift would result in additional performance gains.
We further experimented with local labeling (Cheng & Lapata 2016 and the results on CNN/DM are shown in Table [2] Local (soft) labels sentences with max-min scaled ROUGE scores (which are soft). In contrast, top3 and top5 create discrete scores via assigning positive labels to the top3 and top5 sentences, and all other sentences are given negative labels. As we can see, local labeling schemes are inferior to OREO, underscoring the need to exploit summary-leve information for sentence annotation.
Table 13: Cross- zero-shot performance on test sets of MLSum in ROUGE-L.
Systems
De
Es
Fr
Ru
Tr
AVG
MBERTSUM
Greedy
22.68
20.44
22.70
8.71
27.89
20.48
Beam
28.36
20.55
22.74
9.30
29.38
22.07
OREO
29.13
20.62
22.82
9.13
30.78
22.50
Cross-Lingual Extractive Summarization further initialize BERTSUM with mBERT (Devlin et al. 2019). As we can in Table [3] MBERTSUM finetuned with greedy labels shows inferior performance across languages. Nevertheless, OREO leads to substantial performance gains on both German and Turkish (we observe similar trend when BERTSUM initialized with XLM-R).
17

---

Published as a conference paper at ICLR 2023
Table 14: Results for abstractive summarization on CNN/DM
Systems
ROUGE-1
ROUGE-2
ROUGE-L
GSUM trained with greedy ORACLE guidance
Greedy 44.40
21.52
41.23
Beam 44.41
21.55
41.26
OREO
44.48
21.60
41.33
GSUM trained with OREO ORACLE guidance
Greedy 44.66
21.72
41.43
Beam 44.68
21.74
41.47
ORFO
44.81
21.83
41.60
Supervised Abstractive Summarization We show extended abstractive results in Table 14 The first block shows performance of vamilla GSUM (Dou et (021)) which uses greedy extractive oracles guidance during training. During inference, compare three types of extractive guidance produced by BERTSUM trained with greedy, beam, and OReO labels. Despite the training-testing discrepancy in guidance, extractive guidance with OReO labels helps generate better downstream abstractive summaries.
To further validate the effectiveness of OREO the optimization of GSUM, trained GSUM with OREO ORACLE, and the results are shown the second block. As we can see, adopting OREO guidance during training further boosts system performance, while the system using ORBO for both training and testing achieves the best results, i.e., 0.37 ROUGE-L improvement over GSUM.
Table 15:
Performance on four recently proposed metries for summary evaluation (CNN/DM test set). Results shown for extractive (BERTSUM) and abstractive (GSUM) summarization.
Systems
ROUGE-WE WE
MOVERSCORE
BERTSCORE
SUMMAQA
BERTSUM
Greedy
33.04
27.66
87.72
27.13
Beam
33.09
27.72
87.71
26.91
OREO
33.41
27.94
87.77
27.15
GSUM
Greedy 34.72
29.17
88.83
32.11
Beam
34.75
29.20
88.83
32.11
OREO
35.06
29.49
88.89
32.20
Performance on Other Metries Despite being popular in summarization, ROUGE has its weak- in summary evaluation. Therefore, further evaluated the generated summaries CNN/DM using other recently proposed metrics: ROUGE-WE (Ng & Abrecht, (2015), MOVERSCORE (Zhao)
al. 2019). BERTSCORE (Zhang et al. 2020b), and SUMMAQA (Scialom al. 2019). We show
the results in Table
ROUGE- WE extends ROUGE using soft lexical matching based the cosine similarity of Word2Vec. We report the average of N-gram results (N = (1,23)) for ROUGE- WE. BERTSCORE computes similarity scores between reference tokens and generated tokens but with BERT embeddings. MOVER- SCORE uses Word Mover's Distance to measure the semantic distance between reference and hypothesis summary. SUMMQA QA-style metric which was initially proposed to generate questions from source documents. Following the evaluation of PaPallllmppl & Lapata (2022), we apply the question generation model to reference summaries obtain more relevant and informative questions. As we can see, OREO achieves better performance across metrics for both extractive and abstractive summarization.
18

---

Published as a conference paper at ICLR 2023
G
SYSTEM OUTPUT
The largest single high-definition map of mysterious dark matter has been produced. It is the first in series of maps of the cosmos that will eventually allow 3D view of dark matter across one eighth of the night sky. And the map should allow astronomers to study how galaxies formed in the universe. University of Manchester researchers have revealed an HD dark matter map (shown). It shows clumps of mystery particles across 0.4 per cent of the sky. The goal is to eventually map 12.5 per cent over five years. Red here shows more dark matter, and blue shows less. The moon is shown top left for scale. A team from the University of Manchester, led by Dr Sarah Bridle, has spent the past two years measuring the shapes of galaxies used to construct the map. And the map was released today at the April meeting of the American Physical Society in Baltimore, Maryland. was created using data captured by the Dark Energy Camera, a 570-megapixel imaging device that is the primary instrument for the Dark Energy Survey (DES) led by Fermilab in the US...
Document:
Reference Summary: University of Manchester researchers reveal HD dark matter map. It shows clumps of mystery particles across 0.4 per cent of the sky. The goal is to eventually map 12.5 per cent over five years. Results could help reveal how galaxies form in the universe.
Extractive Summarization with BERTSUM
Greedy: University of Manchester researchers have revealed an HD dark matter map (shown). It shows clumps of mystery particles across 0.4 per cent of the sky. is the first a series of maps of the cosmos that will eventually allow 3d view of dark matter across one eighth of the night sky.
Beam: University of Manchester researchers have revealed an HD dark matter map (shown). The goal is to eventually map 12.5 per cent over five years. the first in series of maps of the cosmos that will eventually allow a 3d view of dark matter across one eighth of the night sky.
OREO: University of Manchester researchers have revealed an HD dark matter map (shown). It shows clumps of mystery particles across 0.4 per cent of the sky. The goal is to eventually map 12.5 per cent over five years.
Abstractive Summarization with GSUM
Greedy: University of Manchester researchers have revealed an HD dark matter map. It shows clumps of mystery particles across 0.4 per cent of the sky. The goal is to eventually map 12.5 per cent over five years. It is the first in a series of maps of the cosmos that will eventually allow a 3D view of dark matter across eighth of the night sky. And the map should allow astronomers to study how galaxies formed in the universe.
Beam: University of Manchester researchers have revealed an HD dark matter map. It shows clumps of mystery particles across 0.4 per cent of the sky. The goal is to eventually map 12.5 per cent over five years. It is the first in a series of maps of the cosmos that will eventually allow a 3D view of dark matter across one eighth of the night sky. And the map should allow astronomers to study how galaxies formed in the universe.
OREO: University of Manchester researchers have revealed an HD dark matter map of the cosmos. It shows clumps of mystery particles across 0.4 per cent of the sky. Red here shows more dark matter, and blue shows less. The goal is to eventually map 12.5 per cent over five years. And the map should allow astronomers to study how galaxies formed in the universe.
Table 16:
Examples of system output the CNN/DM test set. We illustrate differences among labeling algorithms with a sentence from the reference summary labeled in red. BERTSUM trained with OREO labels includes the sentence in its extract. In contrast, Greedy selects suboptimal, verbose sentence highlighted in blue, potentially due to its position in the beginning of the original document (lead bias). The Beam extract includes both sentences and is therefore most redundant. Using these extracts as inference guidance, GSUM creates abstractive summaries which for Greedy and Beam are identically verbose, while OREO summary is more concise.
19

---

Published as a conference paper at ICLR 2023
H
BOUND PRESERVATION
2
J
3
Beams
Ormax
Beam-LS
Top Beam
1
[60]
Sentence Rank
2
[45]
[60]
[0.9]
1
Bean Bank
X
7
2
3
[50]
<S>
2
[40]
2
[50]
[0.1]
3
[40]
[20]
3
[35]
[50]
3
[0.1]
Figure 5: Illustration of beam search (left; early stopped at j 2), ranked beams (middle), and ranked sentences (right). We show sentences (in squares) and their scores (in brackets) under different labeling algorithms. For OREO, there does not exist a y that halves the ranked list in a way that the top half is identical the sentences selected by the top beam.
20


Reviews:
[{"review_id": "ib7NkPFXtn", "reviewer": "Reviewer_perC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel way of extracting silver-standard extractive summaries (Oracle) from gold-standard abstractive summaries which they call “OREO”. Earlier methods mostly relied on the Greedy labelling method, which is sub-optimal and deterministic. The authors propose to instead generate the final oracle summaries by soft labelling so that multiple possible extractive oracles (obtained by beam-search) are considered according to their probability distribution; which leads to the final oracle summary being non-deterministic. \nThe authors have then performed several experiments to show the robustness of their model in different settings - supervised, zero-shot cross-domain, zero-shot cross-lingual, supervised abstractive, etc. The authors have experimented on several datasets, using the BERTSum model (and GSum model for abstractive setting). They show that models trained on OREO perform better than greedy and simple-beam search labels in most cases and generalize better.", "summary_of_the_review": "While the proposed method is simple to implement and is shown to be robust by several experiments, my main problem is that only BERTSum model was used to evaluate the different labelling strategies. Sections 5.5 and 5.6 are also questionable. Even though the method performs better than others, in most cases the increase in performance is quite small.", "strength_and_weaknesses": "Strengths\nThe problem statement is very interesting.\nProposed method is simple to implement and doesn’t require architectural changes for existing models\nExtensive set of experiments on different settings and on multiple datasets performed, showing the method leads to better-performing models\nWeaknesses and suggestions\nOnly the BERTSum model was used for comparison between labelling strategies. Performing similar experiments with some other models would be better, and these models need not be very complex (even some older architectures such as SummaRuNNer [Nallapati et al., 2017] may be used). \nAnother simple labelling strategy that may be a baseline may be to just get the top-K ROUGE sentences from the document individually for every sentence in the original reference summary. [Cheng & Lapata (2016)]\nIn Zero shot multi-lingual setting XLM-base has been used when the given SOTA uses XLM-Large, but then the authors claim the SOTA performs better because of the more complex architecture.\nSome equations are a bit difficult to understand, and it will be good if the terms are defined properly and described in simpler text. \nThe purpose of Section 5.5 is not very clear, and the results do not seem too interesting either. It is also not clear how exactly the ASK metric measures the extent to which a model can attain summary relevant knowledge. This space may be better used for discussing experiments using other models.\nIn many of the results, only a very small improvement in ROUGE scores have been achieved over other labelling strategies (which is kind of expected with ROUGE scores). It is also known that ROUGE scores are probably not a very good way of measuring quality of summaries. Thus it is not very evident if the OREO strategy actually leads to better summaries. It would perhaps be better if some survey analyses were performed with the outputs from models trained on different labelling strategies.\n\nOther Comments:\nThere is some discrepancy with the caption and data in Table 1, it seems the authors talk about sentences 1 & 4 (not 2 & 4) but then the OREO does not provide very high score (relatively) to sentence 1.\nIn line 8 of Algorithm 1, the l’ on the right hand side will probably be x.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UFUgMCx8tU", "reviewer": "Reviewer_mPYc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel labeling scheme for extractive text summarization. Extractive summarization datasets are most often derived from abstractive datasets using a greedy labeling scheme, providing a single extractive reference per document. Greedy multi-hot labels cause sparsity/under-fitting issues so the authors propose a soft-labeling scheme and show it is equivalent to incorporating multiple hypotheses in a non-autoregressive labeling model. The soft-labeling oracle is implemented using beam search. The authors then show improvements over greedy and beam labeling in several settings, although the model itself is not state-of-the-art.", "summary_of_the_review": "The work is novel to the best of my knowledge but its significance is likely to be limited given:\n- recent improvements to abstractive summarization \n- its complexity over labeling schemes that are simpler to implement\n- it is not state-of-the-art\n\nThat said, the work is overall solid and well theorized.", "strength_and_weaknesses": "Overall, this paper is well written and well structured. Although the contribution is fairly simple, it is well theorized. Extensive experiments are also provided.\n\nThis work is likely to have little practical use given the additional complexity of the proposed scheme and its fairly limited experimental improvements. But it does provide interesting insights and may trigger further research.\n\nSuggestions/Comments:\n- can improvements that are statistically significant be added to the result tables? Many improvements seem limited; are they statistically significant?\n- since the method does not yield state-of-the-art results, it would be nice to better explain summary reranking and MatchSum (e.g., how much more complicated/computationally expensive these are).\n- could MatchSum be added as a baseline for the cross-domain experiments?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IsnNMEZDAx", "reviewer": "Reviewer_Sdw2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses two common problems in pseudo-label (or oracle extract) generation for extractive summarization: 1) greedy or beam search does not always lead to a globally optimal solution, and 2) a pseudo-label is a deterministic training target that lacks calibration. \nThe authors then propose to leverage a series of pseudo-summaries generated from beam search to generate sentence-wise soft labels used for extractive summarization training.\n", "summary_of_the_review": "This paper presents a sentence labeling algorithm creating soft labels for extractive text summarization, with clear motivation, novel technical design, extensive evaluation,  and theoretical analysis. ", "strength_and_weaknesses": "**Strengths**\n- The identified problem is significant in extractive summarization, and the motivation is clear.\n- The idea of using soft labels for sentence-level labels is novel.\n- Extensive experiments and thorough analyses demonstrate the superiority of the proposed method in many summarization scenarios across domains and languages in supervised and zero-shot settings, with \n- A theoretical proof is provided, showing that maximizing oracle expectation is equivalent to the proposed non-deterministic objective for extractive summarization.\n\n**Weaknesses**\n- The performance improvements look marginal in some settings, and more sophisticated baselines can be considered.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster