Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees
Authors: Swarnadeep Saha
Shiyue Zhang
Peter Hase
Mohit Bansal
Venue: ICLR 2023 poster (2023)

Abstract: Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales by only highlighting parts of the source document. To this end, we propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence from the source document. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence (root node) to the document sentences (leaf nodes) from which it is derived, with the connecting nodes containing intermediate generated sentences. Edges represent different modular operations involved in summarization such as sentence fusion, compression, and paraphrasing. We first propose an efficient best-first search method over neural modules, SP-Search that identifies SPs for human summaries by directly optimizing for ROUGE scores. Next, using these programs as automatic supervision, we propose seq2seq models that generate Summarization Programs, which are then executed to obtain final summaries. We demonstrate that SP-Search effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior. We also conduct a simulation study to show that Summarization Programs improve the interpretability of summarization models by allowing humans to better simulate model reasoning. Summarization Programs constitute a promising step toward interpretable and modular abstractive summarization, a complex task previously addressed primarily through blackbox end-to-end neural systems.

Published as a conference paper at ICLR 2023
SUMMARIZATION PROGRAMS: INTERPRETABLE ABSTRACTIVE SUMMARIZATION WITH NEURAL MODULAR TREES
Swarnadeep Saha, Shiyue Zhang, Peter Hase, Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill
(swarns rna, shi yue, peter, mbansa 1 jcss uuc. edu
ABSTRACT
Current abstractive summarization models either suffer from a lack of clear in- terpretability provide incomplete rationales by only highlighting parts of the source document. To this end, we propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees, each encoding the step-by-step generative process of an abstractive summary sen- tence from the source document. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence (root node) the document sentences (leaf nodes) from which it is derived, with the connecting nodes containing intermediate generated sentences. Edges rep- resent different modular operations involved summarization such as sentence fusion, compression, and paraphrasing. We first propose an efficient best-first search method over neural modules, SP-SEARCH that identifies SPs for human summaries by directly optimizing for ROUGE scores. Next, using these programs as automatic supervision, we propose seq2seq models that generate Summariza- tion Programs, which are then executed to obtain final summaries. We demon- strate that SP-SEARCH effectively represents the generative process behind hu- man summaries using modules that are typically faithful to their intended behav- ior. We also conduct a simulation study to show that Summarization Programs improve the interpretability of summarization models by allowing humans bet- ter simulate model reasoning. Summarization Programs constitute a promising step toward interpretable and modular abstractive summarization, a complex task previously addressed primarily through blackbox end-to-end neural systems."
INTRODUCTION
Progress in pre-trained language models has led to state-of-the-art abstractive summarization mod- els capable of generating highly fluent and concise summaries (Lewis al., 2020; Zhang et al., 2020; Raffel et al., 2020). Abstractive summarization models do not suffer from the restrictive na- ture of extractive summarization systems that only copy parts of the source document. However, their ability to generate non-factual content (Cao et al., 2018; Maynez et al., 2020) and their lack of clear interpretability makes it harder to debug their errors and deploy them in real-world scenar- ios. Towards interpretable summarization models, Jing & McKeown (1999; 2000) show that human summaries typically follow a cut-and-paste process, and propose a modular architecture involving separate operations that perform sentence extraction, sentence reduction, sentence fusion, etc. Most recent efforts on explainable abstractive summarization follow an extractive-absstrative framework that only provides supporting evidence 'rationales' for the summary (Hsu et al., 2018; Gehrmann et al., 2018; Liu Lapata, 2019; Zhao et al., 2020; Li et al., 2021). These models highlight words or sentences from the source document but not able explicitly capture the generative process of summary, i.e., the reasoning steps performed in order to generate each summary sentence from the source document sentence(s), like sentence compression, fusion, etc. In this work, we seek to bridge
SSupporting code available httph: http//gtttttttttt// com/ swa rww.ub/aHabbaeee Summa ri z PoPPrograss.

---

Published as a conference paper at ICLR 2023
Summarization Program
Summary: Prosecutar Brice Rabin recovered from the crash according Paris Match.
of the cresh 9625. The
S4: Protectter Diice Rozh eaye he not zeane of
Pusion
The prosectoor raading inverdpatio ints Undo on viko lockege.
Prosuater thes route vayy bis is umane of
BZ: The vikoo was recooned lian authone wi
Compression
Pusion
Parophrase
Cerrarwing: right 5525 irs aaud (huch enday thal to wea ainar
Faal so far vidion vere used in dran
02: Ports March Bild reported that 180oo recsvrrd foor phase wencloge dit..
Figure 1: Example of Summarization Program showing the generative process of two summary sentences (marked with labels S1" and S2' in yellow) from the three document sentences (marked with labels DI, D2, and D3 in blue) using compression, paraphrase and fusion neural modules. Edges directed from leaves to roots, and the intermediate generations are labeled with I1 and 12.
this gap by proposing novel Summarization Progrum framework for explaining abstractive sum- marization, that views summarization as systematic reasoning process over document sentences.
A Summarization Program (SP) is a modular executable program that consists of an (ordered) list of binary trees, each encoding the generative process of an abstractive summary sentence from the source document ($3). Fig. shows an example. The leaves an SP are the source document sentences (typically, only small subset that are relevant for generating the summary). Each inter- mediate node represents a generation from a neural module (shown with labeled edges) which are then composed to derive the final summary sentences the roots of the trees. We develop three neu- ral modules for building SPs sentence compression, paraphrasing, and fusion (Jing & McKeown, 1999: 2000) that finetune a pre-trained language model on task-specific data. We evaluate Summa- rization Programs by asking the following two research questions (see Fig. 2 for an overview).
RQ1. Given a wrrtten abstractive summary, can we develop an algorithm for identifying Summarization Program that effectively represents the generative process of the summary? RQ2. Using the SPs identified in RQI as supervision, can we develop models that generate Sum- marization Programs interpretable intermediate representations for generating summuries?
We answer the first research question by automatically identifying SPs for human summaries ($4). Specifically, we develop an efficient best-first search algorithm, SP-SEARCH that iteratively applies different neural modules a set of extracted document sentences in order to generate newer sen- tences, such that the ROUGE (Lin, 2004) score with respect to the gold summary is maximized by these new sentences. SP-SEARCH achieves efficiency through important design choices including maintaining priority queue that scores, ranks, and prunes intermediate generations (appendix A). We conduct experiments on two English single document summarization datasets, CNN/DailyMail (Hermann al., 2015) and XSum (Narayan et al., 2018) to show that SP-SEARCH outputs SPs that effectively reproduce human summaries, significantly outperforming several baselines ($6. 1). More- over, human evaluation shows that our neural modules are highly faithful, performing operations they are supposed to and also generating outputs that are mostly factual to their inputs ($6.2).
We leverage SP-SEARCH to obtain oracle programs for human summaries that also serve as super- vision for answering our second research question. In particular, we propose two seq2seq models for Summarization Program generation from source document ($5, Fig. 2). In our first Extract-and- Build SP generation model, an extractive summarization model first selects a set of document sen- tences, which are then passed another program--generaing generating model. In our second Joint SP genera- tion model, sentence extraction and SP generation happen as part of a single model. We obtain initial promising results and while state-of-the-arr end2end models demonstrate better ROUGE scores, our oracle SP-SEARCH results indicate significant room for improvement in future work ($6.3).
"Neural module faithfulness' refers to whether the modules perform their expected operations. This is different from 'summary faithfulness' that evaluates whether summary contains only factual claims from the source document. The latter will be referred 'summary factuality'.
2

---

Published as a conference paper at ICLR 2023
To evaluate whether SPs improve the interpretability of summarization models, conduct a small- scale simulation study (Doshi-Velle & Kim, 2017: Hase & Bansal, 2020; Zhou et al., 2022) where ask humans to simulate model's reasoning by writing programs for model summaries ($6.4). We observe that after seeing model SPs, humans are able to better predict them for unseen samples, such that the executed summaries match more elosely with the model summaries. Our contributions are:
We introduce the Summarization Program, an interpretable modular framework for explainable abstractive summarization. Summarization Program consists of an ordered list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence through the use of different neural modules (sentence compression, paraphrasing, and fusion).
We propose an efficient best-first search method, SP- SEARCH that identifies Summarization Pro- grams for human-written summaries, obtaining a high ROUGE-2 of 40 on the CNN/DailyMail dataset with neural modules that are highly faithful to their intended behavior.
We present initial Summarization Program generation models that generate SPs from a source doc- ument, which are then executed to obtain final summaries. We demonstrate that SPs improve the interpretability of summarization models by allowing humans to better simulate model behavior.
2 RELATED WORK
Pre-trained Models for Summarization. State-of-the-art pre-trained language models like BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and PEGASUS (Zhang et al., 2020) generate summaries in an end-to-end manner. Lacking complete mechanistic understanding of transformers (Elhage et al., 2021), it remains difficult to understand the reasoning process behind the generated summary. In particular, we cannot determine whether generation follows process similar to that of humans, e.g., using operations like selection of important information, abstraction through vocab- ulary generalization, sentence fusion, etc (Kintsch & van Dijk, 1978; Brown & Day, 1983; Jing & McKeown, 1999; 2000). Moreover, it is hard to determine the source of errors in summaries without knowing the reasoning path to the summary. Recent work on interpretable summarization models highlight words or sentences as rationales for the generated summaries (Hsu et al., 2018; 8; Gehrmann et al., 2018; Liu & Lapata, 2019; Liu et al., 2019; Zhao et al., 2020; Li et al., 2021), but textual highlights do not provide a complete explanation of the summary generation process. Thus, Sum- marization Program is step towards more explicit interpretability in abstractive summarization.
Neural Module Networks. Our work also draws inspiration from Neural Module Networks (NMN) that execute programs as learned functions composed of neural modules (Andreas et al., 2016; Hu et al., 2018; Jiang & Bansal, 2019; Gupta et al., 2020; Subramanian et al., 2020; Saha et al., 202 1a; Le et al., 2022). Typically, the modules in an NMN provide attention- explanations whose interpretability has been debated (Serrano & Smith, 2019; Wiegreffe & Pinter, 2019). Khot et al. (2021) alternatively propose Text Modular Networks that decompose multi-hop questions into sub- questions to be solved by simpler QA models. We also follow this text-in text-out paradigm for our modules but equipped to perform diverse sentence-level operations in document summarization. We evaluate our modules for 'neural module faithfulness' (Subramanian et al., 2020) to demonstrate that the SPs mostly provide a faithful interpretation of the generated summaries.
Multi-step Reasoning over Text. Step-by-step reasoning has received much interest as way to explain various reasoning tasks including QA (Dalvi et al., 2021; Ribeiro et al., 2022) and natural language deduction (Bostrom et al., 2022; Saha et al., 2020; Gontier et al., 2020; Saha et al., 2021b; Tafjord et al., 2021). Summarization Programs similarly encode the reasoning steps in a summary generation process. Our SP-SEARCH method follows forward chaining method that tries reach hypothesis from set of premises (Tafjord et al., 2021; Bostrom et al., 2022), unlike backward chain- ing methods that do the opposite (Gontier et al., 2020; Arabshahi et al., 2021; Kalyanpur al., 2022; flain-of-thoughh prompting (Nye et al., 2021;
Dalvi et al., 2022). In another recent line of work, Wei al., 2022; Wang et al., 2022) encourages LMs to generate intermediate reasoning steps before producing a final answer to problem. However, the lack of explicit chaining between the reasoning steps and the final output may compromise the faithfulness been evaluated as explanations of model behavior per se. Recent work has explored ways to force these reasoning steps to be more like deductive proofs of the final answer (Creswell & Shanahan, 2022) instead use generations from a larger language model as silver supervision for smaller pipeline student model (Eisenstein et al., 2022). In contrast, we aim to develop a method whose ra- tionales (our summarization programs) exactly describe the reasoning process of the overall system
those steps, which have also not yet
3

---

Published as a conference paper at ICLR 2023
and we explicitly evaluate their faithfulness. Specifically, we generate silver programs by search algorithm that tries to emulate the human summaries and then train models that generate programs which are further executed to obtain final summaries (and evaluated via a simulatability study).
3 SUMMARIZATION PROGRAM
We assume that we have a document D - {D consisting of d sentences and a correspond-
ing abstractive summary 8 {S. consisting of sentences. A Summarization Program P = (Ti)i=1 is defined as an (ordered) list of s binary trees where each tree T; = (V, E,) structured representation of the generative process of each summary sentence S, E S. Fig. shows an example of an SP with two trees for two summary sentences. The set of nodes V in each tree consists of single sentences, and the edges E, are labeled with one of the neural These modules represent operations
modules m E (paraphrase(()) compression()) fusion(;, over sentences wherein compression + Y and paraphrase + Y are unary operations and fusion (X, Y) + Z is binary operation. The leaf nodes in each tree are sentences from the doc- ument D; E D, and the root is a summary sentence S E S. All other nodes are intermediates sentences generated by executing a neural module (referred to Il and 12 in Fig. 1). An edge from a node u E Vú to a node E V; labeled with the module m means that v is generated by executing m on u. The summary S is obtained by concatenating the root nodes of the trees in order. We hypothesize that the generative process of each summary sentence can be captured by composing different neural modules that operate over sentences. Following prior work on modular approaches to abstractive summarization (Jing & McKeown, 1999; 2000; Lebanoff et al., 2019; 2020b), we define the following three neural modules for building Summarization Programs.
Fusion Module. Sentence fusion in summarization combines information from multiple sen- tences (Lebanoff et al., 2020a). We finetune a BART-large model, which takes two sentences as input and outputs one fused sentence. Existing sentence fusion datasets either aim to improve discourse connections (Geva et al., 2019) or aim to fuse similar sentences from multiple docu- ments (Brook Weiss et al., 2022). Instead, we want to fuse two disparate sentences into one sen- tence, which requires the model to merge related pieces and remove unimportant information. To obtain training data for such model, we follow Lebanoff et al. (2019) to align each summary sen- tence from CNN/DailyMail with one to many similar and non-redundant document sentences. As training data, we only use examples that align one summary sentence with two source sentences.
Compression Module. The compression module generates a compressed output of a single sen- tence. It involves generating shorter sentence by preserving the essential content and the syntactic structure of the input. We finetune a BART-large model (Lewis et al., 2020) on a large parallel corpus of uncompressed and compressed sentences (Filippova & Altun, 2013).
Paraphrase Module. The paraphrase module generates a sentence that involves syntactic trans- formations or lexical paraphrasing of the input sentence. We use a publicly available PEGASUS- based (Zhang et al., 2020) paraphrase model from HuggingFace (Wolf et al., 2020).² In practice, we observe paraphrased outputs to frequently involve some compression as well, which we analyze in detail as part of the 'Neural Module Faithfulness' evaluation ($6.2).
4 RQ1: SUMMARIZATION PROGRAM SEARCH
Our first research question of interest is whether given a document D and human- summary S, we can generate a Summarization Program P that best explains the generative process of the summary (see the left part of Fig. 2). We achieve this by developing an efficient best-first search method, named SP-SEARCH, outlined in Algorithm 1 in the Appendix. Conceptually, it is similar to a forward chaining algorithm in logic programming (Russell & Norvig, 2009), in which we start from set of premises (equivalently, small set of document sentences) and iteratively apply neu- ral modules on them to generate newer deductions (equivalently, intermediate sentences) until the goal hypothesis (equivalently, the summary) is generated. SP-SEARCH processes each summary
Model available at http::/://oggggggooeeeeo/////rr/
4

---

Published as a conference paper at ICLR 2023
SP-Search
SP-Model
Searched Summary: [S11, $2)
Predicted Summary: [S1)
S1
S1
fuslon
Ausion
S2
S2
D5
11
paraptnazo
Quadon
D3
Jaskor
D3
D1
D2
supervisio
D2
D5
Pe= [bssianffusion--D11>, <02>), <D5>)]]
[compression( (DD31)
P- [lusion<DD>>, 1>, fusion(=D2>>, <D5+j]] [saraphrase(-D3-]
SP-Search
SP-Mode (Extract-and-Buld))
OR
SP SP-Model (Joint)
Algaritim
[D1, D2, D3, D5) [81, S2]
[D1. D2, D6)
D=1D1, D2, D3, D4, D5, ...]
Top-K
MatchSum
D= [D1, 02, D3, D4, ..I S= ($1,
[D1, [D1
DS, D4, D6....
Pre Pre-trained Madules
fusion
compresssion
paraphrase
Figure 2: Overview of the two RQs aimed at evaluating SPs. On the left, we show our SP-SEARCH algorithm that takes source document and human summary as imput and identifies an SP that best explains the summary (RQ1). On the right, we show our SP generation models that take document as input and output a program, which is then executed to generate the summary (RQ2). S1 and $2 denote human summary sentences while S1 and S2' represent identified or generated summaries.
sentence separately, thereby generating a unique tree for each summary sentence. We optimize the trees for their ROUGE-L (R-L)* scores with respect to the gold summary sentences.
SP-SEARCH Algorithm. SP-SEARCH operates as follows (see Algorithm 1). For each summary sentence, initializes priority queue, each element ($1, m, h) of which represents a module m, the sentences $ and 82 on which is defined (82 is empty for "compression" and 'paraphrase" modules) and the corresponding tree height h. The initial elements the queue represent possible operations only the document sentences. Next, at each step of the search, SP-SEARCH pops an element from the queue and exccutes the operation over the corresponding operands to generate new sentence. Each module generates top-5 outputs using beam search and the one with the highest R-L score is chosen. Using this new sentence, SP-SEARCH creates new elements for the priority queue by considering all potential operations involving the new sentence and other available sentences. Then, these new elements may be explored in the next steps of the search. As best-first search method, SP-SEARCH ranks the elements in the queue with the following scoring function.
Score(s1, 82, m, h) max (R- S), R-L(s2,
(1)
The scoring function considers the maximum of the R-L scores with respect to generations $1 and 82 such that if the module m is executed, it can possibly lead to a sentence that is of an even higher R-L. score than its children. Intuitively, it encourages prioritizing elements in the queue that can potentially lead to higher scores. Whenever a new sentence is generated, its R-L score with the gold summary sentence computed and accordingly, the best possible root node of the tree is updated. Upon completion of the search (when the queue becomes empty), the node with the maximum score chosen as the root node and the tree rooted that node the final tree. Since performing SP- SEARCH exhaustively can be prohibitive, it is made efficient through the following design choices.
Design Choices for Efficient SP-SEARCH. Here we discuss the choices briefly and provide more details in Appendix A. (1) Top-k Document Sentences. We rank each document sentence by comput- ing ROUGE-1 with the summary and build the SP with only the top-k document sentences as eligible source sentences. (2) Filtering Queue Elements. To limit the search space, SP-SEARCH defines cer- tain heuristics for choosing the elements to be added to the queue like a compressed sentence is not compressed again, each document sentence is used at most once in tree, etc. (3) Priority Queue and Pruning. SP-SEARCH maintains a fixed queue size and a ranked list of the elements accord- ing to Eq. such that only the top-ranked elements are kept in the queue and the rest are pruned.
WWe conducted some initial experiments to observe that optimizing for R-I. leads to more consistent im- provements across all ROUGE measures. See Appendix C for details.
5

---

Published as a conference paper at ICLR 2023
(4) Parent has higher ROUGE than children. A branch of the search is only expanded further the new generation (parent) has a higher R-L score than the source sentence(s) (children). This greedy approach ensures that every reasoning step is step closer to the summary. (5) Maximum Tree Height. SP-SEARCH chooses a maximum height of the trees. (6) Batching Module Executions. SP-SEARCH executes all operations in the queue together by batching at each depth of the search.
RQ2: ABSTRACTIVE SUMMARIZATION VIA SUMMARIZATION PROGRAMS
Given that we have proposed SP-SEARCH for identifying SPs for human summaries, we now want to leverage the algorithm to generate training data for building SP generation models, as part of RQ2. In particular, we use SP-SEARCII to identify Summarization Programs for all training samples in the Mall dataset (Hermann et al., 2015). Our second research question now asks whether these identified SPs be used as supervision for developing abstractive summarization models via the generation of SPs (see the right part of Fig. 2). Hence, we define a supervised learning problem f: + P that generates Summarization Program P from document D.
Summarization Program Generation Models. We propose two initial models for our task. The SPs encoded strings such that they can be generated by seq2seq generative model. We first label each sentence in the document with unique identifiers like '<DI>' '<D2>>, etc. The Summarization Program is then represented using a nested bracketing structure composed of the modules and the sentence identifiers as shown in Fig. 2. For our first Extract-And-PR rization Program Generation Model, we hypothesize that a document can contain a large number of sentences but only small fraction of those are typically useful for generating summary. Given a training corpus of samples (Dx, P) where De represents the set of document sentences on which the SP P is built, finetune BART such that it can generate Summarization Program given small number of document sentences. During inference, given a document, we first extract a set of relevant sentences using state-of-the-a art extractive summarization model, MatchSum (Zhong et al., 2020), which are then used to generate the program. While this model simplifies the learning problem by separating out the sentence extraction phase from the program building phase, relevant sentence extraction is an essential first step towards generating good SPs. Our next Joint Sum- marization Program Generation Model aims solve sentence extraction and SP generation as part of a single model. We finetune BART on the parallel corpus of document and Summarization Programs and the model learns directly generate the program given an entire document.
Summa-
Summary Generation from an SP. We parse and execute a generated SP through iterative inference over the neural modules to obtain the final summary. We check for well-formedness of the SP by ensuring that the generated sequence (1) has balanced parentheses, (2) does not contain any out-of- vocabuauyl token (besides module names, sentence identifiers, and parentheses), and (3) has consistent number of operands for an operation. During inference, we decode up to top-k SPs using beam search and execute the first well-formed one. When none of the top-k outputs well-formed (<1% of the samples), we generate the corresponding extractive summary.
6 EXPERIMENTS
We experiment with two English single-document summarization datasets, CNN/DailyMMail (Her- mann al., 2015) and XSum (Narayan et al., 2018). We discuss CNN/DM results below and point readers to Appendix F for XSum results.
6.1 RQI RESULTS: CAN SP-SEARCH REPRESENT THE SUMMARIZATION PROCESS?
Our first set of experiments is aimed at answering our first RQ (as discussed in $4) where the goal is to analyze how well the Summarization Programs identified by SP-SEARCH represent the human- written summarics. We consider two variants of SP-SEARCH - (1) SP-SEARCH Top-1 in which each module generates only one output, and (2) SP-SEARCH where each module generates Top-5 outputs (via beam search) and one with the best R-L is chosen.
Evaluation Metrics. We use ROUGE to measure similarity between the SP-SEARCH summaries and the human summaries. We also compute factuality of the SP-SEARCH summaries with respect of the gold summaries using SOTA factuality metric, QuestFEval (Scialom et al., 2021). Note that

---

Published as a conference paper at ICLR 2023
while factuality is typically measured against the source document, RQ1 requires to evaluate how factual the SP-SEARCH summary is to the reference summary (that the SPs are aiming to reproduce).
Experimental Design. We conduct experiments with 1000 random validation samples." 2 We set the number of initial document sentences (k) to 4, the maximum queue size (Q) to 20, maximum height of the trees (H) 2 and the decoding strategy for each module to beam search with beam size of 5. Our choice of hypeparameters is based on a comprehensive study of the trade-off between ROUGE scores and average search time, as presented in Appendix C. As baselines, we consider the following extractive and abstractive oracles. (1) Random SP. For each summary sentence in the gold summary, we randomly sample tree structure (from those obtained through SP-SEARCH) and execute it with randomly chosen leaf nodes from the set of top-k sentences. (2) Top-4 Sentences. Our next baseline is an extractive summarization system with the top-4 document sentences ranked according to ROUGE-1 scores with the gold summary. (3) BART-Oracle. We also compare with two BART- models. In one, we generate top-10 summaries using beam search. In another, we sample 10 summaries with multinomial sampling and the summary with the highest ROUGE-L score with the gold summary is chosen. (4) SP-SEARCH Leaves. Our final baseline is an SP- SEARCH variant where only consider the leaf nodes (document sentences). This represents an extractive summary with upto top-4 sentences. Since the trees are built on top of these nodes, a better ROUGE score for SP-SEARCH will indicate that these leaf nodes document sentences being composed in a way such that the resultant summary is more similar to the human summary. (5) Ablations of Modules. These are SP-SEARCH variants where each module removed.
Results. As shown in Table 1, SP-SEARCH generates summaries that better emulate gold summaries by obtaining an R-2 of 40, a sig- nificant 20 points improvement (p < 0.001) over the SP-SRARCH leaves. The random SP baseline uses the same pre-trained modules as SP-SEARCH but only obtains an R-2 of 15, demonstrating that arbitrary composition of these modules is much less effective than the searched programs. The BART-oracle models exhibit less diversity between the generations, as reflected by their lower ROUGE scores. This demonstrates the utility of a tree structure in an SP. Finally, we observe that each of our neural modules plays significant role in constructing more accurate SPs. Removing the fusion module leads to the highest drop in ROUGE which suggests that fusing information from multiple sentences is one of the core operations in human summaries. SP-SEARCH also obtains the best QuestlEval score, even though it does not specifically optimize for it, suggesting that the summaries human summaries (as measured by ROUGE) but also have high factual overlap with the human summaries. In summary, SP is effective way of representing the generative process of abstractive summaries, outperforming random SPs, extractive oracles and unstructured abstractive oracles.
R1 / R2/ / RL/ / RLsum
QEval
Random SP
37.54 / 15.18 / 25.65 / 33.42
40.50
Top-4 Sentences
35.55 / 17.77 / 25.02 32.42
50.56
BART-Oracle (Beam)
40.07 / 19.57 32.04 37. 06
43.25
BART-Oracle (Sample)
42.07 / 19.44/ 33.56 99.04
45.09
SP-SEARCH Leaves
39.15 20.01 / 27.94 / 35.58
50.53
SP-SEARCH Top-1
56.92 / 34.34/ 52.15 54.80
$2.32
SP-SEARCH
61.88 / 40.11 / 58.46/ 60.32
54.48
-Paraprraee
60.08 / 38.28/ / 56.36/ 58.29
53.80
Compression
59.50 / 37 .4// 55.45 / 57.66
53.85
Fusion
53.73 / 31.30/ / 48.90 51.73
48.78
Table 1: RQ1 101-ROOT ROUGE scores for SP-SEARCH summaries and oracle extractive, abstractive baselines. Final column compares the factuality of the summaries with respect to the gold summaries through a factuality metric. QuestEval.
not only similar to the
6.2 RQ1 RESULTS: NEURAL MODULE FAITHFULNESS EVALUATION
Higher ROUGE and factuality scores obtained by SP-SEARCH with respect to the refer- ence summaries do not guarantee that SPs provide faithful interpretation of the gen- erated summaries.
$6.1 specifically evaluates the final summaries but not the interme- diate steps used to generate them. In fact, prior work on Neural Module Networks has shown that faithfulness may not always be guaranteed for neural modules and that incor- rect intermediate outputs can still lead to correct final outputs (Subramanian et al., 2020).
'Due the expensive nature of SP-SEARCH, we experiment with random subset of the validation set. "SP-SEARCH can ignore certain top-4 sentences if including them do not lead to higher ROUGE scores. WWe use non-parametric bootstrap test (Efron & Tibshirami, 1994) for significance testing.
7

---

Published as a conference paper at ICLR 2023
Hence, we also evaluate the faithfulness of the three modules by conducting human study (with 2 experts having knowledge of NLP) for 20 SPs. These constitute a total of 107 samples, including 54 fusion, 25 paraphrase, and 28 compression op- erations. Table shows the results (with more details in Appendix E). The diagonal entries in the table demonstrate how often module performs its intended behavior and the high values of 0.8-0.98 suggest
Comp
Para
Fusion
Non-Factual
Comp
0.98
0.05
0.00
Para
0.68
0.90
0.04
Fusion
0.86
0.45
0.80
0.20
Table 2: RQ1 Neural Module Faithfulness evaluation of SPs. Entry (i.) shows how often a module 'i' in a row performs the operation 'j' in a column. The final column shows the fraction of non-factual outputs.
that our modules are highly faithful. Interestingly, the values in the last column demonstrate that compression and paraphrase modules almost never generate non-factual outputs while our fusion module is more prone to that (about 20%), indicating room for improvement in the fusion module.
6.3
RQ2 RESULTS: EVALUATION OF SUMMARIZATION PROGRAM GENERATION MODELS
Our next set of experiments addresses our second RQ (as discussed in 55) on evaluating Summa- rization Program Generation models in terms of their summary generation capabilities.
Experimental Design. We compare our SP generation models (Joint and Extract-and-Buile with the following baselines and oracles on the CNN/DailyMail test set. As baselines, we consider (1) SOTA Extractive Summarization Model, MatchSum (Zhong al., 2020) that we also use to extract initial document sentences for our Extract-anddBuild SP generation model, (2) SOTA Abstractive Summarization Models, BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020), and (3) Random SP Models that randomly sample the number of summary sentences from [1,2,3.4), and for each summary sentence, randomly sample and execute a tree structure with ran- domly chosen leaf nodes from all top-k document sentences. As oracles, we consider the same models introduced in $6.1 for RQ1. They provide an estimate of the upper bound of SP models.
Evaluation Metrics. Besides ROUGE, also evaluate the factuality of the generated summaries (with respect to the source document) using QuestEval that is shown to correlate well with humans.
Results. Table 3 shows the results. We observe that models obtain R-2 and R-L scores of 16 and 26 respectively. The Extract- and-Build model performs slightly better than our Joint model, possibly due to the former generating programs over good initial set of document sen- tences. Both of our models outperform the Random SP baselines, demon- strating that they learn useful patterns of which neural modules should act on which document sentences. Compared to SOTA abstractive models, our models' interpretability (as discussed in $6.4) comes at the cost of some drop in performance (about points in R-2). However, the oracle results suggest that our models provide good starting points for better SP generation models and that there is substantial room for improvement in future work (e.g., the SP-SEARCH Top-1 model obtains an R-2 of 35, leaving a large gap of 19 R-2 points to oracle SPs). We hope the community will explore and improve upon our SP generation models, as a way to make progress on the important and challenging task of developing interpretable abstractive summarization models. We observe similar trend for the QuestEval scores where
R1 RI. RLsum
QEval
MatchSum (Zhong al., 2020)
43.98 20.66 28.72 40.02
63.22
BART (Lewis et al., 2020)
44.19/21.29 31.06 41.04
59.15
PEGASUS (Zhang et al., 2020)
44.16/21.55 / 31,38 41.01
58.83
Random SP (Joint)
24.47 6.48 15.60 22.07
40.73
Random SP (Extract-and- -ui-BBild))
31.61 / 12.69 / 21.84 / 28.88
52.83
SP Model (Joint)
37.66 / 16.00 25.68 34,45
56,88
SP Model (Extract-and-Build)
37.23 / 16.40 / 26.48 18.18
55.95
Top-4 4 Sentences (Oracle) SP SEARCH Leaves (Oracle) SP-SEARCH Top-1 (Oracle)
45.35 24.60 31.30 41.92
60.00
47.28 / 25.92 32.73 43.70
59.60
58.17 / 35.31 / 51.87 55.96
54.72
SP-SEARCH (Oracle)
62.69 / 40.58 58.02 61.10
54.11
Table 3: RQ2 Comparison of our SP generation models with state- of-the-art extractive and abstractive summarization models, random SP baselines and oracle systems with our SP-SEARCH variants on the CNN/DM test set. All methods are compared based on ROUGE and a state-of-the-ar factuality metric, QuestEval (Scialom et al., 2021).
8

---

Published as a conference paper at ICLR 2023
models largely outperform the random SP baselines, while showing worse performance than BART and PEGASUS, possibly because of the cascading effect of compositional generation via SPs.
Discussion. BART and PEGASUS optimize for gold summaries while SP models optimize for SP-Search summaries. The SP models, by themselves, are simple and effective - when the generated summaries are evaluated using SP-Search summaries as targets, R-2 of 21 to SOTA models. This shows that retraining our models on even better oracle programs by incorporating more modules SP-SEARCI or further enhancing the existing ones can help close the gap to SOTA models. We hope the community will explore and improve upon our methods, as way to make progress on the challenging task of developing interpretable summarization models.
obtain comparable
6.4 RQ2 RESULTS: INTERPRETABILIT EVALUATION OF SP VIA SIMULATION STUDY
We ask if SPs improve the interpretability of summarization model. In particular, we are interested in evaluating model simulatability, a measure of whether a person can predict model behavior on new inputs Dooshi-Velez & Kim, 2017). Similar to Zhou et al. (2022), we are specifically interested in model reasoning, as represented by the programs that our models generate and execute. The primary motivation for predicting model reasoning is that it is what we want to better understand by virtue of model explanations, and simulation studies that focus on predicting final outputs do so only to show that users have good mental model of model reasoning (Hase & Bansal, 2020).
Study Design. With 3 experts (authors) having prior knowledge in NLP and explainability, we design and carry out a small-scale human study that tests whether presenting humans with Summarization Programs improves their ability to predict SPs on future unseen data. Human subjects are first provided with some test documents and corresponding model summaries (from our Explain-and-Prediin SP model) and are tasked with predicting the model's Summarization Programs. Next, in a training phase, they are shown model-generated SPs for a few training samples and are then asked again to perform the same task with the same set of test examples. The study is conducted with 10 training examples and 31 test examples. We say that an SP is more representative of the model summary if executing it with our pre-trained modules generates a summary that is closer to the model summary. This is a measure of program similarity since variance in the SPs may make graph similarity measures inappropriate (e.g., compression followed by paraphrase and vice versa may generate similar outputs).
RI /2/ / RL/ RLsum
Before Expl. After Expl.
74.37 / 62.79/ 717799711000 71.62 72.54
78.03 / 66.80 / 72.37 / 74.69
Table 4: RQ2 Comparison of mean ROUGE scores between our model sum- maries and their human simulations before and after being trained with SPs.
Results. The results are presented in Table 4. In general, high ROUGE scores suggest that, when models are forced to generate summaries via SPs, their reasoning becomes quite predictable a priori. We also see about 4 points improvement in ROUGE-2, with 61% of samples showing an improved R-2 score (statistically significant at < 0.001), 10% of samples with a drop in R-2, and remaining being ties. This suggests that SPs are potentially good explanations of model reasoning, such that humans can generalize across model reasoning patterns after being given the explanations.
7 DISCUSSION AND CONCLUSION
We proposed the Summarization Program, a novel framework for interpretable abstractive summarization. We demonstrated its effectiveness by developing SP-SEARCH that identifies summarization programs for human summaries with highly faithful neural modules and SP models that produce summaries from source documents. Two most common forms of errors in the generated SPs include (1) redundant or longer paths in an SP, and (2) fusion module generating non-factual sentences or ignoring one of its source sentences. Two other notable issues, arising out of the independence assumption of summary sentences, are (1) final summary sentences having overlapping information, and (2) incoherence between consecutive sentences. One way to improve this is to add a "coherence" module on top of the root nodes before generating the final summary. We build SPs using sentences as the fundamental content unit (nodes) due to the relative ease of defining and training neural modules on sentences and the availability of large-scal training data. Summarization may also involve other text manipulation operations that are not fully captured by modules but framework allows easy inclusion of other modules.
9

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Despite the recent success of pre-trained language models in abstractive summarization, their lack of explainability still remains a major concern and we hope that Summarization Programs prove to be an important step in bridging that gap. That said, summarization is inherently subjective task and existing summarization datasets also significantly vary in terms of their stylistic features like abstractiveness, length, and specificity (Goyal et al., 2022). Hence, more future work is needed to understand the general applicability of our neural modules and how effective they are in encoding different kinds of summaries. Broadly put, Summarization Program is complex NLP problem down into sub- sab-problems and then solving them through neural modules with- having access intermediate supervision. We fine-tune language models for building modules and language models can be prone to generating unwanted content (Weidinger et al., 2021). How- ever, since each module is focused on one particular skill, that should help limit the negative impact and provide users with more control, compared to end-to-end models. Summarization Programs are also specifically designed to trace the origin of any toxie or hallucinated content the generated summaries.
study for breaking
REPRODUCIBILITY STATEMENT
To encourage reproducibility, we make our source code publicly available. The details of our SP- SEARCH algorithm are shown in Algorithm 1. The hyperparameters for RQI and RQ2 are dis- cussed in Appendix C and Appendix D respectively. The CNN//ailyMail and XSum datasets are also publicly available https: htts::///hggigaaceee co /e///aaaasettt cnn dai 1 yma i1 and https:/ / //uggi ng face. co/ cco/dataseess xsum respsteeiiiy
ACKNOWLEDGEMENTS
We thank the reviewers for their valuable feedback. We also thank Archiki Prasad and Prateek Yadav for their help with neural module faithfulness annotations, and David Wan for help with factuality evaluation. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-21 12635, Google Ph.D. Fellowship. and Bloomberg Data Science Ph.D. Fellowship.
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 39-48, 2016.
URL httph:tpp///.cc /ttp///o.... htt./ooo.orgooooo..oooooooooooo. 10. www.11/9/.....22 http///PP............
Forough Arabshahi, Jennifer Lee, Mikayla Gawarecki, Kathryn Mazaitis, Amos Azaria, and Tom Mitchell. Conversational neuro-symbolic commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4902-4911, 2021. URL https:/ /ttps///axxvvvooo
10022.
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduction through search over statement compositions. Findings of EMNLP, 2022. URL https:// arxiv. or.oo/gggao. abs/ 2201 06028.
Daniela Brook Weiss, Paul Roit, Ori Ernst, and Ido Dagan. Extending multi-text sentence fu- sion resources via pyramid annotations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 1854-1860, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 18653/v 1/2022. maacl-mann.. 135. URL https: / haaaaaaanntttttoooo naacl-maine 135.
org/2022.
Ann L Brown and Jeanne D Day. Macrorules for summarizing texts: The development of ex- pertise. Journal of verbal learning and verbal behavior, 22(1):1-14, 1983. URL https:
htwww.sccieeet...cc... rect http://ewwcc........a.aaaaaaaaaa
10

---

Published as a conference paper at ICLR 2023
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful the original: Fact aware neural ab- stractive summarization. In thirty-second AAAI conference on artificial intelligence, 2018, URL
htttps://aaai wwt.absg///aooooooot.../e...ere/ 1711.04434......
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. arXiv preprint arXiv:2208..422: 2022. URL https: //apxiv///xxi..///////aa owg.or////o.o..o pdd/ 2208 14271.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7358-7370, 2021. URL /ttppp://ariiv. org/ abs 2aass/110 /2104. 08661.
Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems. arXiv preprint wrXiv:2204.13077 2022. URL https: /ttp:////xxvvvvi abs 012204113770
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning- arXiv preprint 1702:000000808, 2017. URL httpt:::: wwtpx/iwwwo./a///o/eeababaae 08608.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from a pretrained language model. arXiv preprint arXiv:2210.024498, 2022. URL https: /ttp//aavvvvv//////eeeeeeeeeeesa org/ / 2210. 02496. pdf.
Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, Tonerlly et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
URL wwwps:tpp/.......... /ttt///taasssss former-ci rcui ts wtwppp//p11// frree/rrrawwrkkkoo / index. html.
Katja Filippova and Yasemin Altun. Overcoming the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1481-1499 2013. URL https: / callaactooooooyyoooocococooo wrgg/org//... 1
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, URL htttt:://aclanteeet
pp. 4098 4109, 2018.
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. DiscoFuse: A large-scale dataset for besoursee-bsedd sentence fusion. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pp. 3443-3455, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/119-1344 1348. URL https: / Taaclanhhllogo ogw.ogg.oooo org. NN1 1 1348.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Chris Pal. Measuring systematic generaliza- tion in neural proof generation with transformers. Advances Neural Information Processing Systems, 33:22231-22244 2020. URL 2ttp2/bbbbbce9112--aaaaatttt
cttpps..cc.pe Abstract html.
Tanya Goyal, Nazneen Fatema Rajani, Wenhao Liu, and Wojeiech Kryšeinski. HydraSum: Disen- tangling stylistic features text summarization using multi-decoder models. EMNLP, 2022.
URL hwtph:tt////t /ttpss///axvvvv/a
org/ abs/2110. 04400.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. ICLR, 2020. URL https: arwiv..iivv.////i///.aaaaaaaaavv
http://wwwxxlvooaa//ro//ea wttp///ggg//a..o../eee 04971.
Peter Hase and Mohit Bansal. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5540-5552 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2020..c/ 2att:200.cl--mii49000 491. URL https: //t////aaa////haa /rr/ 2020 acl-mai 4

---

Published as a conference paper at ICLR 2023
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015. URL https: arxiv. org/ abs//506. 03340.
Wan-Ting Hsu, Chieh-Kar Kai Lin, Ming- Ying Lee, Kerui Min, Jing Tang, and Min Sun. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 132-141, 2018. URL https: /awxivvvvvv./////ii//ar.////aiva org/ htt//bb//1110 0ww.s/......666....
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pp. 53-69 69, 2018. URL https: /tapiii.//axxvv...// org/ abs/1807 08556.
Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4474 4484, 2019. URL https: / //axivv.vvv...////avo/aaxa///aaa org/ abbs//bb//// / 99990999 05803.
Hongyan Jing and Kathleen McKeown. Cut and paste based text summarization. 1st Meeting of the North American Chapter of the Association for Computational Linguistics, 2000. URL https: / acl www.annnnnhoogg..o org/ 20000000022222
Hongyan Jing and Kathleen R McKeown. The decomposition of buman-written summary sentences. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and devel- opment in information retrieval, pp- 129-136, 1999. URL / /dl acm, org/ do /10. 1145/312 6244.......2 312666.
Aditya Kalyanpur, Tom Breloff, David Ferrucci, Adam Lally, and John Jantos. Braid: Weaving sym- bolic and neural knowledge into coherent logical explanations. In Proceedings of the AAAl Con- ference on Artificial Intelligence, 2022. URL https: / arxiv. org/ abs / 2011 13354.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1279- 2021. URL https://arxis: org/ hrg/abss/ abs//00s/2209 00751.
Walter Kintsch and Teun van Dijk. Cognitive psychology and discourse: Recalling and summarizing stories. Current trends in text linguisticcs, pp. 61-80, 1978. URL https::/wwwt degruyter. 61/pdf.
com/ /occumen//ooii htt///ooo/100/1 01111///97711105311111
Hung Le. Nancy Chen, and Steven Hoi. Vgnmn: Video-grounder neural module networks for iideo-grounded dialogue systems. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp-
3377-3393, 2022. URL //app:://axxv...//// org/ alss/ / 2104 07921.
Logan Lebanoff, Kaiqiang Song, Franck Dernoncour, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
1175-2189 2019. URL https: / arxiv. org/ abs/ 1 906. 00077.
Logan Lebanoff, Franck Dernnncuur, Doo Soon Kim, Lidan Wang, Walter Chang, and Fei Liu. Learning to fuse sentences with transformers for summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4136-4142 2020a. URL https://///........ /ttp/va.ivv////////////.///.aa org/ abs/ 03726.
Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Lidan Wang. Walter Chang, and Fei Liu. Understanding points of correspondence between sentences for abstractive sum- marization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 191-198, Online, July 2020b. Association for Com- putational Linguistics. doi: 10.18653/v1/20200 atw.d2...l-sss.2....... URL https: : / haolanthology. org/ 2220 acl 121....666
12

---

Published as a conference paper at ICLR 2023
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising requence-to-sequeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 7871-7880 2020. URL https:tp////....... / /ttx/vivvvv..////// orgg abs/ 1911999011 13461.
Haoran Li, Arash Einolghozati, Srinivasan Iyer, Bhargavi Paranjape, Yashar Mehdad, Sonal Gupta, and Marjan Ghazvininejad. EASE: Extractive-abstractive summarization with explanations. arXiv URL https: /arxiv.. org/abs/
preprint arXiv:2105.00 100.:098..00 2021.
06982.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Texr Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: : / /alaaaataatoooyooooooooaa org/w04-1013
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3730-3740, 2019. URL https: /atpaaaaiivv/////////a///aa/aaaa ozg abs 110099908000 08345.
Yang Liu, Ivan Titov, and Mirella Lapata. Single document summarization tree induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 1745-1755, 2019. URL https: aalaaaaaannttoooooooooooooomoaaa org/Nigg (2g/N199 91173.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, 2020. URL https: /tle://aaalla http//aaaaanttooooooooooooo org/
2020. acl amalmmaaa... in. 173.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018. URL https: //
org/ at//DDB/ DDt//D1B// 1011--221 1206.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 2021. URL https: aaaaiiiiiv....... org/ abs////b///// 2112. 00114.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1667, 2020. URL https:://// //tp://aaavvv///av org/ /bs/ 1910.10683.
Danilo Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henry Zhu, Xinchi Chen, Zhi- heng Huang, Peng Xu, Andrew Arnold, et al. Entailment tree explanations via iterative retrieval- generation reasoner. In NAACI., 2022. URL http///axvvvv.///e// org/ abs 22000 09224.
Stuart J. Russell and Peter Norvig. Artificial Intelligence: modern approach. Pearson, 3 edition, 2009.
Amrita Saha, Shafiq Joty, and Steven CH Hoi. Weakly supervised neuro-symbolic module networks for numerical reasoning, arXiv preprint arXiv:2101.11802 2021a. URL https: //arxiv. org/ brgg/bb/// 2111.1111111 11112.002..2
abs/
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. PRover: Proof gener- ation for interpretable reasoning over rules. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 122-136, 2020. URL https:
/ //aclanthology hangy..rg//e wwg/2020..o.......o emnle-mnmmmmmee momml--main.9.
Swarnadeep Saha, Prateek Yadav, and Mohit Bansal. multiPRover: Generating multiple proofs for improved interpretability in rule reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3662-3677, 2021b.
/2000.//2266600 01354.
13

---

Published as a conference paper at ICLR 2023
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. QvestEval: Summarization asks for fact-based evaluation. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 6594-6604, 2021. URL httttps//axxivvcor. alos/ 21 03. 12693.
Soñaa Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2931-2951, 2019. URL https: //acl catcha ogy- .org/P Parg/P19-1282
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. Obtaining faithful interpretations from compositional neural networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp- 5594-5608, 2020. URL https: / wwwaacchnllooooooooe wwtp///g.....oooooo
495.
Oyvind Tafjord. Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 3621-3634, 2021. 2012 13048.
URL https: http:///axxvv.o.....ot. /bs//
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1rXiv:2203.11771, 2022.
URL httph:ttp////ecttm //tsvvv.//ivvvvv/oa org/ abs /2003 11171.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. 2022. URL https:
/tttpp:://axxtee http//aavvvv..///// oerexiiv.orgaae www.://gggabb///aggo att///1l/2221..... 11903.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. URL aattps://arxiv... abs/ /2112. 04359.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCNLP),, pp. 11-20, 2019. URL https: / //a/aaaaaann /aaaaaaaantloooooooee
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, et al. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020. URL https:// aclanthology org/20200022...... www.p-ddmosss....eeeeeom--emomee demos. 6.
Jingging Zhang. Yao Zhao, Mohammad Salch, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Ma- chine Learning, pp. 11328- 11339. PMLR, 2020. URL 08777.
Yao Zhao, Mohammad Saleh, and Peter J Liu. Seal: Segment-wise extractive-absstrative long-form text summarization. arXiv preprint arXiv:2006. 10213, 2020. URL https://axxxii.... abs/2006 10213.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. Extrac- tive summarization text matching. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 6197-6208, 2020. URL https: http:::://alaathologgoo org/202 acl main. 552.
Yilun Zhou, Marco Tulio Ribeiro, and Julie Shah. ExSum: From local explanations to model un- derstanding. In Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp.
Seattle, United States, July 2022. Association for Computational Linguistics. doi: 20118665311/2222 maact-main.392. URL https:////// //tc///alaaee hta.t//////ccaattottt www.laanoooog..rgg22220
naacl -main 392.
14

---

Published as a conference paper at ICLR 2023
Tree Structure
H
Freq (in %)
compression (-)
1
compression fusion (.,
2
8
fusion ( fuuionn (:, fusion ))
2
(+)
0
fusion ( compression (-) fusion (:-))
2
6
paraphrase ( commpesssoo (-))
2
paraphrase ( fusion (:.))
2
6
fusion ( fusion (:, compression ())
2
fusion ( fusion ())
2
5
fusion compression (+))
2
Table 5: Top 10 tree structures for the human-written summaries in the training corpus of CNN/DM. For clarity, each tree structure is accompanied with the corresponding tree height and the frequency (in percentage). An empty structure of "(-)" represents a singleton node.
A DESIGN CHOICES FOR EFFICIENT SP-SEARCH (CONTINUED FROM $4)
SP-SEARCE SEARCE is outlined in Algorithm 1. It is made efficient through important design choices as discussed below. For clarity, we do not show all of these choices the Algorithm outline.
Top-k Document Sentences. The search space grows exponentially at each depth (because of the fusion operation), but one way to limit the growth is ensure that SP-SEARCH starts with small number of document sentences. A summary is typically constructed by using information from only small number of sentences from the document. Hence, rank each document sentence by computing the ROUGE-1 score with the summary and build the Summarization Program with only the top-k document sentences ('k' being a small number) as eligible source sentences.
Filtering Queue Elements. The search space also dependent on how many elements are added to the queue at each step and how many of those are expanded further. Note that whenever new sentence generated via a module, can potentially fuse with all previous generations to create new queue elements. Since doing this exhaustively increases the search space, SP-SEARCE detines certain heuristics for choosing the elements that will be added to the queue: (l) a sentence that has been compressed once is not compressed again, (2) sentence that has been paraphrased once is not paraphrased again, (3) each document sentence is used at most once in a tree, (4) two sentences not fused if they are intermediate generations from the same sentence, and (5) since fusion is not a symmetric operation and can lead to different generations based on the order, sentence fusion happens keeping the temporal order of the sentences in the document intact.
Priority Queue and Pruning. SE-SEARCH performs an additional step of pruning for the elements that are added to the queue. It maintains fixed queue size and while expanding the elements in the queue in a best-first manner, it maintains a ranked list of the elements according to Eq. 1 such that only the tap-ranked elements are kept in the queue and the rest are pruned.
Parent has higher ROUGE than children. Each module generates multiple outputs through beam search and SP-SEARCH ensures that the best output has a higher R-L score than the nodes on which the module is defined. If this is not the case, the corresponding branch of the search is not expanded further. This constraint generalizes to the property that every node in a Summarization Program will have a higher R-L score (with respect the summary sentence) than all other nodes in the subtree rooted at that node. Conceptually, this greedy approach ensures that every reasoning step Summarization Program a step closer the summary (according scoring function).
Maximum Tree Height. SP-SEARCH chooses a maximum height of the trees, beyond which the nodes are not expanded further during the search.
Batching Module Executions. Instead of executing each module separately, which requires a for- ward pass over a neural model and can be Lime-consuming SP-SEARCH executes all operations in the queue together by batching at each depth of the search.
15

---

Published as a conference paper at ICLR 2023
Algorithm 1: SP-SEARCH Algorithm
Input: Top-k Document Sentences Dr, Summary Modules (Me, Mp. My,, Maximum Height H. Maximum Queue Size Q, Number of Generations G, Scoring Function Score
Output: Summarization Program P
function SP-SEARCH(P&, (Me,
MMM My), H, 2, Score)
P=
Initialize Summarization Frogram
forall S
queue <- [], height f-
b Initialize queue of items and tree height. Initialize tree with maximum ROUGE.
maaRouge <- 0, bestRoot 4
forall D1 Dx do currftouge + ComputcRoug((DI.)) S)
P Compute ROUGE with leaf nodes.
if currRouge > mazhlouge then mazRouge + curRRouge, bestRoot + Node(Di) b Update ROUGH and root.
queue + queue U {(D1, Me, height)
Add compression items. Add paraphrase items.
11
queue +- quere {(D), 0, Mp. height)
12
for D2 E De do
queue <- queue U [(Di. D2, My, height))
Add fusion items.
14
preuHeight +- 0, visited t- [] while [queue] > 0 do
15
height queue(O) heiggh if height 7 then continue
Get height of the top queue item. Stop expanding beyond max height.
if height + prev/leight then prevHliigtt height
queue + arg que (queue, C, Score) Rank and prune the queue. gens Execute(qquuee,, (Me, Mp. Execute all operations in queue.
23
newNode + queue.pop()
b Pop the top item from the queue. Get the newly generated sentence. Get zouge with sentence.
newGen +- gens NuewNNdee
newRouge CComputeooug((eeuGGG if newRouge > maaRouge then
26
27
marRouge + newRouge, bestRoot + Node(newGen)
b Update root.
if newGen & visited then visited + visited (newGGn],, height height +11 queue + queue U U [[nneWenn Φ) MM. height)} quene + queue U [[newGen, Φ., Mp, height))
Add compression items.
31
b Add paraphrase items.
forall preeGen E visiled do queue queue U [[newGen, prevGen, height)) itens.
Add fusion
Ts ConstructTrea(bestRool Get by backtracking from best root.
35
+ {Ts}
> Append new tree to the Summarization Frogram.
36
return P
B ANALYSIS OF SP-SEARCH SUMMARIZATION PROGRAMS FOR CNN/DM
We observe that 68% of the summaries in CNN/DM have 3 or 4 summary sentences and, equiva- lently, the corresponding Summarization Programs have 3 or 4 trees. Note that while we initialize SP-SEARCH with the Top-4 document sentences, Summarization Program may choose to ignore some of these sentences including them does not lead to higher ROUGE scores. Upon analysis, we find that 73% of the programs are constructed using all four initial sentences, 23% are constructed with three sentences and 3% have two sentences. We also note that the trees can have any height up to the maximum defined height of 2. A tree of height represents a singleton node with a single document sentence. Thus, Summarization Program with only singleton nodes represents tractive summary, Overall, we observe that our trees have as many as 20 different structures. Table shows the top 10 tree structures in our corpus. As an example, tree structure of "compression (
16

---

Published as a conference paper at ICLR 2023
Q
ROUGE (RI/R2/RL)
20
beam (5)
59.36 37.38 55.59
21.09
4
20
beam (5)
2
61.88 40.11 / 58.46
29.58
20
beam (5)
2
63.72 41.99 / 60.46
35.80
4
10
beam (5)
2
61.21 39.06/ 57.4 .6 61.88 40.11 / 58,46
15.38
20
beam (5)
29.58
4
30
beam (5)
2
62.32 40.57 / 58.97
41.61
4
20
beam (5)
56.45 / 34.16 51.34
13.58
20
beam (5)
2
61.88 / 440.1 58.46
29.58
4
20
beam (5)
3
65.00 43.81 / 62.36
34.68
20
greedy
2
52.87
30.12 47.08
26.85
4
20
beam (1)
2
56.91
344344 52.15
28.56
20
beam (5)
61.88 40.11 58.46
29.58
Table 6: Analysis of ROUGE scores (R1/R2/RL) and average search time (in seconds) for SP- SEARCH on 1000 validation samples in CNN/DM with different search hyperparameters. k= Num- ber of extracted document sentences. Q Maximum queue size. D Decoding Strategy from the modules. Maximum height of the tree. T= Average search time in seconds. Search times are computed single RTX 2080 Ti GPU. Beam(5) refers to beam search with beam size 5 and best of top-5 generations from each module.
Opt. Metric
R1
R2
RL
RLsum
RI
64.28
38.40
54.36
57.68
R2
59.11
41.85
53.28
55.97
RL
61.88
40.11
58.46
60.32
Table 7: Comparison of ROUGE scores for the summaries generated by SP-SEARCH based on the optimization metric. Optimizing for R-L leads to more consistent results across all ROUGE metrics.
fusion (-, )) represents a tree of height in which two document sentences are first fused and then the resultant sentence is compressed.
C SP-SEARCH HYPERPARAMETERS
In order to analyze the effect of different hyperparameters in SP-SEARCH, we compare the trade-off between ROUGE scores and average search time per sample (on a single RTX 2080 Ti GPU) by varying the number of initial document sentences (k), the queue size (Q), decoding strategy (D) and the maximum height of the trees (H) in Table 6. We observe that increasing the number of document sentences beyond leads to some improvements in ROUGE scores. However, this comes at the cost of increased search time. Increasing the queue size (Q) also leads minor improvements but again suffers from increased search time. While increasing the maximum tree height (H) to results in better ROUGE scores, on inspection, fusion of two arbitrary sentences that may not always lead to better programs. Finally, beam search performs better than greedy decoding. and leveraging the best of the top-5 generations from each module improves the results further with almost no increase in search time. Overall, at a moderate search time of less than 30 seconds/sample on single RTX 2080 Ti GPU, SP-SEARCH obtains 20 point increase in R-2 compared to our extractive baseline with leaf nodes (document sentences).
observe that happens primarily due low-quality
UP-SEARCH builds Summarization Programs by optimizing for ROUGE-L because of its better overall performance across all ROUGE metrics. As shown in Table 7, optimizing instead for R-1 and R-2 leads to slightly better R-1 and R-2 respectively but overall R-L performs better. In general,
We use the HuggingFace implementation of ROUGE at https: //g thub. collb..coo/iii www/huggiineeocccccccc.otc.....o ngface///gf//// dataset /bl ob/ http://aiinneeetiieeeeoc
17

---

Published as a conference paper at ICLR 2023
Metric
R1
R2
RI.
Rl.su
RI
61.88
40.11
58.46
60.32
R2
62.07
41.61
59.05
60.76
RL
61.79
40.36
58.66
60.38
Table 8: Comparison of ROUGE scores for the summaries generated SP-SEARCH based on the metric used for choosing the initial sentences (IS). The optimization metric is set to R-L for all.
Compression
Paraphrase
Fusion
Non-Factual
Compression
0.001
0.05
0.00110.00
Paraphrase
0.68+0.04
0.90=0.05
0.04
Fusion
0.86+0.01
0,45=0.03
0.80:10.03
0.20
Table 9: RQ1 Neural Module Faithfulness evaluation of Summarization Programs. Each entry (i.)) shows the fraction time a module i' in a row performs the operation "j' column. The final column shows how often module generates non-factual outputs. We report the mean and variance the scores between the two annotators.
we find that the different ROUGE metries may not correlate well with each other when optimizing for one of these.
SP-SEARCH uses ROUGE-1 scores to select the candidate sentences for building the Summarization Programs. We also experiment with other ROUGE metrics in Table 8 and observe that our algorithm fairly robust to the choice of metric. R-2 obtains slightly better results than R-1 and R-L but overall, all metrics still obtain sufficiently high oracle results of 40-41 points in R-2 through SP- Search, independent of the selection metric.
D SP MODEL HYPERPARAMETERS
We build our models on top of the HuggingFace transformers library (Wolf et al., 2020). All models trained for 40000 steps with batch size of 16, learning rate of 3 * 10 and warmup steps of 500. We set the maximum input length to 512 and maximum generation length to 100. During inference, generate up Top-10 Summarization Programs with beam search and output the first well-formed program. We also set the minimum generation length to 10 to prevent the model from generating too short sequences and repetition penalty to Program execution from the SPs is performed with the same set of hyperparamaters for each module used during SPASEARCH.
E RQI RESULTS: NEURAL MODULE FAITHFULNESS EVALUATION ON CNN/DM (CONTINUED FROM $6.1)
We discuss the faithfulness evaluation of our modules below.
Study Design. We would attribute each sentence to exactly one of the modules, but since mod- ules are generative, we do not know if each module performs only and exactly its named function. For example, fusing two sentences into fluent sentence may in practice also involve phrasing or compression. Hence, we evaluate module faithfulness by analyzing how often a module my performs the role of module my, Two expert annotators, with knowledge of NLP, annotate each intermediate generation (from module) and assign a binary label against each of the modules that could lead that output. Additionally, the annotators also label each generation for whether contains non-factual content (that cannot be verified from the sentence(s) used to generate it). The study conducted with 20 Summarization Programs, consisting of a total of 107 samples, with 54 fusion, 25 paraphrase, and 28 compression operations.
para-
Results. Table shows the results. The diagonal entries in the table demonstrate how often a module performs its intended behavior and the high values of 0.8-0.98 suggest that modules
18

---

Published as a conference paper at ICLR 2023
R1
R2
RL
slsum
Lead
15.88
1.54
11.71
11.81
SP-SEARCH Leaves
25.10
5.89
16,48
19.44
Top-1 Sentence
29.77
7.21
20.85
21.13
Random SP
28.96
9.04
22.38
22.41
BART-Oracle (Sample)
49.23
25.82
42.31
42.30
UP-SEARCH Top-1
47.72
24.83
43,50
43.50
SE-SEARCH
51.86
29.60
48.48
48.51
Table 10: RQI Comparison of ROUGE scores for the SP-SEARCII summaries with different oracle extractive and abstractive baselines on the XSum dataset.
are highly faithful. In about 20% of the cases, a fusion operation may not involve any fusion. We also observe that some form of compression frequently happens as part of the paraphrasing and fusion modules, tend to involve some compression and paraphrasing operations. Interestingly, the values in the last column of the table demonstrate that compression and paraphrase modules almost never generate non-factual outputs while our fusion module more prone to that (about 20%), indicating room for improvement in the fusion module.
shown in the first column of the table. Similarly, the fusion outputs also
F RESULTS ON XSUM DATASET (CONTINUED FROM $6)
To further test the generalizability of Summarization Programs, conduct experiments another single document summarization dataset, XSum (Narayan et al., 2018), a highly abstractive dataset consisting of single-sentence summaries.
F.1 RQI RESULTS: CAN SP-SEARCH REPRESENT THE SUMMARIZATION PROCESS?
As part of answering RQI, we keep the paraphrase and compression modules unaltered and only retrain the fusion module for XSum, with the training data obtained using the same heuristics as used for CNN/DM (Lebanoff al., 2019). We also the same hyperparameters for SP-SEARCH as those used for CNN/DM and compare all methods on 1000 randomly chosen validation samples of XSum. As shown in Table 10, SP-SEARCH obtains R-2 of 29.60 and R-L of 48.48, out- performing all baseline methods by significant margin. Unlike CNN/DM, extractive baselines like 'Lead I', "SP-SEARCH Leaves', and "Top-1 Sentence' do not perform well for XSum, while the abstractive baseline, BART-Oracle does significantly better. The SP-SEARCH results are also lower than in CNN/DM because of the highly abstractive nature the dataset and the relative difficulty in emulating reference summaries.
F.2
RQ2 RESULTS: EVALUATION OF SUMMARIZATION PROGRAM GENERATION MODELS
Next, for RQ2, we train Joint SP generation model that generates the program directly from the document." In Table 11, we compare our model with BART and PEGASUS and random SP generation baseline. While our model outperforms the random SP baseline, there's 's significant room for improvement compared to state-of-the of--th---tt-rrr-- methods, as indicated by the 16 point gap in R-2 compared to the SP-SEARCH oracle.
G EXAMPLES OF SUMMARIZATION PROGRAMS
Figures 3, 4, 5, and 6 show some examples of Summarization Programs identified by SP-S SEARCH human-written written summaries. Figures 8, and show examples of Summarization Programs and corresponding summaries generated by Extract-and-Buile SP generation model. For compact- ness, we show the Summarization Programs by merging the common nodes between the trees.
for human-
9We do not experiment with the Extract-and Build model hecause extractive models do not perform well XSum.
19

---

Published as a conference paper at ICLR 2023
R1
R2
RL
RLsum
BART (Lewis al., 2020)
45.14
22.27
37.25
37.25
PEGASUS (Zhang al., 2020)
47.21
24.56
39.25
39.25
Random (Joint)
28.71
9.20
22.43
22.43
SP Model (Joint)
33.56
12.18
26.18
26.17
SP- SEARCH (Oracle)
51.78
28.88
48.29
48.29
Table 11: RQ2 Comparison of our Joint SP generation model with state-of-the-art abstractive summarization models on the XSum test set.
Wednesday. Police, family, Gold Summary: Cayman volunteers search for eighth- grader.
SP-Search Summary: Cayman Naib, 13, has been missing since Wednesday. Volunteers
searching for missing eighth-grader.
S1" Volunteers one searching for missing aighth-gradir.
Compression
H:: Hundrreds volunteens searching for eighth
$2': Cayman Nalb, 13, has been miesing since Wednescaay.
Fusion
Fusion
Fusion
Dt: The search has drawn hundreds of voluntears on fool and online.
D2: Pannsytvaria community is pulling togother to search for eighth trat- student who been missing since Wednesday.
D3: The parants of Gayman Naih, 13, have been communicatin through the Facebook group "Find Gayman" since day after his diseppearance, according to close friend David Binswainger
Figure 3: Example of a Summarization Program identified by SP-SEARCH. The summary identified by SP-SEARCH matches closely with the gold summary.
Gold Summary: Real Madrid slump defeat against Atheetic Bilbao Solttay goal from Artz Aduriz enough to give the Basques vistory. Bayem Munich continue Bundesligo ga comination
SP-Search Real Madrid
headed
Munich
Bundeslige
51: Hoad Musted Alticic B.aan.
Aobaic teacced Blisoo
Panephease
111 Mosdiot celoot
JrI Adbett roaded hot out tal
Bupur Munic muach lo Bandnoggu RAk,
Comprastion
Ponppraa
Compramioon
farce Alt slc bilooo Saturdes, poketiabl hutcingg
DC: suunning headee Arts Adurte eooigh to day organbres iepasazio Blisas
D2: Capar Munics contrund seaningle Insocaable morch trind conaecuttie Bandenigo with Sors vetter nwily
Figure 4: Example of Summarization Program identified by SP-SEARCH. The summary identified by SP-SEARCH matches closely with the gold summary.
20

---

Published as a conference paper at ICLR 2023
Gold Summary: Comic book ertial Normen Lee went, missing in the Ceyman Islands on Thursdly, Authorllie called off search Frida evening.
SP SP-Search Summary: Comic book artist Norman Lee went missing the Cayman Islands Monday. The search lasted Friday avaning.
Come: baok wrter hormann weet
Honday
82: The search kinled until Friday evening.
Packon
Compression
Nt:harnnn Lew ward misinn ahii ancreeling the ooast Cayman
kaands
Fribyy euening
Compression
Cosgression
Paraphrase
Norman Les, arhat for and Marvel cariac, seet missing ancilolinn Grand Caymar, CNN att sie SeCVS reported.
D2: The wearch for comn book artal massing the Thursday called necovery masiios
Strong
Frdday
everingg Caymar 27 reported
Figure 5: Example of a Summarization Program identified by SP-SEARCH. The summary identified by SP-SEARCH contains a factually incorrect part "on Monday". The Summarization Program lets us identify the exact step in which the error was introduced.
Gold Summary: Two American pornography film reportedly
inbaas Colasseum wall. Meamwhlie, Egyst investgating Russian Pyramids. Cambodia's Angkor Archeological Park experienced string of nudit--alatedd incidents this year.
SP-Search Summary: Amarican amested carving inttals have regortecly been Coloeseum, Cambodis's Anggkor Archeological Park experienced own string nudit-relltted incidents this year.
Two
42-year-old
Amneiion
yeancid reportacly seen arrased Colcaeam
D44501 Cambodo's Angkor Archeoogical Park experienced Es own siting of rudiy- reciera this year
caniin thair trtass io wal
Compricsion
Raiion
It:s American wamer Aem srrenedd
heer anwored
reportusly
sruppedia saltte of themaaivee the intiile before they were amosked Coloseourn
Poraphrase
Pustan
Pasion
D1i Two American women have importetly been amaneed carving thai retara into wall with
D2: The yoar old man wis aspretended oñt guard of the
DS: The worton, buth fro Ca fomi is ropar judly srippod so of
Figure 6: Example of a Summarization Program identified by SP-SEARCH. It contains a singleton node (D4/S3). The first two summary sentences have overlap in information, originating from the independence assumption in the generative process of each summary sentence.
Gold Summary: Marselle prosecutor says Journaistss at Bid Paris Match are "very confident" the video diip an editor says. Androas Lubitz informed Lufthansa training school an opisade of severe depression, airline says.
investigation" despite media reports.
Predicted Summary: Prosecutor Brice Robin says he is not aware of any video footage of the crash of Flight 9525. The video recovered from phone the crash according Paris Match.
of
viãco Boolige crish Rightt $525.
Pusion
It: The prosecutor loading an invest gattion into cash Germaninngs iight 8005 makted was not
says video lootage plans.
anyy
crsah according to Parta Mater.
Compression
Fusion
Fusion
Peraphrose
D1: The Frcnch prossouter loading inverliation Bhe clish of Cemmunatige rlight 8575 iextee Widdneddyy that wae senn fortaga
D2: Marselllo prosecutor oo Robin xil CNN thee "eo vieer wern uied the creeh inseiigttion
D3: Parts Match and Blld reported fiet the video vas necovered from phane wreckape site.
Figure 7: Example of Summarization Program and summary generated by the Extract-and-Build SP generation model.
21

---

Published as a conference paper at ICLR 2023
Gold Summary: Lucknow, vehicles are being fitted cameras and pepper spray subdue engry protesters, Some Indians have questioned why police are resorting 'aumortarian and foroeful methods""
bought
using drones spray prowds with aerial solution crowds. Drones between Summary: $9,560 and $19, 300 and have beon losted in controlled conditions.
51: Police armedd with an awbel soudion crowdss
Droncs $3.659
bwen lected in combriile conellione.
Faskon
Fuskon
Fusion
It: Police is indis putling in lavar an soludon crowds.
I2: Picios en indt uaing minattur drames op-ty crowis sepper spiiy.
12: cald that drones hed been lested controlee condition,
Compresiion
Fusion
Porophrase
D1: Pulice Indla putlin uside thair bakons fovor rverteed angry and unidy apeayyiigggggg
D2: The menantare arroalt be lited with carrena
peppor spray: eech drone nonts bwiteeen 58 590 and
Figure 8: Example of Summarization Program and summary generated by the Extract-and and-Blld SP generation model.
Gold Summary: Monning is serving a 35-year sentence leaking thousands of classified documants, says she using voice phone to diclate her tweets.
Predicted Summary: Manning appears have joined Twither this waek. Manning sald she suffers from gender dysphoria. In Augusi 2013, Marning said she wanted transitinn from male lo femaal after being sentenced 35 years in prison.
Marning toom gender dpsphoria,
Viaaaion malc fomali oftor boing serkaroed years prisn.
Karning oppoars joi this
var carted to lnn Whr make feetst sther baing aertencod crima
cnntinnca in 36 panrs prlton 2015 and ahs Iramaliion ferma
Compression
Panophoose
Fusion
Fuskon
Bt: Veenning who in sending Migene prlion seninnce sined Twittr the wesk.
appeon
DG: Manning wis sontence 2018, and August of yoar
Movningg
gooder dymphoris
Figure 9: Example of a Summarization Program and summary generated by the Extract-anc and-Buulld SP generation model. The generated summary contains factual error that Manning sentenced to "35 years in prison".
22

---

Published as a conference paper at ICLR 2023
Taik
Bestfoncta taber
Figure 10: Interface for simulating Summarization Programs given source document and model
summary. Annotators construct trees for all summary sentences by creating one edge at time.
H SIMULATABILITY INTERFACE OF SUMMARIZATION PROGRAMS
In Figure 10, show the interface for simulatability stuyy in which annotators asked write Summarization Pograms for model-generated summaries.
23


Reviews:
[{"review_id": "zjlu3VglpJ", "reviewer": "Reviewer_Bbv2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an explainable abstractive summarization framework by exploring the two research questions: RQ1 and RQ2. Experiment results on CNN/DailyMail dataset are given to demonstrate the effectiveness of the proposed SP framework.", "summary_of_the_review": "This paper proposes a summarization program, which is an interpretable modular framework to explain the abstractive summarization. SP-SEARCH could identify SPs for human-written summaries, then a summarization program generation model is generated with SPs. Experiment results are given to demonstrate the effectiveness of the proposed approach.", "strength_and_weaknesses": "It is nice to propose a novel framework to explore the interpretability of abstractive summarization on a public benchmark dataset. The proposed framework is novel. But some explanations are lacking in this paper,\n1. why neural modular tree could be employed in this paper? Could you give some explanations?\n2. In Appendix A, the top-k sentences are selected by computing ROUGE-1 with a summary. Why is \"ROUGE-1\" used as the evaluation metric?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "stlt-j4Fu_P", "reviewer": "Reviewer_wpG8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework, summarization programs (SPs), for interpretable summarization. It starts with source document sentences as leaves, applies a series of transformations of fusion, compression, and paraphrasing, and finally leads to the root nodes for each summary sentence. Such design makes the generation process clear to humans and makes it easier to catch any hallucination created along the way. The authors come up with a best-first search algorithm, SP-SEARCH, to take document sentences and gold summaries as input, and output SPs for supervised training. They also attempt to train models to generate the SPs, and conduct simulation studies to gauge the interpretability of the summarization. Nevertheless, their results still lag behind those of the SOTA models.", "summary_of_the_review": "Overall this paper proposes an interesting idea, and the writing is easy to follow. Most details on the experiments are reported clearly, and there is extensive analysis of the results as well. However, there is limited novelty in the model, and the proposed approach is too simple and cannot give good results as compared to the SOTA models. ", "strength_and_weaknesses": "Strength:\n‒ SP is an interesting and novel idea \n\n‒ The design makes it much easier to trace the generation process, and to manually detect if any hallucination has occurred; the method helps to limit the issue of hallucinations down to mostly a single module (fusion)\n\nWeaknesses:\n\nOn experiments for RQ2\n\n‒ The results for generating SPs are much weaker than the SOTA both in terms of fluency and factuality: as compared to the model for the base architecture (BART), the best SP model experience a 6.53 drop in Rouge-1 and a 2.27 drop in QuestEval. Even though the authors show the Oracle scores to be much higher, it is not guaranteed if such potential can be realized since the Oracles make use of gold in the first place.\n\n‒ From what I understand, for the RQ2 experiments, authors only finetuned BART for generating SPs (or used an existing model for the sentence extraction stage). There is little novelty and the approach seems unnatural. It is possible that the target output (SPs) differs too much from the pre-trained text. The three modules (fusion, compression, and paraphrase) are decoupled from the SP-generating module, and it is possible that the SP-generating module is totally unaware what are the exact implications of its generated modular instructions. \n\n‒ The authors used three separate large neural models for paragraph, compression, and fusion; from the perspective of parameter sizes, it may be unfair to compare the performance of the SP Models to the BART or PEGASUS models. \n\nOn experiments for RQ1\n\n‒ It is not explicitly stated in the paper, but from what I understand, the BART-Oracles (Beam/Sample) are first fine-tuned on the training set and then used for generation? In this case, it is not surprising that it should have worse results than SP-SEARCH, which refers to the gold summaries for practically every step of its search process.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wzk0cH6Lhsq", "reviewer": "Reviewer_G3xd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to model summaries in the form of \"summarization program\", a sequence of 3 actions (compression, fusion, paraphrase) over sentences. Such an approach has the advantage of being more interpretable, as the final summary comes together with a \"proof\" on how it was obtained. \nThis open very interesting avenues for interactive summarization, as well as inspection of the intermediate steps performed by the system.\n\nTwo types of experiments are reported in this paper:\n1. An exploratory one, which aims to evaluate what could be achieved with such an approach. This is done in by optimizing for the oracle summary.\n2. A predictive one, which evaluates the summaries predicted on unseen example (the more traditional ML approach)", "summary_of_the_review": "An original proposal, with lots of potential for interpretability work. The search-space seems wide enough to also be possible to use the method to obtain SOTA results - as measured by Rouge scores. However, this potential is far from being realized in thie paper.\n\nMy recommendation is still for acceptance, because of its originality and because it takes \"the path less trodden\".", "strength_and_weaknesses": "Strengths\n\n1. This is a very original paper, in an undervalued research line - that of interpretable summarization. The proposal is compelling and its advantages well argued for\n2. The empirical results showing the potential of proposed model are promising. The best obtainable summary is +20 Rouge-L points above what can reasonably be expected to be obtained with state-of-the-art methods\n\nWeakness\n1. The second set of experiments are underwhelming. Not only do they perform rather far from current pure neural methods, they do so even when using those as starting point: \"Extract-and-Build\" (RL=21.8) starts from the extractive summary obtained through MatchSum (RL=28.7). A very ruthless reading of Table 3 would be that the proposed technique starts from something good and turns it into something worse, although not as bad as random selection.\nThe authors discuss why this could be the case, but the results in the paper do not indicate that this gap could be overcome.\n2. Those underwhelming results come as a total surprise to the reader, as they are not forshadowed in the abstract or introduction. This disappointment could easily be avoided, at the cost of turning away those readers that only come for the SOTA results (which would in any case not be interested after finishing the paper)\n3. The simulation protocol is very interesting to study the intepretability of SP programms, but it doesn't seem to me that too much should be read into it. In short, in the study humans have to predict the program before and after seeing the program explained by the proposed system. The Rouge score of the summaries obtained by the human-chosen program _after_ the training phase is slightly better. However, it should be noted that the 3 humans who performed this were authors of the paper. So, on one side they were clearly not neutral, and on the other hand they probably started with a very good understanding from the form of the program (therefore the difference might be smaller than with non-experts).\n4. Experiments are only reported on one dataset, and the last years of summarization papers have but the bar higher than that.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1m2MaNZ-2dH", "reviewer": "Reviewer_T8vS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a system/paradigm called “summarization programs”. The goal is to build an interpretable search-then-assemble program for abstractive summarization. Best-first search is deployed to find the the neural module operations, including compression, fusion, and paraphrasing. There are comprehensive analysis and results about the proposed model and its ablated variants. \nAlthough the system does not beat end-to-end SOTA models, it provides a data point for building modular text generation systems. The idea behind is novel and exciting, and the author explains the design choices well. I would like to accept this paper.\n", "summary_of_the_review": "It's a strong paper with novel ideas and solid experiments. Although the system does not beat end-to-end SOTA models, it provides a data point for building modular text generation systems. The idea behind is novel and exciting, and the author explains the design choices well. I would like to accept this paper.\n", "strength_and_weaknesses": "Accept:\nThe paper introduces the notion of “program” to decompose the generation of summaries. It’s an exciting direction that blurs the boundary of extractive and abstractive, end-to-end and modular summarization systems. \n\nThe paper carefully chooses components and explains the reason behind the choices. \n\nReject:\nThere are some reasons to “reject” this paper, but they are mostly inevitable on the way building a complicated modular systems. \n\nThe search algorithm and process is very complicated, and I intuitively can imagine how fragile it is. \nThe experimental results (non-oracle) are not competitive. \n\nThe definition and derivation of the three meta operations are worth further consideration. For example, compression and paraphrasing are highly overlapped with each other. The granularity of operation can also be more fine-grained (EDUs, clauses, etc.).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster