Information-Theoretic Distillation for Reference-less Summarization
Authors: Jaehun Jung
Ximing Lu
Liwei Jiang
Faeze Brahman
Peter West
Pang Wei Koh
Yejin Choi
Venue: COLM (2024)
Keywords: summarization; self-training; distillation; PMI maximization
TL;DR: We present a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM’s capability or human-written references.

Abstract: The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method---that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata of summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information-centric measures of ideal summaries. Distilling from the improved teacher, we arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone state-of-the-art unsupervised methods, and wins over ChatGPT in controllable summarization.

Published as a conference paper at COLM 2024
Information-Theoretic Distillation for Reference-less Summarization
Jaehun Jung'
Ximing Lu Liwei Jiang'
Faeze Brahman't
Peter West
Pang Wei Koh* Yejin Choitt
*Paul G. Allen School of Computer Science & Engineering, University of Washington
AAllen Institute for Artificial Intelligence
hoony 20nny1366a 123dcs washington. edu
Abstract
The current winning recipe for automatic summarization using pro- prietary large-scale language models (LLMs) such as ChatGIT as is, or imitation learning from them as teacher models. While increasingly ubiqui- tous dependence such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek alternative learning method-that allows for a more cost-efficient, controllable, yet powerful summarizer. We present INFOSUMM, a novel framework to distill a power- ful summarizer based on the information-theoree rization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.88 as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information maries. Distilling from the improved teacher, arrive at compact but powerful summarizer with only 568M parameters that performs compet- itively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone vised methods, and wins over ChatGPT in controllable summarization.
objective for summa-
measures of ideal sum-
unsuper-
1 Introduction
The winning recipe for summarization today is to prompt a gigantic, proprietary LLM such as ChatGPT, either as a summarizer itself or as a teacher model for imitation learning (Goyal et al., 2023). In order to reduce the inference cost, one maybe particularly tempted to distill a compact summarizer from the LLM: by collecting some documents, instructing the LLM to summarize them, and supervising a small model simply imitate the generations (Xu et al., 2023; Sclar et al., 2022). Despite its intuitive appeal, this process does not involve how we explicitly define a good summary-the feasibility of data generation is fundamentally dependent on the LLM's capability to follow the instruction. With no quantifiable objective for summarization, our best option is use the largest and strongest LLM as the teacher, and enumerate much imitation data as possible from it (Li et al., 2023; Mukherjee et al., 2023). Despite this increasing dependence on large LMs, it is still unclear whether the distilled summarizer will fully generalize across diverse use cases (Gudibande et al., 2023), whether it be zero-shot adaptation to unseen domains or generating controllable summaries.
In this work, we shift our attention from using a larger and stronger teacher model, and show that even the small, off-the-shelf LMs can teach themselves to excel at summarization, provided we define an information-thene theoreetic objective for summarization. Concretely, we propose that the three evaluative dimensions of summarization-saalle saliency, faithfulness and brevity-can be incorporated into unified search objective, where we look for summary

---

Published as a conference paper at COLM 2024
Information-mmxximiznng objective for Summarization
yy < 8
Document x Summary
Find (x.y) such that PMI(x;y) a and
Self-Improve teacher through Expert Iteration
Self Self-supervised critics ($2.1)
PMllmaximization decoding ($2.2)
P' AMzzarlxx (Xmustt y)
Sallency: log
> dj
Generate Jummary y LLae(
P PuLLNX (xmash)
PaLLA Ymuust X)
Small LM (Pyth--2008
Cancidate pars
Faithfu: log
02
Filtored pairs
Generalee document
PaIAa Ymaok)
Paul 19)
1y
Pzu(
Brevity:
|x|
Supervise on filtered pairs ($2.2)
3. Distill student summarizer
Generate pairs
Fine-tune student LM
Improved
Critic filtering
Sugh-Qualtyy
Expert
Teacher
Summarizer
Figure 1: Overview of INFOSUMM. We formulate summarization as (1) information maximization objective under length constraint, which allows us (2) self-train an expert teacher from only a small, off-the-shelf LM and self-supervised critics. Finally, (3) distilling from the improved teacher, we obtain a compact yet powerful summarizer without relying on an LLM already competent at summarization or human-annotater references.
y that maximizes its point-wise mutual information (PMI) with the document under length constraint. By self-improving the teacher through expert iteration (Anthony et al., 2017) to align with our objective, we yield high-quality summarization dataset only from small teacher LM not tuned for summarization. This method, INFOSUMM (Figure 1), decouples what we expect generate (i.e., the explicit search objective for summarization) from how we generale them (i.e., Sata-generating LM), allowing us to distill a powerful summarization model without human-written references or an LLM already competent at summarization. Compared to prior work that distills from small, off-the-shelf LM (Jung et al., 2023), INFOSUMM targets substantially longer, decument-level summarization, and operates entirely without human-supervised critics.
Applying our method, we train a 568M summarizer with the dataset generated from Pythia- 2.8B (Biderman et al., 2023), an off-the-shelf autoregressive LM that itself cannot reliably summarize a given document. We test our model on diverse tasks spanning news summa- rization, zero-shot generalization to unseen domains, and controllable summarization. Our model, despite its small scale, exhibits surprisingly strong performance compared to the state-of-the-art: significantly outperforms all unsupervised methods in reference-based evaluation, improving more than R-1 points across benchmarks. In GPT-4 and human evaluation, our system is even preferred to the in-domain reference-supervised models, and outperforms 175B ChatGPT with simple re-ranking approach. Notably, our model, as compact, expert model for summarization, exhibits significantly better controllability than prompting ChatGPT (e.g,, to generate a long, highly specific summary of the given docu- ment), establishing promising alternative to imitating human references or I.L.Mgeneraeedd summaries.
2 INFOSUMM: Information-Theored Distillation for Summarization
2.1 Summarization as Information Maximization
Intuitively, good summary y should be a brief representation of the original document (brevity), that focuses on the key information of x (saliency), without hallucinating unsup-
2

---

Published as a conference paper at COLM 2024
ported content (faithfulness) (Fabbri et al., 2021b). In this section, we first quantify these desiderata of summarization in an information-theores perspective, then discuss how they can be unified maximizing the mutual information between the document x and summary subject to length constraint on
Saliency A good summary y should well represent the salient information of the document; information-wise, it should effectively reduce the uncertainty of document x without directly observing it. To empirically measure saliency, we employ masked language model (MLM)- by masking the tokens in the document x to produce Xmakkr then measuring how well an MLM can recover x from maakk when given the summary y. Leveraging this idea, we introduce a saliency critic fs:
fs(x,y) = log PMLM ( Kmaakkr y)
y
TS
(1)
PMLM (x(Xmask)
Concretely, we normalize the score with the likelihood of reconstructing x from m without y, as some masks maybe easily inferred without observing y. The critic operates by filtering out y) pairs with the normalized score lower than threshold, where iş is hyperparameter. The compression ratio /y|||x] in the threshold reflects the trade-off between summary length and saliency-i.e, a longer summary should better preserve the information of the document. Notably, the proposed critic requires only a self-supervised MLM as a proxy model, as opposed to human-superviser critics in Jung et al. (2023)?.
Faithfulness Neural summarizers often suffer from hallucination, i.e., adding information to the summary that was not present in the original document (Chen et al., 2022; Laban et al., 2022). Under our formulation, faithfulness can measured in a reverse direction of saliency, by recovering the summary y from Ymask given the document
fx = log PMLM Mmask x) PMLM ylyskak) TF
Intuitively, masks on y would be easier to infer given if did not add additional infor- mation not x. Unlike the saliency critic, we do not include the compression ratio in the filtering threshold, as expect good summary to be always faithful to the document regardless of its length.
Brevity Finally, summary y should be brief representation of the input x, evaluated by the compression ratio between the summary and the document:
lyl |x|
fu(x,y) = 1
< TB
(3)
cntermation-Maximmzaee Objective Essentially, the saliency and faithfulness critics can both be considered as filtering based on the PMI between x and y, where the two critics differ in how they approximate the mutual information. Specifically, in case of the saliency critic,
PMLM X Xmaskr y)
pxxly) log p(x)p(y)
p PMI(x;y
log
Rs log
PMLM (x) xmast)
p(x)
The same derivation applies to faithfulness critic. Therefore, incorporating all dimensions
above, our objective for distilling text-summary pairs can be crisply described as
Searching for a pair (x,y) of fluent text that maximizes its mutual information pM](x:y), subject to
[x]
<
TB.
Broadly seen, the mutual information between the input data and its compression have been utilized in prior works, primarily as metric for unsupervised text evaluation (Kim
While we do not require specific masking strategy to produce Xmask we find that masking salient keywords identified by TF-IDF (Laban et al., 2020) allows efficient approximation practice. "Specifiially, Jung et al. (2023) employs supervised NIII model as critic for sentence summariza- tion, which does not generalize well to document-level inputs.
3

---

Published as a conference paper at COLM 2024
et al., 2022; Vasilyev et al., 2020) and feature extraction (Padmakumar & He, 2021; Covert et al., 2023). Compared to these approaches, we formulate PMI maximization as a unified objective for abstractive summarization, which can be optimized even with an off-the-shelf LM by rejection sampling with the self-super critics defined above.
2.2 From Off-the-shelf LM to Expert Summarization Model
Our goal in INFOSUMM is start from a small, off-the-shelf teacher LM Minity then generate large-scale summarization dataset Dsumm, which we use to distill an expert summarizer Msunm- Our key idea to self-train the teacher Minit through expert iteration (Anthony et al., 2017) to align with our cetemmation-mxxiizee generator My for summarization prior to distilling a student.
objective, yielding an improved data
Generating Initial Dataset We start by generating an initial dataset Dinit from the off-the- shelf teacher Mimit, by over-geneg candidate socumant-summary pairs with the teacher and subsequently filtering them using the critics defined in $2.1.
To generate the candidate pairs, we take a simple rugo-regressvve approach-we first sample text from Minit, then just take 1-5 leading sentences as a summary of the remaining content, i..,, y Pinit( (-)P), ~ Pimit (-IP.y). Here, P is a simple prefix for better generation quality (e.g., New York, (CNN) - to promote ntws-style generation). We find this approach particularly effective for two reasons. First, it is an easy way to condition the generation of x on y, without fine-tuning the autoregressive teacher Minit Next, the leading sentences often contain the most salient information of the document, hence have been used as useful proxy of summary in previous works (e.g,, Zhu al. (2019)).
A limitation of the autoregressive approach, however, is that it generates a long document conditioned on a handful of sentences in the beginning - as the generation gets longer, easily loses coherence from the original context. To mitigate this issue, we devise decoding algorithm inspired by Product of Experts (Hinton, 2002; Liu et al., 2021) on top of M
y (- X Pina (-IP.)) Pinte (,) a >
55)
By penalizing the unconditional likelihood of the document, we encourage the teacher to attend more to the leading sentences y while generating Note that if we set
X PLM(-IPP
therefore x is generated to maximize its PMI with the summary y. Using the decoding algorithm, we distill the initial dataset Dinit by over-generating candidate set Cinit of document-summary pairs, then filtering it using the critics defined in 52.1:
P
Cinit = {1x1,)1), . ly: ~ Pinit (-IP), X; ~ Pine (- Panic ()) Dinit (x,,)) E Ciniti fs(x,y) A fr(x,y) A (x = =}
(6)
(7)
Distilling Expert Summarizer While the critic models and decoding algorithm effectively implement our search objective, the sampling efficiency of the generated pairs is of central concern when distilling a large-scale dataset. That is, most candidate pairs in Cisit are unlikely to pass the critic filtering stage, our initial teacher model is assumed to be not aligned for summarization.
To resolve the bottleneck of low sampling efficiency, we perform loop of expert itera- tion (Anthony et al., 2017; Silver et al., 2017) on the teacher M where the off-the-shelf LM is supervised on its own generated, high-quality pairs. Concretely, instead of distill- ing a student summarizer Msamm with Dinit, we self-train the teacher Minit to maximize ((x.))-Dmm) [log PM.... x yielding an improved teacher My. By training exclusively on high-quality pairs identified by the critics, the teacher model is gradually aligned with our search objective; we show in the later section, even single step of expert iteration dra- matically boosts the sampling efficiency of the teacher model. Compared to previous works expert iteration, we yield specialized data generator entirely from pre-trained LMs, without resorting to ground-trute references (Zelikman et al., 2022) or a human-superrised verifier (Lightman et al., 2023).
Finally, we distill an expert summarizer Mşumm from the improved teacher My. First, a large-scale dataset Dsumm is distilled from the improved teacher My, following the same
4

---

Published as a conference paper at COLM 2024
Dataset
CNN/DM
XSUM
Model
R-1
R-2
R-L
BERTScore
G-Eval
R-1
R-2
R-L
BERTScore
G-Eval
In-domain Supervision
PEGASUSSIT
44.2
21.5
41.1
88.4
4.14
47.2
24.6
39.3
91.4
4.05
Unsupervised Methods
T1,DR
211.1
5.5
19.6
74.9
3.12
10.8
1.3
8,3
77.8
2.32
ChatGPT
35.0
13.6
86.4
4.47
30.6
9.8
22.5
94.7
4.45
SEQ*
23.2
7.1
81.6
12.3
10.7
80.4
Summary Loop
37.7
14.8
34.7
83.2
3.81
11.7
1.5
9.0
79.2
2.75
TED
38.7
16.8
35.4
Wiki Transferr
INFOSUMM-0.5B
42.0
88.1
4.38
33.4
14.0
28.2
85.5
4.21
Table 1: Quantitative results on news summarization. INFOSUMM-0055 our 568M model distilled from 2.8B teacher, achieves comparable zero-shot performance to prompting ChatGPT. For G-Eval, across the 4 evaluation criteria (coherence, consistency, fluency, and relevance) proposed in the original paper (Liu et al., 2023b). Note that G-Eval is known to have preference bias towards summaries from ChatGPT.
GPT-4 evaluation based 1-5 Likert scale, averaged
process with Dini- Then, we fine-tune a student LM into an expert summarizer Msunm by maximizing ((x,))-Damma [log Paaas (y]x)), i.e., the conditional log-likelihood of given As byproduct of this process, obtain a large-scale, high-quality summarization dataset Dsumm that can be interpreted and reused, e.g., to train a summarization model without re-iterating the overall distillation process.
Endowing Controllability Controllable summarization has recently emerged as an im- portant research direction (Fan et al., 2018; He et al., 2022), allowing users to customize diverse aspects of the generated summary (eg., length and specificity). Under INFOSUMM, controllable summarizer can be trained simply by post-hoc annotating the generated data with the control attributes (for more details, see Appendix B). Moreover, as our framework operates with a small LM data generator, we can down-sample over-generated pairs to increase the diversity of control attributes. After annotating control attributes, student can be trained to be controllable by prepending the control code (Keskar al., 2019) an instruction to the input document.
Experimental Results
3.1 Implementation Details
We start from Pythia-2.8B, an off-the-shelf decoder-only LM the teacher model. Using T5-large (Raffel et al., 2023) as the MLM in the critics, we generate an initial datasett D of 140K news style text-summary pairs. After self-training the teacher model, generate a large scale dataset Dsumm with 4.5M samples, among which 1M pairs are additionally focus
annotated for controllability. In our experiments, attributes - length, extractiveness, specificity, and keywords proposed in Zhang et al. (2023b). We also consider a composition of these control attributes, hence allowing the distilled model to follow fine-grained instructions (e.g., to generate a highly specific, medium length summary focusing on a given keywoord). Using this dataset, we train PEGASUS (Zhang et al., 2020a) with 568M parameters into an expert summarization model. We refer to this model as INFOSUMM-0.57 Further implementation details are in Appendix A.
dimensions of control
3.2 Zero-shot News Summarization
Evaluation Setup We first test INFOSUMM-0.5B for zero-shot summarization on widely used benchmarks, XSUM (Narayan et al., 2018) and CNN/DM (Nallapati et al., 2016). For baselines, we consider state-of-the-art unsupervised summarizers (I.e., trained without human references)-TTe TL,DR prompting (Radford et al., 2019) on Pythia-2.8B, SEQ (Baziotis et al., 2019), Summary Loop (Laban et al., 2020), TED (Yang et al., 2020) and WikiTransfer

---

Published as a conference paper at COLM 2024
8
3
2.5
2.75
2
2.5
1.5
2.26
Overall
Fluency
Faithful
Fluency
Saleet
Supenfsed PEGABUS
Summary Loop intoSiumm
ChaIGPT
ChatGPT Beat of 10 Intoluumm Blest of 10)
ChanGin
Intols mnmt.umm
Figure 2: Human evaluation results. INFOSUMM-0.5B is consistently scored higher than in-domain supervised PEGASUS, and outperforms ChatGPT with simple re- ranking approach (best-of-10). Left: We compare INFOSUMM-0.55 against baselines across dimensions of summary quality. Right: We test best-of-10 approach on top of ChatGPT and INFOSUMM-0.5tH by sampling 10 summaries per document and ranking them using the critic models of INFOSUMM.
Dataset
WikiHow
Reddit
TIFU
Reddit
Model
R-L
G-E
R-L
Model
R-1
8-5
R-I.
B-S
Summary Loop
PEGASUScNN/DM
20.0
19.5
3.59
3.67
17.7
12.4
4.19
PEGASUSSFT
/EGASUSCNN/DM SET
34.8
88.8
21.6
87.6
ChalGPT
21.1
4.45
19.8
4.38
INFOSUMM-O.5D
35.2
90.0
23.0
87.7
INFOSUMM-0.5B
22,9
4.35
17.9
4.26
Table 3: INFOSUMM is effective for transfer learning. After fine-tuning, our model bet- ter matches the reference of each benchmark than PEGASUS, measured by ROUGE-L (R- L) and BERTScore (B-S).
Table 2: INFOSUMM better generalizes to unseen domains than human-supervised PEGASUS. We report ROUGE-L (R-L) and G-Eval (G-E) on WikiHow and Reddit do- mains.
Fabbri al. (2021a), as well as zero-shot prompted ChatGPT (gpt-3.5-turbo) We also consider an in-domain supervised baseline PEGASUSSFT: fine-tuned on the respective train sets of the benchmarks. For metrics, we report ROUGE, BERTScore (Zhang et al., 2020b) and average G-EVAL (Liu et al., 2023b), reference-less metric based on GPT-4 known to better correlate with human judgements of summary quality.
Results We present our quantitative results in Table INFOSUMM-0.5B significantly unsupervised baselines including Summary Loop
outperforms summarization-speiiin trained with in-domain articles of CNN/DM, and WikiTransfer that leverages the stylistic bias of each benchmark (hence not considered to be strictly zero-shot). Overall, our model marks similar performance across metrics with ChatGPT, an order of magnitude larger, human-aligned L.M.. In GPT-4 evaluation XSUM, it even outperforms PEGASUSgFT. the same base model PEGASUS supervised on the in-domain references.
Human Evaluation To better compare the quality of model-generated summaries, we conduct human evaluation. We generate summaries for 200 CNN/DM articles with each system, then ask annotators
score their fluency, faithfulness and saliency following Stiennon et al. (2022). To adjust the confounding effect of summary length, we sample only those articles for which all systems output summaries with the same number of sentences. To minimize subjectivity, we use strict 3-level Likert scale, leading to high inter-annotator agreement (Krippendorff's alpha=0.65; substantial agreement).
The left part of Figure 2 presents the results. We find that summaries from INFOSUMM-0.5B outperform PEGASUSSET across all dimensions, and are considered to be more faithful than ChatGPT generated summaries. In Appendix E.2, we additionally conduct pairwise human evaluation of INFOSUMM against the baselines. We find that summaries from INFOSUMM are at least of equal quality with ChatGPT for more than 80% of the time, and are preferred to PEGASUSSFI for more than 50% of the time, demonstrating the strong performance of our distilled model.

---

Published as a conference paper at COLM 2024
3.3 Generalizing to Unseen Domains
Zero-shot Generalization Next, we evaluate models on their generalization capability to benchmarks in unseen domains, specifically for WikiHow (Koupaee & Wang, 2018) and Reddit TIFU (Kim et al., 2019). We compare INFOSUMM-0.53 against three zero-shot baseline summarizers: Summary Loop, PEGASUSCNN/DM fine-tuned on CNN/DM train set, and zero-shot prompted ChatGPT.
In Table 2, we find that INFOSUMM-O.55 outperforms strong models in this setup, gener- ating similar quality summaries as prompting ChatGPT. Notably, the results imply that INFOSUMM generalizes better than training on human-authored datasets: INFOSUMM-0.55 trained on our distilled summaries, performs better in unseen domains than the same base model PEGASUS, fine-tuned on human-written summaries of CNN/DM.
Fine-tuning Unseen Domains One strength of a compact expert model is that can be fine-tuned, to better follow the specific style and domain of benchmark references. This motivates us to consider another use-case of INFOSUMM, where a model is first distilled into performant summarizer using synthetic data, then is further supervised on human- authored references to better adapt to an unseen domain.
In Table 3, while initial fine-tuning on CNN/DM PPEGASUSCNN/DM-SS degrades the final model performance on unseen domains, INFOSUMM-O.. improves over vanilla PEGASUS after fine-tuning. 3 We attribute this to the relatively narrow distribution of summary styles in commonly used summarization datasets (Tejaswin et al., 2021), including CNN/DM, As we show in $3.5, our dataset exhibits substantially larger scale, more extensive coverage of the summary space compared to the existing benchmarks, allowing the model to readily adapt to a specific summary style through fine-tuning.
3.4 Controllable Summarization
As the final application of INFOSUMM, we evaluate our model on controllable summarization. We use MACSum-Doc dataset (Zhang et al., 2023b), where sum- maries are collected from human annota- tors across 4 control dimensions: length (short, medium, long), extractiveness (low, medium, high), specificity (medium, high) and keywords. For baselines, we consider PEGASUSMACSum fine-tuned with the gold references MACSum train set, along with zero-shot / few-shot prompted ChatGPT with 5 demonstrations sampled from MAC- Sum train set (for few-shot prompting, we use 3pt-3.5-turbo-16kt To evaluate the controlla- bility over length, extractiveness and specificity, we report control correlation (Zhang et al., 2023b), measuring how well a summary follows the given control code. In addition, we conduct human evaluation to assess the keyword usage and overall quality of generated summaries. See Appendix and C for further evaluation details.
Control Correlation 1
Human Eval
Model
Len.
Ext.
Spe.
Keywonl
Ovenall
PEGASUSNACCom
30.6
0.18
0.48
1.53
1.66
ChaiGPT
29.8
0.01
0.07
2.65
2.58
ChaiGPTs
32.8
0.12
0.35
2.71
2.75
INFOSUMM-O.5B
37.6
0.16
182
2.71
2.70
Table 4: Results on controllable summariza- tion. INFOSUMM-0.5B achieves better control- lability across summary length, extractiveness, specificity than 5-shot prompted ChatGPT or human-supervised model.
INFOSUMM-O 5B significantly outperforms baselines in controllability across dimensions. While large-scale supervision crucial for reliable controllability, human-authored train set is hard to scale. Accordingly, PEGASUSMACSum fine-tuned on the 4K samples of MACSum train set yields sub-optimal performance, although the references were curated by humans. Meanwhile, our model better correlates to the control codes than ChatGPT, even when the LLM was given 5-shot in-domain demonstrations. The result substantiates that while a textual description of constraints could signal some degree of control to LLMs, it may not suffice to control more sparse and fine-grained composition of attributes (Chen et al., 2023).
We focus on reference-based evaluation, as fine-tuning does not improve G-Eval for both PEGASUSCNN/DD and INFOSUMM-D.5B.
7

---

Published as a conference paper at COLM 2024
Length Medium
Length Mesium
Short
1 lon
Long
1ow
12.2%
7.4%
2.7%
LOw
B.4%
2.5% (1 1333)
1.0% (45k)
(5-18K)
(333K)
(122K)
(377K)
Extractivenss Mod
3.7%
8.4%
7.2%
High Medkum
3.8%
5.5%
4.8%
(378k)
(324k)
(248k)
High
3.9%
3.3%
5.7%
11.2%
(90K)
(176k)
(355k)
(149k)
(257k)
(504k)
Medium Specificity
High Specificity
Figure 3: Summary style distribution of the distilled dataset from INFOSUMM. Compared to authn-authored datasets (Appendix SE.4), our dataset entails substantially diverse coverage of summary styles, leading to more robust and generalizable student.
3.5 Additional Analyses
The critic models defined in $2.1 play pivotal role in INFOSUMM, identifying high-quality pairs for distillation. Thus, it is reasonable to ask whether the critic models are all we need, i.e., strong performance can simply be achieved by re-ranking summaries generated from existing systems (e.g,, ChatGPT). To validate this hypothesis, we test best-of-10 approach on top of ChatGPT and INFOSUMM-0.5B we first sample 10 summaries per document, then output the generation with the largest sum of faithfulness and saliency score.
Re-ranking summaries
In human evaluation (Figure and Appendix E.1), best-of-10 on ChatGPT slightly improves its faithfulness and saliency, but undermines the fluency of the generated summaries. This is in contrast to our method, where best-of-10 consistently improves in all evaluated aspects. The results show that (1) optimizing for PMI maximization indeed improves the human perception of faithfulness and saliency for both LLM and INFOSUMM generated summaries, but (2) unlike our model, moderate sample size of 10 may not be enough for meaningful exploration with ChatGPT, as the model has not been trained to align with our search objective.
Analyzing Data Diversity We directly evaluate the diversity of the distilled dataset against human-curated benchmarks widely used for news summarization. We first compare the summary length statistics and lexical diversity of each dataset (Table 5). To measure lexical diversity, we follow Jung et al. (2023) to report 2/3-gram en- tropy, along with mean segmented token type ratio (MSTTR; Torruella & Capsada (2013)) of the summaries. In addition, we analyze the summary style distribution of each dataset (Figure 3 and Appendix E.4), by categorizing summaries into 18 style groups proposed in MACSum.
Length
Lexical Diversity
Dataset
Median (std)
Hg
Hj
MSTIR
XSUM (0.2M)
23.0 (5.7)
16.9
19.9
55.5
CNN/DM, DM (0 (0.3M)
51.0 (22.9)
18.4
22.0
54.3
Gigaword (38M)
(2.8)
21.2
Ours (4.5M)
58.0 (26.7)
18.8
23.2
57.7
Table 5: INFOSUMM yields a high-quality dataset with larger scale, more diverse sum- maries than existing benchmarks.
The results demonstrate that our dataset, in sample size but also is substantially more diverse than existing datasets. As shown in Appendix F.4, human-authored datasets are typically skewed to narrow region of style distribution-in XSUM, more than 70% of the reference summaries fall into short, less extractive and less specific group. Our dataset, the other hand, covers significantly broader region of summary styles, along with consistently higher lexical diversity.
purely synthetic corpus, is not only larger
Analyzing Sampling Efficiency In Appendix E.3, we conduct an ablation study INFOSUMM, specifically focusing on the sampling efficiency the framework (i.e., the ratio of candidate summary pairs that pass all the critics). We leave the full results in Table 10, and sunmarize the results here. First, we find that the sampling efficiency initial teacher prior to expert iteration is only 0.9%, indicating the importance of expert iteration for large-scale
8

---

Published as a conference paper at COLM 2024
distillation. However, we also find that multiple steps of expert iteration can over-optimize the teacher, leading to less diversity in generated data despite better sampling efficiency. We also find that PMI-maximizztion decoding (Eq. 5) improves the sampling efficiency by 4% compared to temperature-based sampling, representing its usefulness as inference-time algorithm to efficiently search for high-quality samples. See Appendix 3.5 for additional ablation results that focus on distilled model performance.
Analyzing Effect of Expert Iteration We also test whether expert iteration indeed improves the performance of the distilled summarizer. Specifically, we train an addi- tional summarizer by directly fine-tuning PEGASUS-large on Dinit, the initial dataset of 140k document-summary pairs gener- ated from the off-the-shelf teacher Mini-
Dataset
CNN/DM
XSUM
Model
R-L
G-E
R-L
G-E
No expert iteration
35.8
4.07
82.5
4.05
INFOSUMM-0.5B
38.4
4.38
28.2
4.21
Table 6: Ablation results on expert iteration.
The performance of this configuration against the full INFOSUMM is shown in Table 6. INFOSUMM, trained with the large-scale summarization dataset generated by the improved teacher, yields consistently better scores in both reference-based (ROUGE-L) and reference- free evaluation (G-Eval). Surprisingly however, compared to the various unsupervised summarization models in Table 1, our model trained without expert iteration already outperforms majority of baselines in both CNN/DM and XSUM. The result shows that while expert iteration clearly benefits the student by scaling the training data, distilling with the information maximizing objective is as important to yield a reliable summarizer.
Does PMI align with human evaluation? We have shown through best-of-n analysis that optimizing for the PMI between the document and summary improves human evaluation results. In Appendix D, we further verify this by comparing the human-judged quality of references in XSUM against the PMI values estimated by the two critics of INFOSUMM. Overall, we find that the estimated PMT is an excellent predictor of human-evaluater quality, especially in the two tails of the score distribution (i.e., when the pair is certainly of low- quality or high-quality). We also find that PMI estimation can often reveal annotation error in the widely-used dataset, indicating that our proposed objective can serve as a useful tool for filtering high-quality data for summarization.
4 Related Works
Unsupervised Summarization Prior approaches to unsupervised abstractive summariza- tion have focused on devising proxy task - e.g., reconstruction of the original document- that may guide the model towards the desired behavior (Baziotis al., 2019; Févry & Phang, 2018; Laban et al., 2020). While these methods typically require carefully designed training loop or reward shaping process (Yang et al., 2020), their performance often fall behind supervised models. Apart from the conventional methods, recent findings report that L.LMs such as ChatGPT excel at summarization, surpassing the quality of suman--uperrised models without task-specific fine-tuning (Goyal et al., 2023; Zhang et al., 2023a). Subsequent works also show that a compact summarizer can be trained by imitating LLM-generated summaries (Sclar et al., 2022; Xu et al., 2023). Beyond LLM distillation, Jung et al. (2023) shares similar motivation to ours, presenting Impossible Distillation that distills strong task model from small, off aff-the-shelf teacher model. While Impossible Distillation is only applicable to sentence level tasks and requires supervised NLI model, we target more complex task of abstractive summarization with document-level inputs and operate entirely without human supervision.
Beyond summarization, a growing line of works proposes to directly generate a dataset using language models, tailored to specific domains and tasks such as commonsense knowledge (West et al., 2022; Brahman et al., 2023), mathematical / textual reasoning (Yu et al., 2023a; Mukherjee et al., 2023) and social dialogues (Kim et al., 2023). Nonetheless, challenges abound in automatic data generation - while the quality of data is critical factor in downstream performance (Gunasekar et al.,
Generating Data with Language Models

---

Published as a conference paper at COLM 2024
Police are searching for two missing teens believed to be together who disappeared after reportedly threatening to hurt themselves. Erika R-, 14, of Holiday and Caleb B-, 13, of Tampa are dating and it is suspected that she was picked up in a car on her way to Tampa with him, said police. B-, a white male, left his home near Dale Saturday morning, and police are concerned
Mabry Avenue and Lois Avenue due to his age as well as threats of him harming himself and his girlfriend, according to ABC Action News. Erika R- (left), 14, of Holiday and Caleb B B- (right), 13, of Tampa are dating and it is suspected that she was picked up in a car on her way to lampa with her boyfriend, said police. Authorities are searching for the teens who disappeared after reportedly threatening to hurt themselves It is not known what he was wearing when he left his residence. R-,a white female, was last seen at her home around 11.30pm on Friday wearing a long-sleeve t-shirt, low-top gray Converse sneakers and possibly blue jeans. The teen girl, who also threatened to harm herself, took $200 from her mother's 's purse along with her school backpack before leaving her residence on Westchester Drive, according to WTSP. She has scars on both arms, on her upper thighs and under her armpits from self-mutilation, as well as a red mole on the left side her forehead. The teen girl (R- pictured above), who also threatened to harm herself, took $200 from her mother's s purse along with her school backpack before leaving her residence on Westchester Drive. She has scars on both arms, on her upper thighs and under her armpits from self-mutilation as well red mole on the left side of her forehead B- (above), white male, left his home near Dale Mabry Avenue and Lois Avenue on Saturday morning, and police are concerned due to his age as well as threats of him harming himself and his girlfriend. R- is 5'6' tall, has auburn hair, light brown eyes and is 120lb. B- is 5'4' tall, has brown hair and is 130lb. Pasco Sheriff's Office spokeswoman Melanie Snow said R- moved to Tampa from Holiday about three weeks ago, according to the Tampa Bay Times. She said: "We don't 't think that she is in danger, but she is only 14 years old and away from home on her own accord. Her mother wants her home."
Document
Police are searching for two missing teenagers, Erika R-, 14, of Holiday and Caleb B- 13, of Tampa, who disappeared after reportedly threatening to harm themselves. R- was last seen her home around 1 11:30pm on Friday wearing a long-sleev t-shirt, low-top gray Converse sneakers, and possibly blue jeans. B- left his home near Dale Mabry Avenue and Lois Avenue on Saturday morning, and police are concerned due to his age as well as threats of him harming himself and his girlfriend.
Summary
Table 7: Unconstrained summary generated by INFOSUMM-0.5B, an entirely self-supervised summarizer with 568M parameters. We randomly sample a CNN/DM article and anonymize names of non-public figures in the table. More samples Appendix F.
2023), even the strongest LMs suffer from unexpected mistakes mistakee (Jones et al., 2023; Jung et al., 2022) and lack of diversity in its generations (Yu et al., 2023b). Several techniques have been sampling (Uesato et al., 2022; Lightman et al., 2023) and attributed prompting (Yu et al., 2023b; Yue et al., 2023; Eldan & Li, 2023), albeit in limited domains. Aligning with these works, we show that an explicit formalization of the target task can significantly boost the quality of generated dataset, and further demonstrate that prompting a larger, stronger LLM is not the only way to distill performant summarization model.
introduced to improve the data quality, such as verifier-guided
5 Conclusion
We propose INFOSUMM, a novel method to distill a performant summarizer based on the information maximizing objective for summarization. We find that INFOSUMM, without either human annotated references or gigantic LLM,, often outperforms the strongest base- lines across benchmarks in news summarization, zero-shot fine-tuning adaptation to unseen domains, and controllable summarization. In addition, we produce a large-scale summarization dataset as a byproduct of INFOSUMM, demonstrating the most extensive coverage of summary style and lexical diversity compared to existing benchmarks widely used in prior works. INFOSUMM demonstrates way to distill powerful summarizer based on how we formally characterize summarization, rather than how a teacher model behaves when instructed for summarization.
10

---

Published as a conference paper at COLM 2024
References
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search, 2017.
Christos Baziotis, lon Androutsopoulos, loannis Konstas, and Alexandros Potamianos. autoencoder for unsupervised
SEQ'3: Differentiable abstractive sentence compression. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 673-681, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 111665//11/N19-1077 URL https: / aallattonannooo org/N19-1011
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. Plasma: Making small language models better procedural knowledge models for (counterfactual) planning, 2023.
Xiang Chen, Xiaojun Wan, and Xiang Wan. A comprehensive evaluation of constrained text generation for large language models, 2023.
Xiuying Chen, Mingzhe Li, Xin Gao, and Xiangliang Zhang. Towards improving faithfulness in abstractive summarization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Adoances in Neural Information Processing Systems, 2022. URL https://openrevieee
lan Covert, Wei Qiu, Mingyu Lu, Nayoon Kim, Nathan White, and Su-In Lee. Learning to maximize mutual information for dynamic feature selection. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. Improving zero and few-shot abstrac- tive summarization with intermediate fine-tuning and data augmentation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 704-717, Online, June 2021a. Associ- ation for Computational Linguistics. doi: 10.1 10.18653/v1/22 990665//11//21..mmmmmm...77 tttps://aalaattolleee hwwpithhooogg.oor/22222 prg/2021.o0 naacl-main.
URL
Alexander R. Fabbri, Wojciech Kryšciñski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation, 2021b.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Alexandra Birch, Andrew Finch, Thang Luong, Graham Neubig, and Yusuke Oda (eds.), Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp- 45-54, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: URL htttps:/aaclanthooogg...
Thibault Févry and Jason Phang. Unsupervised sentence compression using denoising auto-encoders. In Anna Korhonen and Ivan Titov (eds.), Proceedings of the 22nd Conference Computational Natural Language Learning, pp. 413-422, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: ww211653/v1/KKi-11 URL httt:::///cllathhoooooooogg////e
11

---

Published as a conference paper at COLM 2024
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3, 2023.
Maarten Grootendorst. Keybert: Minimal keyword extraction with bert., 2020. URL https://doi, org/10. 5281/zenodo 4461265.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better, 2023.
Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. Towards generic controllable text summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5879-5915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 v1 h 122/222eemmlmmmii.339. URL https: / /alaaalaaanooooggo..o.mom org 2022. emnlp-main.33
CTRLsum:
Geoffrey E Hinton. Training producss of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002. 2002.
Erik Jones, Hamid Palangi, Clarisse Simôes, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Awadallah, and Ece Kamar. Teaching language models to hallucinate less with synthetic tasks, 2023.
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022.
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing, 2023.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks, 2019.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. Soda: Million-scale dialogue distillation with social commonsense contextualization. 2023.
Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. Mutual infor- mation divergence: A unified metric for multimodal generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Infor- mation Processing Systems, 2022. URL htttt:://oppnreviii
het//Forumidd<xKo2Xt555
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset, 2018
Klaus Krippendorff Computing krippendorff's alpha-reliability annenberg school for communication departmental paper 43, 2007.
12

---

Published as a conference paper at COLM 2024
Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. The summaay loop: Learning to write abstractive summaries without examples. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Linguistics. doi: 10. 0011663/v1/2000aalmmaiill /2aa://m222200mmmmma40000000 2020 acl-main, 460.
Online, July 2020. Association for Computational
URL https://aclanttiiit
org/
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re- visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163-177, 2022. doi: 10.1162 att//aaalaaaaaaa a.00553.
URL https: http:: http::///acaaathollooo org/ www.//g///22/. wttgg222..aacl1..11
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-tingg time controlled text generation with experts and anti-experts. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691-6706 Online, August 2021. Association for Computational Linguistics. doi: 10.18653 wtt.///1.2allloon..555555 URL https:://cclntthoo//
org/2021 acl mell-long.a 522.
Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general-purpose plausibility estimation model for commonsense statements, 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nig evaluation using gpt-4 with better human alignment, 2023b.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summarization, 2022.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang- Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Stefan Riezler and Yoav Goldberg (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, Berlin, Germany, August 2016.
Association for Computational Linguistics. doi: 1011865//1/K16--1 //aclanthology org/Ko6- K
URL https:
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujü (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-Novembe 2018. Association for Computational Linguistics. URL https: // aclanthologgyy org/Di8-1200
doi: 10.18653 116655/v11/D8-111
Vishakh Padmakumar and He He. Unsupervised extractive summarization using point- wise mutual information. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfatt (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, Pp. 2505-2512, Online, April 2021. Association for Computational Linguistics. doi: 10.18653 /v///vvv/// wet.///eeecllamai.i... 213. URL https: //aclanthologg.org//222 eat.:/e.al-maan.... 213.
13

---

Published as a conference paper at COLM 2024
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. In OpenAl, 2019. URL https: //api. seap.assaannii .ww.arorg/ooooeeo org//orpusID: 160025533.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023.
Melanie Sclar, Peter West, Sachin Kumar, Yulia Tsvetkov, and Yejin Choi. Referee: Reference- free sentence summarization with sharper controllability through symbolic knowledge distillation, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, L. Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354 359, 2017. URL https://api. semantion cwcholarrcc.....r.ommmmrrraar...
2052660034
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Priyam Tejaswin, Dhruv Naik, and Pengfei Liu. How well do you know your summarization datasets? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3436-3449, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/20211111100/ wtt.://...niiingeeee. findings--cc.333. URL https: / aalaathoooogg org/2021, 2021. findings-ane ac1.303. .033.
Joan Torruella and Ramon Capsada. Lexical statistics and tipological structures: measure of lexical richness. Procedia Social and Behavioral Sciences, 95:447-454, 10 2013. doi: (0011066/.sbspr.2013..kk
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-basee feedback, 2022.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. Fill in the BLANC: Human- free quality estimation of document summaries. In Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, and Eduard Hovy (eds.), Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pp. 11-20, Online, November 2020. As- sociation for Computational Linguistics. doi: 1111663//112220 https://aclanthonn
URL
org/2020. eval4nlp-1.2.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 4602-4625, Seattle, United States, July 2022. Associ- ation for Computational Linguistics. doi: 10. 1118663/1l.. /tt/://0222mmaccmmaann https://aclanthologs org/2022..02/. naacl-main.
URL
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021.
Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu, Shuohang Wang, Chenguang Zhu, and Michael Zeng. InheritSumm: A general, versatile and compact summarizer by distilling from GPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13879-13892, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 2023 findings-eenllp.222 URL https: /ate:://cccla /aaclalttoooogyy.....omm .rrg 2023 wwtp:/023..nnneeeeeeeee findings-emmlp.222
14

---

Published as a conference paper at COLM 2024
Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. TED: A pretrained unsupervised summarization model with theme modeling and de- noising. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1865-1874 Online, November 2020. Associ- 168. URL
ation for Computational Linguistiis. doi: 10.1 10.18653/v1 2ww...inniings--ieeee www./o20.........oo wwwdingg--ennlll.. emnlp.162
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023a.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias, 2023b.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2020a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi, Bertscore: Evaluating text generation with bert, 2020b.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summarization, 2023a.
Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. Macsum: Controllable summarization with mixed attributes, 2023b.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. Make lead bias your favor: A simple and effective method for news summarization. ArXiv, abs/1912.116022 2019. URL https: //api. semanticscholar sem.aniicccala....oorrrrrraaaaaa 2095500822
15

---

Published as a conference paper at COLM 2024
Acknowledgements
This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001- 19-2-4031), ONR N00014-24-1-2207 and IARPA HIATUS via 2022-22772200003.
A Implementation Details
A.1 Generation Stage
We start off from Pythia-2.8B as the initial teacher model. Following,Jue Jung al. (2023), we use a simple prompt P formatted as "(City Name), ((Media Name)) -" in order to generate news-style summary and article. 23 media names and 984 city names were collected by the authors to automatically generate diverse P. We find that this way of prompt construction not only encourages the fluency of LM, but also significantly improves the diversity of generation compared to sampling without prompt. While generating both the summary and article, we use top-p threshold of 0.9 with temperature 1.0. We first generate summary by sampling 1-5 sentences conditioned on P, where the number of sentences are randomly chosen. Then, the article is generated by autoregressively conditioned on both and the summary. We do not fix the number of sentences in the article, and generate until the max number of tokens are reached (1024 for our experiments).
A.2 Filtering Stage
We use T5-large, a masked language model with 770M parameters as the backbone for the faithfulness and saliency critics. To determine critic thresholds, we run series of small-scale experiments to generate 1K samples from MLM and manually inspect the sunmary quality. Based on the results, we set Ts = log 14, TF = log 1.7 and тş = 0.2. Intuitively, Tş = log 14 constrains that when the summary length is 10% of the original document, the likelihood of accurately inferring the masked tokens has to be 40% higher when an MLM is provided with the summary. We qualitatively find that while the high threshold leads to low sampling efficiency with the initial teacher, it improves the quality of the distilled dataset and hence leads to better performance of the end-stage student model.
A.3 Training and Evaluation
After expert iteration, we train PEGASUS-large, an encoder-decoder pre-trained LM with 568M parameters the 4.5M samples distilled from the improved teacher My. We fine- tune the model for 2 epochs with batch size 64 and max target tokens = 128. For all other hyperparameters, we use the same setting as chosen in the original paper (Zhang et al., 2020a).
For our main experiments, we report ROUGE, BERTScore along with G-Eval. G-Eval is model-based metric computed by prompting an LLM (e.g., ChatGPT, GPT-4) with chain- of-thought style prompt for text evaluation. Specifically, we use GPT-4 as the base LM for G-Eval, averaging 1-5 Likert scale scores averaged across 4 dimensions (coherence, consistency, fluency and saliency), following the setup the original paper (Liu et al., 2023b). To instruct GPT-4, we use the prompt in the official implementation. Although G-Eval shows substantially higher correlation with human judgements of summary quality compared to conventional metrics, Liu et al. (2023b) also reports that GPT-4 is biased toward LLM generations, assigning higher scores to summaries from ChatGPT than the baselines (including human-authored summaries). Indeed, we find that ChatGPT consistently obtains the highest G-Eval score among all baselines throughout our experiments, even though it underperforms the baselines in reference-based metrics.
16

---

Published as a conference paper at COLM 2024
B
Controllable Summarization Details
B.1 Distilling for Controllable Summarization
INFOSUMM can be extended to controllable summarization, by additionally annotating the generated summaries with the corresponding control attributes. We use 4 dimensions of control attributes proposed in Zhang al. (2023b), iee, length, extractiveness, specificity and keywords. For length, extractiveness and specificity, we follow the original setup of MACSum define the respective metric function Mattr that maps each summary its corresponding scalar value. Specifically, for length, Jen is the number of tokens in the summary. For extractiveness, Mext is the average of ROUGE-2 precision and ROUGE-3 precision of the generated summary against the imput document. For specificity, Mspe is defined as (0.1 x vb 0.2 x tok + 0.3 x ni + 0.4 / where vb, tok, 1i1,, cd represent the number of verbs, tokens, nouns and numerical tokens, and |x|s denotes the number of sentences in the summary. For keywords,
extract 1 or keywords from the summary identified by an off-the-shelf keyword extraction tool (Grootendorst, 2020).
Based on the computed values from each metric function, we annotate summaries according to their length (short, medium, long), extractiveness (low, medium, high) and specificity (medium, high), followed by keywords in each summary. For example, to annotate the length of summary we define
short, if my)n(y) Teen,1 length(y) medium, if Ten, S Meen (y) Teen2
(8)
long, otherwise
Ideally, the thresholds I and 12 should reflect how humans perceive the control attributes e.g., how short sunmary should be in order to be perceived as short by humans. To this end, define the thresholds based the statistics of human-written summaries in MACSum train set. For ex- ample, the threshold Ten,1 between the short and medium length summaries is defined as the median length of short summaries and medium length summaries authored by hu- man annotators. The specific values of the thresholds are as shown in Table
Control attributes
Ti
72
Length Extractiveness Specificity
38
69
0.34
0.51
4.8
Table 8: Threshold values de- fined for control attribute an- notation.
We annotate 1M subset of distilled dataset following the above process. Then, we train the student model on the distilled dataset, to generate controlled summary when it is prompted with specific instruction for control attributes (e.g. Generate long summary with low extractiveness and high focusinn
focusing on given keywordss. We provide examples of controlled summaries generated by INFOSUMM-0.5B in Appendix F.
B.2 Evaluation for Controllable Summarization
For quantifiable attributes i.e., length, extractiveness and specificity, we report the control correlation (CC) of each summarization system. Control correlation measures how well system follows the given instruction for a specific control dimension. Specifically, for control attribute (egg,, length) with a control value pair [i, 22] (egg,, short, medium), we first generate two sunmaries 1/1 and 1/2 for the same document but with the different control values ii and 22 while all other attributes are unchanged. Then, CC is defined as
mattr my1tr mattr (y2)
CCattr
(21 CCattr(y1.
(9)
d( v2)
where d(v),o2) defines the distance between the two control values, e.g., d(short, medium) = 1, d(long, ((long,shorrt) = 2. Note that CC can be either positive or negative; when CC is negative, it indicates that the model has negative correlation with the control instruction. We evaluate the system's average CC over control dimension as the arithmetic mean over all samples in MACSum test set. For keyword and overall summary quality evaluation, we conduct human evaluation; details in Appendix C.
17

---

Published as a conference paper at COLM 2024
C
Human Evaluation Details
Task
morrdiioe siocts oxui stollpap)
the 18T. sated puoee oute ou macenn wu geve gew.tai summinyyy of decument
-B0 villo
anted hollowing cuanticee meliaate
Fluengy Jurt iffrrnd sunmarg?
The Orev sumene
pummen anterneey
Aarteae tuczA Loc fa th moaug
Ti and rerorrraii in canca pocomens your par pusu 90 meare PAR
trorustonss brict erpad tollape))
parteoppttng
rhe 18T, yount sskedttz wallame at suilty a begword coaddd cumonary of decumene.
lorpoard
upyh sareellig gro protgrennen resereeg
anysh colloooooo
fomsution desuunee.
gvenleyywoeed
inforation
Figure& 4: (Upper) Human evaluation template for unconstrained summary evaluation.
(Lower) Human evaluation template for controllable summary evaluation.
For both unconstrained summary evaluation and controllable summary evaluation, human annotators are recruited from Amazon Mechanical Turk (MTurk) with an IRB approval. In unconstrained news summary evaluation, we generate summaries for 200 CNN/DM articles using each system, then ask annotators to evaluate them across 3 evaluation dimensions (fluency, faithfulness and saliency). For controllable summary evaluation, we sample 200 documents (along with the control attributes) from MACSUM-Doc test set. We ask annotators evaluate the generated summaries for their keyword usage and overall quality (averaged across fluency, faithfulness and saliency). We compensate annotators with an hourly wage of $20.
18

---

Published as a conference paper at COLM 2024
D
Does PMI align with Human Judgements of Summary Quality?
Falthitunees (Hurman Evai)
Sallency (Humann Eval)
2
Estimated PMI Saleecy
Eatimated PMI Farthuiness
0.6
1.2
1.8
2.4
0.75
Estimated PMI (fs)
Estimated PMI (fp)
Figure 5: Human-evallated quality of summary vs. estimated PMT by the saliency and faithfulness critics in INFOSUMM.
In this section, we further verify that PMI maximization under length constraint aligns with the human-perceived quality of the summary. Following the same setup as in the main section, we conduct human evaluation to evaluate the quality of 100 reference summaries random-sampled from XSUM dataset. Then, we plot the human-judged saliency (faithful- ness) of each summary against the PMI estimated by our proposed saliency (faithfulness) critic. We specifically choose XSUM because its reference summaries are bounded to be single sentence, making it easier to control the confounding effect of length.
We present the results in Figure 5. In both dimensions, the estimated PMI exhibits positive correlation with the human-judged quality of reference summaries; maximizing PMI leads to more salient and faithful summaries, judged by humans. In addition, particularly low value of estimated PMI often indicates annotation error - egg, the reference summary is completely irrelevant to the document, stating "All images are copyrighted", This finding shows that even the widely-used summarization benchmarks are noisy, and PMI estimation can serve a useful tool for data cleaning prior to supervising summarizer.
find that
Notably, PMI becomes a better predictor of the human-judged quality in the two tails of the score distribution (i.e,, when the documeent-summary pair is certainly low-quality or high-quality), while the prediction gets slightly noisier in the middle range this is expected, as even the human annotators show less agreement for the pairs with ambiguous quality. In fact, the result supports our choice of expert iteration as the learning algorithm while directly optimizing for PMI with online learning may include training with the noisy reward in the middle, our method discards those pairs with susceptible estimated quality, training with only the high-quality samples we are more confident about.
19

---

Published as a conference paper at COLM 2024
E
Additional Results
2.75
2.25
Overall
Fluency
Saleent
PEGARUS InfeSuumm
PEDAASIS
alfoSumm (Boot
Figure 6: Additional human evaluation results with best-of-10 on top of PEGASUS
E.1 Re-ranking summaries from the human-supervissd model
In Figure 6, we provide human evaluation results of best-of-10 summaries sampled from PEGASUSsFI, a supervised model trained on CNN/DR /DM train set, and compare it against INFOSUMM. While re-ranking the supervised model's summaries consistently improves the quality of summary across fluency, faithfulness and saliency, it still substantially falls behind INFOŠUMM, indicating that the proposed information maximizing objective provides better supervision signal than imitating reference summaries of CNN/DM dataset.
E.2 Pairwise human evaluation
To better compare the quality of sum- maries, we provide pairwise human eval- uation results. Following the setup of prior works (Goyal et al., 2023; Stiennon et al., 2022), we ask annotators to de- termine which summary is better. We use 200 CNN/DM articles and compare INFOSUMM-0.5B, ChatGPT and in-domain supervised PEGASUSŞFT, with and without re-ranking.
re-ranking: left wins ties loses (%)
INFOSUMM vs. ChatGPT
INFOSUMM vs. PEGASUSSET
14.5 70.2 / 15.3
50.5 26.0 23.5
Best-of-10: left wins ties loses (%)
INFOSUMM ChatGPT
INFOSUMM PEGASUSSET
19.9 /4.5 / 15.6
52.0 23.2 24.8
Table 9: Pairwise human evaluation results on CNN/DM.
The results are shown in Table 9, We find that summaries from INFOSUMM are evaluated to be at least equal quality with ChatGPT for more than 80% the time, both with and without re-ranking. Our model is consistently preferred to PEGASUSSFT supervised on CNN/DM train set, winning for more than 50% of the samples in both settings.
E.3 Sampling Efficiency Analysis
We conduct an ablation study on INFOS- UMM, focusing on the sampling efficiency of the framework in Table 10. Concretely, we quantify the sampling efficiency as the ratio of candidate summary pairs generated by the teacher that pass all the critics.
Model
Sampling Efficiency
expert iteration expert iteration steps No PMIlmaximization decoding
0.9%
59.8%
54.5%
INFOSUMM
58.5%
Table 10: Sampling efficiency analysis on IN-
First, we ablate the expert iteration and di- rectly measure the sampling efficiency of the initial teacher Minit (No expert iteration). As expected, the off-the-shelf LM's sam- pling efficiency is near zero, indicating the importance of expert iteration for large-scale distillation. However, we also find that multi- ple steps of expert iteration can lead to over-optimizing the data generator. While steps of expert iteration yields slightly better efficiency than single step (2 expert iteration steps),
FOSUMM. Sampling efficiency of each config- uration defined as the ratio of the generated pairs that pass the critics.
20

---

Published as a conference paper at COLM 2024
qualitatively find that it significantly reduces the diversity of generated data due to over-fitting the teacher model. We also consider an ablation (No PMI-maximization decoding) our decoding algorithm by replacing it with Nucleus Sampling. The sampling efficiency in this case degrades by 4% than the original pipeline, attesting to the usefulness of the inference-time algorithm to efficiently search for high-quality samples.
E.4 Baseline summary style distributions
In Figure 7, plot the summary style distribution of 4 widely-used summarization datasets CNN/DM, XSUM, Gigaword and WikiHow. Note that all these datasets were curated by humans and are of large-scale, consisting of least 200K train samples up to 3.8M samples in Gigaword. Nonetheless, compared to INFOSUMM, the reference summaries in each dataset are skewed to distinctive regions of summary style.
Generation Samples
In Table 11-14, we provide unconstrained controlled summaries generated by INFOSUMM- 0.5B for non-cherry-picked XSUM, CNN/DM and WikiHow documents. To demonstrate controllability, we provide the control instruction (if applicable) along with the correspond- ing summary to the document.
21

---

Published as a conference paper at COLM 2024
Length Medium
Length Medium
Short
Long
Short
Lona
Lowl
0.1%%
6.7%
3.8%
Mecur Low
0.0%
0.2%
0.3%
(0.3K)
(19k)
(10x)
Extracilvenee Mosku
CNN/DM
0.196
0.496
0.3%
(23k)
(44k)
(154)
(0:3k)
(1.1k)
(0.9K)
High
25.6%
26.2%
4.0%
5.0%
0.25 (0.4k)
(1.24)
(1.4k)
Medium Specificity
High Specificity
Length
Long
LON
LOW
26.3%
0.35% (0.7k)
0.0%
(144K)
[1]
(0)
(5414
(3)
Extractivened Modium
XSUM
1,7%
0.0%
0.0%
Medium
(1.3k)
(11)
(0)
Hihh
High
0.0%
0.0%
0.0%
(0.5k)
[D]
(0)
(0)
Medium Specificity
High Specificity
Length
Long
50 2. (1.9M)
0.0%
0.0% (0)
LDW
0.0% (0.94)
0.0% (4)
0.0% (0)
(0)
Extractiveness
Gigaword
Modium am
0.0%
0.0%
Meduum
0.0%
0.0%
0.0%
(D.444
(0)
High
31.2%
(1.22 (1.2M)
0.0%
0.0%
(0)
High
0.0% (1.114
0.0%
0.0%
(1)
Medium Specificity
High Specificity
Length Mcdium
Length Mediium
Short
Long
Shor1
LaW
17.7%
0.7%
0.3%
Loe
1%
(24k)
(208)
Extractiveness
WikiHow
slodum
1.99
0.556
0.39%
Vedium
4.8%
5.1%
8.55
(3.3K)
(0.714
(0.5k)
(8.114
(8.5k)
(14k)
Hgh
3.4%
(0.8K)
(0.31)
(0.3K)
(1.7%)
(2.2k)
(5.7%)
Medium Specificity
High Specificity
Figure Summary style distribution of 4 commonly-use used summarization datasets.
22

---

Published as a conference paper at COLM 2024
Police searching for two missing teens believed to be together who disappeared after reportedly threatening to hurt themselves. Erika R-, 14, of Holiday and Caleb B-, 13, of Tampa are dating and it suspected that she was picked up in a car on her way to Tampa with him, said police. B B-, a white male, left his home near Dale Mabry Avenue and Lois Avenue on Saturday morning, and police are concerned due to his age as well as threats of him harming himself and his girlfriend, according to ABC Action News. Erika R- (left), 14, of Holiday and Caleb B- (right), 13, of Tampa are dating and it is suspected that she was picked up in a car on her way to Tampa with her boyfriend, said police. Authorities are searching for the teens who disappeared after reportedly threatening to hurt themselves. It is not known what he was wearing when he left his residence. R-, white female, was last seen at her home around 11.30pm on Friday wearing long-sleeve t-shirt, low-top gray Converse sneakers and possibly blue jeans. The teen girl, who also threatened to harm herself, took $200 from her mother's 's purse along with her school backpack before leaving her residence on Westchester Drive, according to WTSP. She has scars on both arms, on her upper thighs and under her armpits from self-mutilation, well as red mole on the left side of her forehead. The teen girl (R- pictured above), who also threatened to harm herself, took $200 from her mother's s purse along with her school backpack before leaving her residence on Westchester Drive. She has scars on both arms, on her upper thighs and under her armpits from self-mutilation, as well red mole on the left side of her forehead B- (above), white male, left his home near Dale Mabry Avenue and Lois Avenue on Saturday morning, and police are concerned due to his age as well as threats of him harming himself and his girlfriend. R- is 5'6' tall, has auburn hair, light brown eyes and is 120lb. B- is 5'4' tall, has brown hair and is 130lb. Pasco Sheriff's s Office spokeswoman Melanie Snow said R- had moved to Iampa from Holiday about three weeks ago, according to the Tampa Bay Times She said: "We don't 't think that she is in danger, but she is only 14 years old and away from home on her own accord. Her mother wants her home."
Document
Control Codes: Specificity - High / Length Long
Output: Police are searching for two missing teenagers, Frika R-, 14, of Holiday and Caleb B B-, 13, of Tampa, who disappeared after reportedly threatening to harm themselves. R- was last seen at her home around 11:30pm on Friday wearing long-sleeve t-shirt, low-top gray Converse sneakers, and possibly blue jeans. B- left his home near Dale Mabry Avenue and Lois Avenue on Saturday morning, and police are concerned due his age and threats of harming himself and his girlfriend. R- has scars on both arms, on her upper thighs, and under her armpits from self-mutilation, as well as red mole on the left side of her forehead.
Summary 1
Control Codes: Keywords- 200 Length- - Short
Output: Police are searching for two missing teenagers, Erika R-, 14, and Caleb B-, 13, who disappeared after reportedly threatening to harm themselves, with R- taking $200 from her mother's 's purse and B- leaving his home near Dale Mabry Avenue and Lois Avenue on Saturday morning.
Summary 2
Control Codes: Keywords- Melanie Snow / Length- Medium
Output: Police in Tampa are searching for two missing teenagers, Erika R-, 14, and Caleb B-, 13, who are believed to be together. R- was last seen at her home around 11:30pm on Friday wearing a long-sleeve t-shirt, low-top gray Converse sneakers, and possibly blue jeans. According to Pasco Sheriff's Office spokeswoman Melanie Snow, R- had moved to lampa from Holiday about three weeks ago and her mother wants her home.
Summary 3
Table 11: Controlled summaries generated by INFOSUMM-0.5B for the same CNN/DM article as in Table 7. Names of non-public figures have been redacted in the table; the non-anonymized document can be found in the original dataset.
23

---

Published as a conference paper at COLM 2024
Media playback is not supported on this device
Farrell, 25, is set to move past 500 international points this weekend against Fiji, and second in the England all-time list behind Wilkinson.
Asked whether Farrell could one day beat his record of 1,179 points, Wilkinson said: "I have no doubt".
"I would be very surprised if Owen Farrell didní go on to score way, way more," he told BBC Sport. "500 points for guy who is 25 years old, you don't ton't have to do the maths but if he plays until he 35, he will be in good place." Wilkinson has been involved in the England camp this year as kicking and skills consultant, and says he can identify with the way Saracens fly-half Farrell approaches the game. "Sometimes when we are discussing things, you hear something that you definitely correspond with," said Wilkinson, who on Thursday became one of 12 new inductees into the World Rugby Hall of Fame.
" "[Working with England] winning drop goal in the 2003 World Cup final, continued. "But there is no telling or teaching, it's kind of sharing, and therefore there has to be room in all of us to keep growing. I am also there to learn." Wilkinson has also compared the current midfield combination of Farrell, who has scored 497 points for England, and fly-half George Ford to his partnership with inside centres Will Greenwood or Mike Catt.
a really exciting opportunity." Wilkinson, who scored the
Document
"Both those guys were phenomenally important to me in my career, in the way they supported me, got the best out of me, and helped me to uncover more about myself," Wilkinson explained.
"With Owen and George, they are both very, very open individuals, both very humble but not because they have been taught what to say, but genuinely its who they are. There is not a script being followed here. They are following enthusiasm, passion and serious devotion."
"It's very similar when you mention names like Mike Catt and Will Greenwood, it all makes perfect sense. It's just about having good people in there." You can hear more from Jonny Wilkinson on the Matt Dawson Rugby Show on BBC Radio live from 19:30 GMT on Thursday, 17 November.
Owen Farrell, 25, is set to move past 500 international points this weekend against Fiji, and is second in the England all-time list behind Jonny Wilkinson. Wilkinson has been involved the England camp this year as kicking and skills consultant, and has identified with Farrell's 's approach to the game. Wilkinson has compared the current midfield combination of Farrell and fly-half George Ford to his partnership with inside centers Will Greenwood or Mike Catt, citing their enthusiasm, passion, and devotion.
Summary
Table 12: Unconstrained summary generated by INFOSUMM-0.5B for random-sampled XSUM article.
24

---

Published as a conference paper at COLM 2024
Media playback is not supported on this device Farrell, 25, is set to move past 500 international points this weekend against Fiji, and is second in the Fngland all-time list behind Wilkinson.
Asked whether Farrell could one day beat his record of 1,179 points, Wilkinson said: have no doubt".
"I would be very surprised if Owen Farrell didní go on to score way, way more," he told BBC Sport. "500 points for guy who is 25 years old, you don't t have to do the maths but if he plays until he is 35, he will be in good place." Wilkinson has been involved in the England camp this year as kicking and skills consultant, and says he can identify with the way Saracens fly-half Farrell approaches the game. "Sometimes when we are discussing things, you hear something that you definitely correspond with," said Wilkinson, who on I'hursdyy became one of 12 new inductees into the World Rugby Hall of Fame.
"[Working with England] winning drop goal in the 2003 World Cup final, continued.
a really exciting opportunity." " Wilkinson, who scored the
Document
"But there is no telling or teaching, it's kind of a sharing, and therefore there has to be room in all of us to keep growing. I am also there to learn." Wilkinson has also compared the current midfield combination of Farrell, who has scored 497 points for England, and fly-half George Ford to his partnership with inside centres Will Grdenwood or Mike Catt.
"Both those guys were phenomenally important to me in my career, in the way they supported me, got the best out of me, and helped me to uncover more about myself," Wilkinson explained.
"With Owen and George, they are both very, very open individuals, both very humble but not because they have been taught what to say, but genuinely its who they are. There is not a script being followed here. They are following enthusiasm, passion and serious devotion."
"It's very similar when you mention names like Mike Catt and Will Greenwood, it all makes perfect sense. It's just about having good people in there." You can hear more from Jonny Wilkinson on the Matt Dawson Rugby Show on BBC Radio live from 19:30 GMT on Thursday, 17 November.
Control Codes: Extractiveness- High / Length Short
Owen Farrell is set to move past 500 international points this weekend against Fiji, and is second in the England all-time list behind Jonny Wilkinson. Wilkinson has been involved in the England camp this year as kicking and skills consultant, and has identified with Farrell's approach the game.
Output:
Summary 1
Keywords- 2003 World Cup Length Medium Specificity- Medium Output: In 2003, Jonny Wilkinson scored the winning drop goal in the World Cup final for England. Wilkinson has been involved in the England camp this year as kicking and skills consultant, and has identified with the way Saracens fly-half Owen Farrell approaches the game. Wilkinson has compared the current midfield combination of Farrell and fly-half George Ford to his partnership with inside centers Will Greenwood or Mike Catt.
Control Codes:
Summary 2
Control Codes: Keywords- 2003 World Cup / Length - Medium / Specificity- High
Owen Farrell is set move past 500 international points this weekend against Fiji, and is second in the England all-time list behind Jonny Wilkinson. Wilkinson, who scored the winning drop goal in the 2003 World Cup final, has been involved in the England camp this year as a kicking and skills consultant and has identified with Farrell's 'ss approach to the game. Wilkinson has compared the current midfield combination of Farrell and fly-half George Ford to his partnership with inside centers Will Greenwood or Mike Catt.
Output:
Summary 3
Table 13: Controlled summaries generated by INROSUMM-0.5B for the same XSUM article as in Table 12.
25

---

Published as a conference paper at COLM 2024
Be sure to include coffee mugs or any other dirty cup that needs a good washing. Fill your kitchen sink with very hot water, enough water, that is more than half, but less than to the very top. Add squirt of dish soap in the water, and on sponge, and make sure your sponge is wet, with dish soap it. Put in all the cups into the hot water in sink and start cleaning. Once you are done with the cups, you can wash your plates, cutlery and cooking utensils. Coffee stains, and other stains, need some work to get clean, so scrub a little harder, but also let all the cups soak in the hot water longer, for that gets the stains looser, and easier to get clean. Rinse off the soap and water, and make sure you rinse well in very hot water. Some people like to buy a rack to hang your cups, so the water drips off the cups. But, if you want your dishes be done quickly, hang your cups on rack. When you're done washing all the dishes, wipe the dishes with a cloth towel. You might want to wash your towel after drying the washer. Put away your clean cups, and other dishes in a cupboard, or somewhere safe from getting dirty, and where you grab one with out breaking cup other dish. Purchase and place kitchen rug the sink. If you drop cup during the washing, it's less likely to be broken and one under where you will gather cups from time to time.
Document
Control Codes:
None (Unconstrained generation)
Output: To clean coffee mugs and other dirty cups, fill a kitchen sink with hot water and add squirt of dish soap. Scrub the cups with the soap and water, and then rinse them thoroughly with hot water.
Summary
Control Codes: Keynoords - rack / Extractiveness- - Low Length- Medium
Output: clean coffee mugs and other dirty cups, fill a sink with hot water and add a squirt dish soap. Scrub the cups with a sponge and let them soak in the water for a few minutes, then rinse them off with hot water and cloth towel. If you want your dishes to be done quickly, hang them on rack, but if you want them to be clean quickly, wipe them with cloth towel.
Summary 2
Control Codes:
Length Long / SSpecificity- High
Output: To clean coffee mugs and other dirty cups, fill a kitchen sink with hot water and add squirt of dish soap. Scrub the cups with sponge and let them soak in the hot water for few minutes. Once the cups clean, wash other dishes such as plates, cutlery, and cooking utensils. It's important remove coffee stains, but also them soak in the hot water longer to loosen them. Rinse off the soap and water, and then wipe the dishes with cloth towel; if you want your dishes to be done quickly, hang them on rack, but if you want them to be done quickly, wipe them with cloth towel. Finally, put away the clean cups and other dishes in cupboard or somewhere safe from getting dirty, and place kitchen rug near the sink to prevent spills.
scrub the cups harder to
Summary 3
Table 14: Unconstrained Controlled summaries generated by INFOSUMM-0.5B for random-sampled WikiHow post.
26

---

Published as a conference paper at COLM 2024
G Limitations and Future Works
In this work, we primarily use expert iteration and distillation to optimize for the information-maximizing objective for summarization. While our approach allows the generation of reusable, high-quality dataset for summarization, in principle, the proposed objective can be optimized using alternative training methods such as online reinforce- ment learning. Therefore, straightforward extension to INFOSUMM would be to compare the performance and robustness of summarizers optimized through different learning techniques.
As conventional metrics such as ROUGE fail to accurately evaluate model outputs, recent works propose to further fine-tune summarizers with human preference data, egg,, through reinforcement learning (Stiennon et al., 2022; Wu et al., 2021). Although we show through extensive experiments that INFOSUMM is capable of distilling powerful summarizer from an off-the-shelf teacher, our search objective may fall short of representing the subtle demonstrated by the superior fine-tuning
nature of human preferences. Nonetheless, to specific benchmarks, we envision that INFOSUMM can still function as useful base model for learning from human feedback. In this scenario, INFOSUMM-00558 can be harnessed as strong base model for summarization, equipped with better initial performance and transferability than the models naively fine-tuned on human-authored references.
performance of INFOSUMM-O.
In addition, the high level methodology of INFOSUMM can be generalized tasks beyond summarization. While different tasks would require different set of critic models, the method can be adopted to tasks where the correctness of input-output can be measured and evaluated, either through external verifier (egg, commonsense reasoning; Liu et al., 2023a) or through symbolic execution (e.g,, code generation; Haluptzok et al., 2023). The distillation stage of INFOSUMM can also be improved by incorporating advanced learning techniques such as BRIO (Liu et al., 2022), beyond maximizing conditional likelihood of the summaries in the generated dataset.
27


Reviews:
[{"review_id": "CDg9aysMyA", "reviewer": "Reviewer_LVKS", "summary": "The paper proposes a framework, InfoSumm, to distill a summarizer by leveraging the quantifiable information maximizing objective for long and document-level summarization (saliency, faithfulness and brevity), based on the dataset generated from an off-the-shelf LM. The proposed formulation outperforms the baseline models in both news summarization benchmarks and unseen domains, and generates controllable summaries across dimensions of length, extractiveness, specificity and keywords, all without relying on human-annotated references or large language models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive comments. Below, we address each of your questions:\n\n---\n### Choice of Base LM\nWhile we chose Pythia-2.8B as it is a small LM with sufficiently long position embedding for pair generation, our framework does not require using a specific base model. Here, we additionally report the sampling efficiency of our framework using GPT2-XL with 1.5B parameters as the teacher model. As the core contribution of InfoSumm lies in self-training the teacher model through the information-centric objective for summarization, we evaluate whether InfoSumm can self-train a summarization data generator even when using GPT2 as the base model, measured by sampling efficiency.\n\n|           | **Sampling Efficiency** |\n|-----------|:---:|\n| No expert iteration (GPT2-XL) | 0.5% |\n| **InfoSumm (GPT2-XL)** | 46.2% |\n\nAs expected, zero-shot prompting GPT2-XL almost never generates a correct document-summary pair, as quantified by the sampling efficiency of 0.5%. After expert iteration, however, the sampling efficiency dramatically increases to 46.2%. The result shows that even when using a weaker teacher model such as GPT2, InfoSumm can generate high-quality pairs that pass all the defined critics.\n\n---\n### Masked tokens for Saliency & Compression Ratio for Brevity\nWe agree with the reviewer that measuring mask likelihood given the summary may not suffice to fully capture the saliency, primarily when the mask selection is suboptimal. However, as shown in the denominator of Eq (1), the noise introduced by suboptimal masks are effectively mitigated by normalizing with the unconditional likelihood of the mask (i.e., without the summary). Despite its simplicity, we find that this normalization (along with a lightweight mask selection using TF-IDF) suffices to align with the human-perceived degree of saliency, as shown in Appendix D. For brevity, we would like to also note that while compression ratio is simple, it has been considered as an important attribute of summary style, and has been used as an scalable proxy of brevity in prior works (e.g., [1]).\n\n---\n### Figure 1, Typo\nThank you for the pointer, we will update these in the revised version of the paper.\n\n[1] Grusky et al., Newsroom: A dataset of 1.3 Million Summaries with Diverse Extractive Strategies., NAACL 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducted comprehensive and extensively experiments and analyses. The proposed framework demonstrates good performance to the in-domain reference-supervised models and shows better controllability than prompting ChatGPT. It also can be used for filtering high-quality data for summarization. One large-scale summarization dataset was generated as well in this work.\n\nThis paper is beneficial to readers, well-structured, and easy to read.", "reasons_to_reject": "One might be curious why choosing Pythia-2.8B but not other LMs.\n\nIn section of Saliency on page 3, it is not fully convinced for saliency perspective by measuring how well an MLM can recover x from x_mask when given the summary y. The masked tokens might not represent important information.\n\nMeasuring brevity by the compression ration between the summary and the document is not fully representative.\n\nFigure 1 was not mentioned in the main content.\n\nOn page 8 line 3, 'than 'should be replaced with 'then'.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p52Nv8Lzwu", "reviewer": "Reviewer_MKad", "summary": "This paper presents a method for distilling a summarizer based on the information-maximizing objective. The proposed method begins by self-training a teacher model from a T5-large, alongside self-supervised critics. Subsequently, the dataset is generated from this self-trained teacher model. Finally, the method fine-tunes a smaller student LM specialized for summarization. Experimental results demonstrate that the distilled summarizer exhibits superior performance compared to baseline models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for both positive and thoughtful feedbacks! Below, we seek to address each of the reviewer's concerns.\n### Benchmark exposure on the teacher model\nWe agree that analyzing the possibility of contamination on existing summarization benchmarks is important for the robustness of reported result. To mitigate this concern, we following the suggestion of Reviewer SZDS to measure the token overlap between our generated dataset vs. CNN/DM and XSUM. We sample a subset of 100K summaries from InfoSumm, and compare it to CNN/DM and XSUM. For each of the sampled summaries, we measure the Max ROUGE-1 across all references in CNN/DM and XSUM. For comparison, we also report the ROUGE-1 between InfoSumm and a subset of NewsCrawl consisting of 5K articles between May~Aug 2023 (which is well after the knowledge cutoff of the Pile in 2020).\n|**Datasets**|**ROUGE**|\n|-|:-:|\n|CNN/DM|15.08 (± 2.25)|\n|XSUM|14.11 (± 2.13)|\n|NewsCrawl|14.65 (± 2.70)|\n\nThe overlap of CNN/DM and XSUM against InfoSumm is nowhere larger than that of NewsCrawl, where there is no possibility of test set contamination; this suggests that superior performance of InfoSumm does not merely come from the memorization of the teacher model.\n### Another base LM\nFollowing the reviewer’s comment, we add additional result using a different base LM than Pythia-2.8B. To make our setting further challenging than using Flan-T5 (which has been fine-tuned for instruction-following), we instead use off-the-shelf GPT2-XL with 1.5B parameters as the teacher model. Due to the spatial constraints, we leave the detailed experiment setup and the results in author response to Reviewer LVKS. Overall, we find that even when using a weaker teacher model such as GPT2, InfoSumm can generate high-quality pairs that pass all the defined critics.\n### Source of input document\nAs illustrated in Section 2.2, we do not make use of any input documents in both generating $D_{init}$ and $D_{summ}$. We remove dependence to an external source of human-written data by generating both the document and the corresponding summary from the base LM, using PMI maximization decoding.\n### Diversity of generated data\nIn fact, we conduct multiple analyses on the generated data diversity in Section 3.5 and Appendix E.5. We find that InfoSumm, as a purely generated corpus, is more diverse than human-curated datasets for summarization, both in terms of (1) lexical overlap between samples, and (2) summary style diversity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and straightforward. The concept of distillation based on the information-maximizing objective for summarization is well-conceived. Furthermore, it provides an extensive and well-executed experimental validation of the proposed method, along with thorough analyses.", "reasons_to_reject": "In this paper, Pythia-2.8B serves as the teacher model. The Pile dataset used for training Pythia-2.8B includes subsets like news and Wikipedia pages used in CNN/DM, XSum, and WikiHow. The lack of analysis related to these subsets raises concerns about the robustness of zero-shot and unseen domain experiments.\n\nTo underline the effectiveness of the proposed method, it is essential to expand the experimentation to include models such as Flan-T5, in addition to Pythia-2.8B. While Pythia-2.8B has shown promising results, the need to demonstrate the method's robustness across different models remains critical. This broader testing will provide a clearer indication of the method’s generalizability and effectiveness beyond its current scope.\n\nThe paper states that an initial dataset of 1.4 million was created for the experiment, but fails to clarify which seed document was used as input. This detail is crucial for understanding the dataset's composition and the experimental setup.\n\nMoreover, the issue of generating candidate pairs and considering data diversity has not been addressed. The potential for duplication in the multiple datasets of document-summary pairs ($D_{summ}$) generated by the teacher model highlights the need for further analysis to ensure variety and prevent bias in the generated summaries.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7XAmOSRCeN", "reviewer": "Reviewer_NLbV", "summary": "This paper focuses on information-theoretic distillation methods, investigating their application and effectiveness in machine learning models. The authors analyze the theoretical underpinnings of these distillation techniques, discuss their practical applications, and demonstrate their empirical performance through experiments. The goal is to contribute to the understanding of knowledge distillation in the context of reducing model size while maintaining accuracy.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and encouraging comments. We are happy that you found our method innovative and our analyses to be thorough both theoretically and empirically. Below, in addition to the 5 benchmarks evaluated in the main section of the paper, we additionally evaluate whether InfoSumm can generalize to dialogue summarization, a domain substantially different from news articles that InfoSumm was trained for.\n\n---\n### Application to another dataset\nWe additionally report results on DialogSum [1] that aims to summarize real-life dialogue scenarios. Following the same setup with Section 3.3, we use ROUGE-L and G-Eval to compare InfoSumm against $\\text{PEGASUS}_\\text{CNN/DM}$ and $\\text{gpt-3.5-turbo}$. The results are shown below:\n\n| **Datasets** | **R-L** | **G-E** |\n|--------------|:-------:|:-------:|\n| $\\text{PEGASUS}_\\text{CNN/DM}$ |   15.6  |   3.43  |\n| $\\text{gpt-3.5-turbo}$ |  23.8  |  4.56  |\n| $\\text{InfoSumm-0.5B}$ |  23.6 |  4.53  |\n\nConsistent to the results in unseen domain evaluation, we find that InfoSumm significantly outperforms human-supervised PEGASUS and is comparable to gpt-3.5-turbo, as measured by both reference-based / reference-less metrics.\n\n[1] Chen et al., DialogSum: A Real-Life Scenario Dialogue Summarization Dataset. Findings of ACL 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents innovative approaches to information-theoretic distillation, advancing existing literature and proposing new strategies that could lead to more efficient models.\n2. The authors provide a thorough theoretical analysis, supplementing it with empirical results that demonstrate the potential impact of the methods proposed.", "reasons_to_reject": "The empirical evaluation could be expanded to include more diverse datasets or use cases to ensure the results' generalizability across different scenarios.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuJUubWjyY", "reviewer": "Reviewer_SZDS", "summary": "# A summary in bullet points\n- The paper presents a method to distill a small state-of-the-art (unsupervised) summarizer (Pegasus 0.5B) from a small-ish LLM (Pythia 2.8B).\n- Distillation involves generating a large-scale synthetic **English news summarization dataset** of 4.5 million samples.\n- Synthetic data is generated by prompting a base LLM to continue from prefixes like \"New York, (CNN)\" to promote news-style generation.\n- The method requires an off-the-shelf base LLM + a masked language model (T5-Large) but no human references.\n- To improve the quality of the synthetic data the LLM is iteratively improved by combining 3 proposed \"critics\" and rejection sampling.\n- The 3 critics model *saliency, faithfulness, and brevity* and are implemented via mutual information (probabilities are estimated by the pre-trained T5-Large model)\n- The improved LLM is then used to generate a large-scale synthetic summarization dataset of 4.5 million article-summary pairs.\n- 1 million of these pairs are posthoc annotated for controllable summarization.\n- The paper evaluates the performance of training small summarizers (0.5B) on the synthetic dataset.\n- Consistent improvements are shown across automatic benchmarks and human evaluation (in/out-domain and zero-shot.)\n- Presented experiments show that the proposed summarizers surpass prior unsupervised state-of-the-art systems and are comparable \nto summaries generated by ChatGPT (x300 larger).", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for both positive and thoughful comments! Below, we seek to address each of your concerns:\n\n### Model & Dataset\nWe agree with the reviewer that sharing both the models and dataset is an integral part of reproducible research. We will release the resources once they are free of anonymity constraints.\n\n### Potential data contamination\nWe follow the reviewer's suggestion to directly measure the token overlap between our generated dataset vs. CNN/DM and XSUM. Due to spatial constraints, please refer to the author response to Reviewer MKad for specific experimental setting and the results. Overall, the results show that the overlap of CNN/DM and XSUM against InfoSumm is nowhere larger than that of NewsCrawl, where there is no possibility of test set contamination.\n\n### Rejection sampling details\nIn fact, we included the implementation details in Appendix A, including the specifics of filtering stage for rejection sampling. During rejection sampling, we leave the samples that satisfy each threshold in all 3 critics, as specified in Eq (7). We will strengthen the reference from the main section to the Appendix in the revised version of the paper.\n\n### Effect of multiple expert iteration steps\nWe validate this effect in Appendix E.3, where we test with 2 rounds of expert iterations. While 2 expert iteration steps marginally improve the sampling efficiency, it significantly reduces the diversity of generations (e.g., MSTTR reduces from 57.7 to 52.9). This suggests that multiple rounds of expert iteration can over-optimize the teacher, compromising the quality of the dataset for better quantity (as measured by sampling efficiency).\n\n### Computational Requirements\nWe thank the reviewer for notifying this - as our framework only employs a relatively small model with <3B parameters, one would essentially require only 1 research-scale GPU with 11G VRAM. Compared to prompting LLMs, our framework is significantly cheaper in inference, while providing comparable performance to ChatGPT. In controllable summarization, one would roughly have to spend ~$3 per 1K inference for 5-shot prompting GPT-3.5-turbo (assuming each example consisting of 1024 token documents), only to see less controllability than InfoSumm, which only requires 568M model running on a local GPU. \n\nLastly, we would like to refer the reviewer to Appendix G, where we discuss the limitation and future implications of InfoSumm. Again, we appreciate your thoughtful comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes several impactful findings that are immediately valuable to the research community.\n- An effective method to create large-scale quality synthetic summarization datasets without relying on existing labels or instructions.\n- A novel summarization data generation model allowing further exploration.\n- The first large-scale synthetic summarization dataset with promising results.\n- Findings suggesting that synthetic summarization data generalizes better than existing summarization datasets with human-written summaries.\n- State-of-the-art English summarizers for both generic and controllable summarization.", "reasons_to_reject": "The paper lacks a few central details related to the contributions, methods, and experimentation.\n\n- It is unclear if the LLM generator, dataset, and summarizers will be shared or whether the method is the contribution. The former would greatly strengthen the weight of this paper.\n- There is a possibility of data contamination wrt. the main results on the CNN/DM and XSum datasets. The off-the-shelf LLM utilized to generate synthetic data has likely been trained on CNN, DailyMail, and BBC articles. If so this would mean that the generator LLM has been trained on the test sets and in turn, potentially generated synthetic dataset with the test set in it. This suspicion could be resolved by reporting the overlap between the synthetic data and the test sets but is not reported in the paper. My suspicion comes from the very high ROUGE scores on par with early SOTA **supervised** transformers models (BERTSumABS, Liu and Lapata, 2019).\n- The process of generating and training on the synthetic dataset is unclear making it hard to reproduce.\n  - Rejection sampling is central to the final size of the synthetic dataset (and the performance of the summarizers?), however, details are included nor cited. This may be simple but more clarify in this regard would significantly strengthen the paper.\n  - The paper mentions that further \"expert iterations\" are bad, however, this claim is not quantified. How is the ideal number of iterations estimated?\n  - The computational costs and requirements are unknown. At what point is this method more cost-efficient than simply running ChatGPT?\n  - The target size of the synthetic dataset seems to be 10M but it's unclear why. The limitations and potential implications of the method are unclear.", "questions_to_authors": "Great work! Most questions are implicit in the bullet points in \"reasons to reject\".", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes infosumm, an information-theoretic distillation framework for summarization. The core idea is to train a strong teacher model using summarization task motivated information theoretic objectives. This teacher model is iteratively self-improved using expert iteration. Then, a smaller summarization model is distilled from this strong teacher. Crucially, this entire pipeline does not rely on any reference summaries. 

Pro:
Generally, the reviewers agree that this is a strong paper. The proposed approach is interesting, performs well on the datasets evaluated, both in in-domain and out-of-domain scenarios. The experiments are exhaustive. The reviewers' issues about data contamination have been addressed in the rebuttal. 

Cons: 
The data processing pipeline to filter and create data should be shared as it seems non-trivial to reproduce.
Decision: Accept