Do Summarization Models Synthesize?
Authors: Jay DeYoung
Iain James Marshall
Byron C Wallace
Venue: Submitted to ICLR 2023 (2023)
Keywords: Summarization
Factuality
Sentiment
Systematic Reviews
Evidence Synthesis

Abstract: Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately \emph{synthesize} inputs with respect to a key property or aspect. For example, a synopsis of film reviews all written about a particular movie should reflect the average critic consensus. As a more consequential example, consider narrative summaries that accompany biomedical \emph{systematic reviews} of clinical trial results. These narratives should fairly summarize the potentially conflicting results from individual trials.

In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this type of synthesis? To assess this we perform a suite of experiments that probe the degree to which conditional generation models trained for summarization using standard methods yield outputs that appropriately synthesize inputs. We find that existing models do partially perform synthesis, but do so imperfectly. In particular, they are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., the ratio of positive to negative movie reviews). We propose a simple, general method for improving model  synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or \emph{abstaining} when the model produces no good candidate. This approach improves model synthesis performance. Our hope is that by highlighting the need for synthesis (in some summarization settings), this work motivates further research into multi-document summarization methods and learning objectives that explicitly account for the need to synthesize. 

Under review as a conference paper at ICLR 2023
Do SUMMARIZATION MODELS S SYNTHESIZE?
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to key property or aspect. For example, synopsis of film reviews all written about a particular movie should reflect the average critic consensus. As more consequential example, consider narrative summaries that accompany biomedical systematic reviews clinical trial results. These narratives should fairly summarize the potentially conflicting results from individual trials.
In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this type of synthesis? To assess this we perform a suite of experiments that probe the degree to which conditional generation models trained for summarization using standard methods yield outputs that appropriately synthesize inputs. We find that existing models do partially perform synthesis, but do imperfectly. In particular, they are over-sensitive to changes in input order- ing and senestveesitive to changes in input compositions (c.g., the ratio of positive negative movie reviews). We propose a simple, general method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate. This approach improves model synthesis performance. Our hope is that by highlighting the need for synthesis (in some summarization settings), this work motivates further research into multi-document summarization methods and learning objectives that explicitly account for the need to synthesize.
INTRODUCTION
Multi-document summarization (MDS) models aim distill inputs into concise synopses that pre- serve key content. Examples of MDS include summarizing news articles (Dang) 2005) Fabbri et al.,
2019 Ghalandari al.) 2020 Evans al., 2004), answering questions from muluple sources (Dang
2006 and producing overviews of scientific literature (Liu et al. 2018 Lu et al. 2020 Mollá & Santiago-Martnnz/] 2012, Wallace et al.) [0200] De Young et al., (021)) We expect summarization models to produce outputs consistent with inputs (Kryscinski et al. 2020 Nan et al. 202 e.g.,
discussing the same types of entities (Nan et al. 2021b) and allowing one to answer questions similar in way that is consistent with individual inputs (Wang et al. 2020a Scialom et al. 2021).
In some applications models must synthesize inputs -icuts-i.c... aggregate potentially conflicting information-to yield accurate synopsis (Figure D. As simple example, consider the meta- reviews of movies featured on Rotten Tomatoes which provide a consensus view of individual critic opinions. These reviews should therefore reflect the mean and range of sentiment implicit in the input critiques: summary of mostly negative reviews (e.g., Gigli) should communicate that the film was widely panned; summary of mixed reviews (as in the case of The Fifth Element) ought to convey that critics disagreed and discuss the main positive and negative attributes.
A more consequential example is the task of summarizing the evidence presented in clinical trials. Individual trials will frequently present conflicting evidence about whether or not a particular health intervention is effective. An ideal summary of the evidence would appropriately weigh the findings presented in the constituent inputs and reflect the evidence on balance.
A website that aggregates film reviews: https:/www.rottenttomm/

---

Under review as a conference paper at ICLR 2023
Syntheeizing movie reviews
Syntheeizing reports clinical
Mecatwelyy ch chaleegss veraly ceeptrecing
There nor ro rephiitt pilowenc tr Nuthaegnerrsuuhe and plessco graze
The Fis Planect is eteggsing accistion of the wong clenentt
hat dhoaet winntal sicce pap oaker toll awricayy
The enddence not bwwhonyhanngiing bealing COV
The office alae clrephoyyceeessssn durraasaaly vilctinn dltough not ablloeeutty oppetoad
meedy San, edtovogant solence Aotic
Figure I: Two multi-document summarization tasks where models must implicitly synthesize inputs produce accurate summaries. Left: Summarizing film reviews with varying sentiment yield a critics consensus. Right: Summarizing trials that have evaluated particular medical invention.
What are the desiderata of multi multi-document synthesis? First, summaries produced by models should be consistent with the input data, with respect to the latent property of interest. In the case of Rotten Tomatoes, the sentiment of the summary should be in line with the aggregate sentiment expressed in the individual critic reviews. corollary to this is that models should be sensitive to changes in the composition of inputs, e.g., removing most of the negative reviews from set of inputs should yield a summary with a corresponding increase in the expressed sentiment.
In this work we evaluate neural MDS models with respect to these criteria. To this end we use a meta-reviews dataset from Rotten Tomatoes (Leone 2020) and a dataset systematic reviews (meta-analysed summarizing the evidence about medical interventions (Wallace et al. 2020). For the former we probe the degree to which generated meta-review sentiment agrees with the expected aggregate sentiment score; for the latter we evaluate whether the generated summary indicates that the input evidence suggests, on balance, that the intervention under consideration was effective.
Our main contributions summarized follows. (1) To the best of our knowledge, this is the first work to investigate implicit synthesis in summarization, and the degree to which modern models are capable of this (2) We show that "off-the-shelf" neural MDS models are somewhat inconsis- tent and insensitive with respect to performing synthesis in summarization.
(3) We propose and evaluate simple and general technique which involves generating a diverse set of output candidates
(Vijaykkumrr et al., (2166) and then selecting from these on the basis of agreement with an expected aggregate measure (based on inputs), with promising results.
2 SYNTHESIS AND SUMMARIZATION
In standard multi-document summarization, we assume inputs (X,, Vi). where X; (r......F.jxx /. We then typically train a summarization model with parameters 6, consume X, and yield summaries ú; as similar as possible to targets yı. More precisely, the standard ob- jective entails finding estimates for which maximize target token probabiiiiiiies. Assuming the input documents Tij in X, have been linearized (i.e., concatenated, usually with adjoining special tokens to demarcate individual inputs) into string z!@ of input tokens, this objective takes the form:
t= log Po(y Vy1.. .... Vi(e- 1), x where pe is probability assigned to the token at position t in the (linearized) target _ by a summarization model with parameters 6. By myopically focusing on encouraging the model to produce tokens that mimic the targets, this objective aligns with stan- dard (but flawed) measures of automated summary quality like ROUGE (Lin) 2004), which quantify n-gram overlap between targets yi and outputs fi-
E Mkl
We are interested in settings in which there is an additional, latent property implicit in the constituent input texts Tij. Zgj- For example, Ziy might reflect the sentiment in critique of the film indexed by i. Summaries should synthesize this aspect, i.e., the generated summary d. should implicitly convey an aggregated z; which reflects a synthesis or aggregation G over Z; = (x.x,i1 That is, we assume z; = G(Z,) In both cases considered here-summmrres of film critiques and synopses of clinical trials evidence-OC can reasonably be assumed to be (weighted) mean, G(Z.) म EjX1 Aajziy. That is, summaries should roughly reflect the average sentiment and reported treatment effect in the cases of movie reviews and clinical trial reports, respectively.
"See Appendix B for related content aggregation work, over structured relations Shah al. (021a))
2

---

Under review as a conference paper at ICLR 2023
Train
Dev
Test
Train
Devi
Test
Number of metareviews Avg. metareview length
7251
932
912
1675
360
397
32.0
32.6
32.4
101
107
111
Total number of inputs
195033
24336
24474
11054
1238
2669
Avg. number of inputs
26.9
26.1
26.8
6.6
3,4
6.7
Avg length of individual input
30.6
30.8
30.6
475
379
449
Avg length of concatenated inputs
822
804
822
2641
1336
2544
Target Percent Positive
59.5
62.1
61.2
31.9
31.4
35.0
Table I: Dataset statistics for movie reviews (left) and systematic reviews (right). Number of meta- reviews, average meta-review length (tokens), number of input reviews per split, average number of inputs per instance, average total length of an input to instance. For movie reviews, the target percent positive reports the fraction of metareviews with positive sentiment; for systematic reviews this refers the fraction of metareviews reporting a significant effect. We subset the original dev set to instances of < 4k tokens (to accommodate T5; the other models can consume up to 16k).
We investigate the following questions. (1) Do model summaries ú: reflect the anticipated aggregate aspect of interest? That is, how well calibrated is the aspect communicated in the generated summary (2ig) compared to the expected 2;? (2) Can we improve the ability of summarization models to synthesize by explicitly incorporating synthesis targets 2; into the decoding process?
We propose simple inference-time procedure to explicitly preference output candidates that align with the expected aggregate property of interest (e.g.. average sentiment), and report promising results for the approach. This strategy also naturally lends itself to cautious summarization, i.e., approaches in which we allow the model to abstain from generating an output if it does not produce any candidates that reflect the anticipated aggregate measure.
DATASETS AND MEASUREMENTS
3.1 MOVIE REVIEWS
We first consider a dataset comprising movie reviews and associated meta-reviews summarizing these from Rotten Tomatoes. An in-house staffer summarizes audience reviews into meta-reviews. These meta-reviews synthesize the constituent input reviews, and reflect the aggregate critic recep- tion of a film. Each meta-review is associated with a numerical "Tomatometer" score, which is an overall measure of what percent reviews were positive for the corresponding film (G then is an average of the positive indicator per review). The Rotten Tomatoes dataset we use comprises 9095 movies with meta-reviews constructed from 244,000 individual reviews (Table
Measuring sentiment in movie reviews. As our measure we train BERT model (Devlin al.) 2019) using the continuous (fine-grained) sentiment targets provided the SST dataset (Socher et al. 2013 F We trained this model for 3 epochs using a learning rate of 5e-5 using the fuggingface library with no hyperparameter tuning. While the raw text of the SST dataset is in-domain, the targets themselves are not. We find reasonably strong correlation between our sen- timent estimates and the "true" meta-review sentiment ("Tomatometerr" score): The R² (centered) is 0.696, mean squared error (MSE) of 0.022, and Pearson's 's r of 0.836 (Figure 2 upper left).
3.2 BIOMEDICAL SYSTEMATIC REVIEWS OF TREATMENTS
Our second dataset is collection of systematic reviews from the Cochrane Collaboration 5 This dataset comprises roughly 2600 systematic reviews summarizing total of 16,500 clinical trials evaluating interventions in healthcare (Table I Each review includes both natural language sum-
wriiten by designated "top-critics", audience members recognized for quality and quantity of reviews *SST is itself based on collection of Rotten Tomatoes critic reviews (Pang Lee 2005). We verified that the SST text fragments do not overlap with any of our target reviews by manually checking any (fragment, review) pair with substantial (>= 75%) overlap for approximately one quarter of all reviews.
/https://ii thubbb....tbbb..o/o com haam/huge huggi ng wwweefttcaaaaare
Formers s//b////lll /'b1 ob/ /ma n/ examples/
pyytorch/txxx cl classi fi cat 1 on/ run_rnng gll_gleueee www..luepppp. py
"An international non-profit dedicated helping healthcare providers make wvidence-hased decisions.
3

---

Under review as a conference paper at ICLR 2023
1.0
Human
PRIMERA
0.4
0.0
Pagasus
0.8
Predided Metareview Sentiment
Figure 2: Movie Reviews; Actual vs. Predicted Sentiments on generated summaries. We replaced LED with human outputs (upper left) for comparison; see Figure in Appendix for all models.
mary and accompanying statistical meta-analysie results. The latter provides aggregate statistical summary of the individual (study-level) data extracted from the trials included in each review. The natural language summary should accurately convey and contextualize the findings of the meta- analysis. Therefore, the (lack of) treatment efficacy communicated in a given summary should generally agree with the direction of the corresponding meta-analytic point estimate.
Mcasuring effects evidence syntheses For systematic reviews of clinical trials, we resort to a less granular classification model g(z4g) g(y.) which attempts to infer whether given piece of text reports a significant result or not. In particular we use RobotReviews 2017 De Young al., 2020). Given narrative describing clinical trial result (or a systematic review summary of such results), RobotRev Rebbeeeeieewrr predicts whether the reported result indicates a significant effect of the treatment being investigated, or not. can compare this prediction to the "truth", which here is derived from the meta-analytic result (specifically by checking whether p 0.05). Applying this off-the-shelf model to the manually composed summaries accompanying the meta- aatly-ssayyss in Cochrane set, observe macro-average Fl score of 0.577 (Table I0. Appendix D). providing a reasonable (if weak) measure for this task.
(Marshall et al.,
4 MODELS
We evaluate suite of transformer (Vaswani et al. 2017) summarization models: Longformer (Belt- agy et al. 2020), Pegasus (Zhang al. (020), PRIMERA (Xiao et al. (202T), and T5 (Raffel al. 2020). PRIMERA was designed and pre-trained specifically for multi-document summarization specifically. And while not explicitly designed as multi-document summarization models, both Pe- gasus Zhang et al. (202)) and T have been used on multi- molti-doccmmnt tasks, while Longformer has been used for related multi-document summarization task (DeYoung al. (02T). For all models we mostly use hyperparameters defaulted to in their respective hugging face implementations. We conduct hyperparameter sweep over optimization steps and learning rate, selecting the best model by ROUGEI performance on the dev set (Appendix C Tables
9
5 EXPERIMENTS
5.1 How WELL DO SUMMARIZATION MODELS SYNTHESIZE?
We report sentiment performance for all models Table 2 These are metrics quantifying the strength of the relationship between (a) the continuous sentiment inferred (via our text-regression
https::/hhugginggface..
summa zat 1 news
4

---

Under review as a conference paper at ICLR 2023
R-
Pearson's s
MSE
ROUGET
Fc-oe
ROUGET
LED PRIMERA
0.551
0.742
0.042
0.242
LED PRIMERA
0.490
0.259
0.608
0.780
0.037
0.254
0.526
0.253
T5
0.516
0.720
0.046
0.253
T5
0.521
0.206
Pegasus
0.530
0.730
0.044
0.245
Pegasus
0.568
0.212
Reference 0.697
0.836
0.023
Reference 0.577
Table 2: Base synthesis results. Movie reviews (left): correlations between sentiment measured in model outputs and target sentiments. We report R², Pearson's and mean-squared errors. System- atic reviews (right): we report macro-averaged Fls. ROUGEI included for reference.
LED
PRIMERA
1.00
LED
Pegasus
0.3
0.75
0.25
0.0
0.4
Pegasus
Denddty
0.75
0.50
0.1
0.00
Sentiment Difference from Mean
Average Predicted Effect Entropy /Significant Effects
Figure 3: The spread of sentiment/rreatment effect measured in outputs produced from permuted input orderings. Left: Movie review sentiment. Right: Systematic review significance prediction entropy indicates order insensitivity) the subset of reviews that report significant effects.
g) over model generated or reference (human written) summaries and (b) the reference sentiment (Tomatometer) score. Across these metrics, correlations between the sentiment measured in model generated outputs and the Tomatometer score considerably lower than that between the same measurement over human-composed summaries and said score. Based on these metrics, human authors do better job of synthesis than the models when composing their summaries.
For systematic reviews (Section (.2). we are able to measure whether a text appears to report sig- nificant treatment effect or not, and can compare this against the p-value from the corresponding statistical meta-analysis. This permits only a coarse assessment of synthesis, as we are unable to measure correlations. Instead we report classification metrics describing how often the effect signif- icance inferred from summary (generated or manually written) matches the ground truth derived from the meta-analysis (Table The results are qualitatively similar the sentiment case, in that the humans appear to do better job of synthesis - as best we can measure, the significance reported in their summaries better aligns with the statistical results than in model generated summaries.
5.2 SENSITIVITY TO INPUT ORDERING
Synthesis of inputs should be invariant ordering (e.g., the critics' consensus on film does not depend on the order in which one reads the reviews). Here we evaluate if models are sensitive to input orderings with respect to the synthesized aspect of interest (Zig) in the resultant outputs. Specifically, X; = [I.1, xijx, will constitute an arbitrary ordering of inputs reflected in the linearized version x.a. This ordering should not affect the aggregate aspect Zig in the summary.
To evaluate models realize this invariance, we permute the instance inputs X, (and, consequently, the linearized x:()) one hundred times, randomizing input orderings. For each such permutation X; (and associated x ), we generate a summary Ø: and estimate the resultant aspect žgg. using the corresponding measurement model. By repeating this process for each instance i, an empirical distribution over ăgg" under different random orderings.
construct
Movie reviews. We zero-mean the Lng's 's inferred from all instances into a histogram (Figure 3 left). This shows the spread of sentiments inferred outputs under random input orderings minus the corresponding instance mean sentiment. Were
each instance, and combine the distributions
5

---

Under review as a conference paper at ICLR 2023
R-
Pearson's
MSE
Fl-score
Accuracy
LED
0.524
0.724
0.057
LED
0,510
0.684
PRIMERA
0.572
0.756
0.052
PRIMERA
0.533
0.680
T5
0.481
0.694
0.063
T5
0.469
0.675
Pegasus
0.499
0.706
0.060
Pegasus
0.452
0.658
Table 3: Movie reviews (left): Correlation between subsampled inputs and generated meta-reviews.
Systematic reviews (right): macro-averaged results (FI and accuracy) for subsampled inputs.
Predicted Sentiment
1.00
LED
PRIMERA
T5
Pegasus
0.75
0.50
0.25
0.00
0.
1.0
0.0
1.0
0.0
0.5
10
Aggregate Inputs Sentiment
Figure 4: Model sentiment sensitivity to manipulated input sentiment. The intensity patterns indicate that models tend to oscillate between low and high sentiments in outputs, and not responsive to subtler shifts in imput sentiment compositions. For context we include a model regression (blue) and the reference sensitivity regression (black).
model completely invariant to ordering, the empirical distribution over these differences would collapse to Instead, we observe a relatively wide spread in the sentiment measured over outputs generated from different permutations, indicating counter-intuitive sensitivity to orderings [
Systematic reviews. For each X; we have 100 order permutations and associated summaries; we infer whether these report significant results not, and record the fraction that do (p;). If models were invariant to ordering, this fraction would always be 0 or 1. Values in-between suggest the result of different input orderings. We calculate the entropy of to quantify this. Figure 3 (right) shows a histogram of these entropies calculated over the subset of examples where the associated meta-analysis indicates significant effect ! Densities away from zero indicate sensitivity to ordering.
model flips the report conclusion
5.3 SENSITIVITY TO INPUT COMPOSITION
Synthesis models should be responsive to changes in the distribution of the attribute to be synthe- sized in the input composition: If we increase the ratio of positive negative reviews in an input set, we would anticipate a concomitant change in the sentiment communicated in the meta-review Zig. To assess if models meet this synthesis desiderata, we manipulate model inputs X; in such way to induce an expected change in the target measure Zig: we then measure if the output yields summary that aligns with this expected change.
Movie reviews. We manipulate the ratio of positive to negative reviews and observe the resultant change in the property of interest latent the corresponding output. We take movies with mixed reviews, and delete 10%, 20%, 30%, .... 100% of the positive inputs, retaining the negative inputs; we then repeat the process but instead remove negative inputs. For each of these permutations, we measure the input sentiment, the meta-review sentiment, and how well they correlate (Table
Figure 4 plots the relationship between the fraction of positive reviews the (manipulated) input sets and the granular sentiment score inferred over the resultant outputs. The models are gener- ally undersensitive to changes in their input: rather than having a change in meta-review sentiment equivalent in size to changes in input sentiment slope of 1, the human written summaries). Models tend to have trouble changing their sentiment, and require a large change in input distribution to substantially change the sentiment communicated in the output.
observe when fit model to
*For a ROUGE1 comparison, see Appendix F Figure 10
"hheee the more interesting cases; we provide results the entire dataset in Appendix Figure

---

Under review as a conference paper at ICLR 2023
+ Linearized input 2j
Candidate outputs from Diverse Beam
Summarization model
Search and Inferred sentiments over these
(
excalient [SEP] Meandering but dirested.. [SEP] film over long
A moss; diticutt to watch and too long.
[SFP)
Despte some iscues with the film
direction.
-/+
biooted movie locking direction.
2, = ((x))
arg max A 2ng) gEC,
Predict expected aggregate sentiment (e.g., 'mixed"
Select candidate that most agrees
-/+)
with predicted aggregate sentiment
Figure 5: Proposed strategy to improve synthesis. We generate intentionally diverse set of output candidates (Vijayakumar et al. 2016) and then select from these the text that best agrees with the predicted aggregate property interest (here, sentiment). We can also abstain when the model fails to yield an appropriate output.
Systematic Reviews. To measure sensitivity changes in input composition, we manipulate our inputs X; such that the meta-analysis result (target zig) flips from a significant effect to effect, or from no effect to an effect. Operationally, we do this by first taking of a subset of the reviews that have conflicting evidence (yielding 139 unique reviews). We then order inputs in these by (weighted) effect sizes 10 and remove subsets which ought to flip the significance result.
IMPROVING SYNTHESIS IN SUMMARIZATION
We propose simple post-hoc approach to improving the synthesis performed by multi-document summarization models. This involves the following steps: (I) Generate explicitly diverse set of output candidates"] [1] (2) Select from these as the final output the candidate that best agrees with the expected synthesis result (as predicted by an external model)
For (1), we rely on previously proposed technique for generating diverse outputs C, from input 2 namely Diverse Beam Search (DBS) /Vijayakumar et al. 2016). This method modifies stan- dard beam search to maintain multiple groups of beams. During decoding, term is added to the next-token log probabilities which effectively penalizes production of (partial) strings similar to candidates on beams in other groups
In (2) would like select the output that best synthesizes the property of interest; this requires mechanism for specifying what we expect the synthesized property be, given the inputs. For exam- ple, if know the sentiment scores associated with input movie reviews, might enforce that the sentiment expressed in the output agrees with the average of these. To realize this intuition, we can select as final output from C, the string that best aligns with this anticipated aggregate property (sen- timent score significance finding). Operationally, this requires an external model to measure- or estimate- the aspect of interest as latent approach, but in many settings it may be feasible to identify construct model; we were able to do so for both tasks considered this paper.
given candidate output. This is a limitation of the
There is no guarantee that any member of C, will align well with the anticipated aggregated property. In such cases, we have no means of yielding an output consistent with respect to synthesis, and may be desirable to abstain from outputting anything all in such cases; that is, to be cautious summarizer (Fern et al.) 2004) Hechtlinger et al. 2018). We consider this strategy in the case of generating narrative synopses of evidence, as this constitutes case in which (a) one would very
IIn fixed effects meta-analysis the weights inverse variances associated with study-level effect estimates. See Appendix Tables 11 12 for an ablation over diversity vs. standard heam search outputs
12For related annerate-and-selees approach Oved & Levy 2021) Appendix B
This penalty is associated with a hyperparameter À that encodes the relative importance of realizing diverse:
>=0.5 here and did not extensively tune this. Other hyperparameters include number of groups and total number of beams; used 5 for hoth of these, retaining beams used for analysis above.

---

Under review as a conference paper at ICLR 2023
LED
LED
PRIMERA
PRIMERA
Pegasus
Pegasus
-0.1
&Peerson's Vanilia decoding
APearson's
AR?
Generate Gererale-diveese then select
decading
Generater diverse then
Figure Differences relative to human summaries under vanilla decoding and the proposed generate-diverse then select strategy on the Rotten Tomatoes dataset and task. We report Pearson's r and R2, both measures synthesis "calibration". Vanilla decoding yields synthesis performance worse than humans, but explicitly considering synthesis at inference time proposed results in per- formance comparable to and sometimes better than the human summaries (as best we can measure).
LED
PRIMERA
LED
Pegasus
0.75
0.1
0.25
Density
Density
1.00
Pegasus
PRIMERA
T5
0.2
0.75
0.50
0.25
0.0
0.00
1.
Diverse Sampling Sentiment Ranges
Average Predicted Effect
Figure 7: Distributions of outputs for the candiate summaries. Movie reviews (left) show his- togram for the range of differences between lowest and highest output sentiments. Systematic reviews (right) show histograms of the fractions of outputs reporting significant results.
much prefer not to produce misleading summary of clinical evidence (Kell et al. 2021), ). and, (b) we observe many cases where the diverse decoding strategy yields an output that seems to communicate (at granular level) the aggregate findings expected.
Movie Reviews For movie reviews we use BERT (Devlin et al. 2019) model trained on IMDB (Maas et al. 2011
predict the sentiment of each input Zij. using the proportion of T,j X, with a positive score as an approximation for the target sentiment Z:g. For each diverse prediction C, we predict sentiment Zig using sentiment regression model (Section 3.1, and select the prediction cloest to the estimated target sentiment |ăag - Zzgl- We find this improves model performance to human-like levels in terms of synthesis, best can measure (Table F6.
Systematic Reviews. In the case of systematic reviews, we can have only binary measure of significant effect (or not). As proxy for Zig. we again use RobotRevi ewer extract effect for each of the model inputs Tıj. using the majority vote (i.c., do the plurality of zij E X, indicate that there was effect). We classify each output candidate in C, again using Robot estimate zig. We then select for output the highest probability candidate in C, which agrees with the majority vote of the inputs, and abstain where there are no viable candidates. For the models we do choose a summary for, we find performance similar to our measure (). Movie reviews show a wide range of sentiments; systematic reviews show some improvement but are biased towards no effect (qualitatively observed in Appendix G).
to
7 RELATED WORK
Automatic (multi-document) summarization (Nenkova & McKeown 2011 Maybury 1999) has
been an active subfield within NLP for decades. We have focused our analysis on modern, neural abstractive models for conditional text generation (Bahdanau et al. 2015). In light of their empirical
8

---

Under review as a conference paper at ICLR 2023
R
MSE
Pearson's r
R1
MSE
Pearson's r
R1
LED
0.656
0.032
0.821
0.229
LED
0.763
0.022
0.878
0.227
Pegasus
0.694
0.029
0.835
0.229
Pegasus
0.799
0.019
0,894
0.232
PRIMERA
0.749
0.024
0.880
0.240
PRIMERA
0.890
0.011
0.948
0.240
T5
0.721
0.026
0.856
0.231
0.876
0.012
0.938
0.230
Reference
0.697
0.023
0.836
Reference
0.697
0.023
0.836
Table Movie Reviews: Generate diverse movie meta-reviews and then choose among them using an approximate target sentiment (left) or the oracle sentiment (right), RI is ROUGEI score.
FT
Ahstenntion
ROUGET
Abstention-Oracle
ROUGE1-Oracle
LED
0.557
0.386
0.252
0.233
0.259
PRIMERA
0.581
0.336
0.251
0.213
0.248
T5
0.568
0.350
0.202
0.228
0.210
Pegasus
0.588
0.383
0.211
0.242
0.225
Reference
0.577
Table 5: Systematic Review results with modified-then-seeeene predictions. Fl is macro-averaged Fl on the set of returned results. We abstain when no output matches the expected synthesis result.
success, we have specifically evaluated set of Transformer-based (Vaswani et al., 2017 models which have recently been used for multi multi-document summarization (Beltagy et al. [2200] Zhang et al., 2020 Xiao et al., 2021 Raffel et al.) 2020). There has been some work on highlighting conflicting evidence in health literature specifically (Shah al. 2021h a), though this focused primarily on highlighting conflicting evidence, and explicitly aggregating extracted content.
Sentence fusion One view synthesis might be that is a particular kind of sentence fusion (Barzi- lay & McKeown, 2005). However, past work on "fusing" sentences has assumed that the aim is to generate an output that contains the information common similar sentences (Thadani & McKe- own 2013). This is intuitive in the context of, e.g., summarizing multiple news articles covering the same event. But here we are interested in the more challenging setting in which the output should reflect an aggregate measure of potentially conflicting evidence or opinions.
Interpretation and analysis of neural models for NLP This work is also related the emerging body of work on analyzing neural NLP models, their behaviors, "knowledge", and "abilities" in general e.g.. Linzen al. (2016): (enney et al. (2019): Petroni et al. (2019); Niven Kao (2019); analyzing neural summarization
Meng et al. (2022). There has been some work specifically models. Xu et al. (2020a) investigated when model is likely to extract (copy) rather than abstract (generate). Xu & Durrett (2021) furthered this analysis by assessing when models were relying on the local input to produce particular output tokens, and when they instead rely on mostly on a background language distribution acquired in pre-training.
Factuality neural summarizers Neural conditional generation models have proven adept at pro- ducing fluent outputs, but in the context of summarization they are prone hallucinating content unsupported by input documents (Maynez et al.) 2020; Kryscinski et al. 2019). Automated metrics
such ROUGE do not reliably capture such phenomena (Falke al. 2019 Maynez et al. 2020
This has motivated several efforts to design automated factuality metrics (e.g., Wang et al. (2020b)
Xu al. (2020b); see Pagnoni al. 2021 for an overview).
8 CONCLUSIONS
We have outlined and investigated the problem of synthesis as related to some summarization tasks. We showed that existing models are partially able to synthesize implicitly, but do so imperfectly: For instance, the aggregation they perform is sensitive to input ordering, and they are not as sensitive to perturbations the composition of inputs as one would hope. We proposed and validated a straightforward inference time method to improve model synthesis capabilities by preferentially outputting summary candidates that align with a predicted aggregate measure, and demonstrated empirically that this offers gains in performance. Our hope is that this work encourages additional research into summarization models that explicitly optimize to accurately synthesize potentially conflicting evidence and information.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
Regina Barzilay and Kathleen R McKeown. Sentence fusion for multidocument news summariza- tion. Computational Linguistics, 31(3):297-328, 2005.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. ArXiv, abs/2004.05150, 2020.
Hoa Trang Dang. Overview of duc 2005, In Document Understand Conference, 2005.
Hoa Trang Dang. Overview of DUC. In In Proceedings of HLT--AACL, 2006.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computarionnll Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1111653/1/N119-4222
https:
Jay Young, Eric Lehman, Benjamin Nye, lain Marshall, and Byron Wallace. Evidence infer- ence 2.0: More data, better models. In Proceedings of the 19th SIGBioMed Workshop on Biomed- ical Language Processing, pp. 123-132, Online, July 2020. Association for Computational Lin- guistics. doi: 10.18663//1/2020.bbiol wii.:/...oonnp.111 1.13. URL ULL/ttpps///aalaattcooooo.oo// bionlp-1.13
Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. MS'2: Multi-document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7494-7513, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URI. emn emnlp-ma 594
httts:://cclame waachhooogge
David K Evans, Judith L Klavans, and Kathleen McKeown. Columbia newsblaster: Multilingual news summarization the web. In Demonstration Papers HLT-NAACL 2004, pp. 1-4, 2004.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//11-11022 URL
https: htt///aaaanttt aclanthologyy.....o www.ttooooo.oog//1-----11.. org/P19-1102
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Rank- ing generated summaries by correctness: An interesting but challenging application for natural language inference. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2214-2220, 2019.
César Ferri, Peter Flach, and José Heráández-Orallo Delegating classifiers. In Proceedings of the twenty-first international conference on Machine learning, pp. 37, 2004.
Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. A large-scale multi-document summarization dataset from the wikipedia current events portal. arXiv preprint arXiv:2005, 2020.
Yotam Hechtlinger, Barnabás Póezos, and Larry Wasserman. Cautious deep learning. arXiv preprint 10X5091005.09460, 2018.
Gregory Kell, Iain Marshall, Byron Wallace, and Andre Jaun. What would it take to get biomed- ical QA systems into practice? In Proceedings of the 3rd Workshop on Machine Reading for Question Answering. pp. 28-41, Punta Cana, Dominican Republic, November 2021. As- sociation for Computational Linguistics. doi: 10.18653/v1/2221 w.w.aamraaa... 1.3. URL https: 7/aclantholoog.oor
10

---

Under review as a conference paper at ICLR 2023
Wojeiech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNIP--JCNILP), pp. 540-551, Hong Kong, China, 2019.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. ArXiv, abs/1910. 1ete/191..12400,,, 2020.
Stefano Leone.
Rotten
tomatoes
movies
and
critic reviews
dataset.
https: htt.op////nuugggge...ooo/rrrrrrr
2020.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521- 535, 2016.
Peter Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam M. Shazeer. Generating wikipedia by summarizing long sequences. ArXiv, Abs//800,,0019, 2018
Yao Lu, Yue Dong, and Laurent Charlin. Multi-xscience: A large-scale dataset for extreme multi- document summarization of scientific articles. arXiv preprint arXiv:2010. 14235, 2020.
Andrew I. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011, Association for Computational Linguistics. URL http:
7/www. www.b.clweb.oo org/ wwthalogyhhhll111 1015
lain J Marshall, Joël Kuiper, Edward Banner, and Byron C Wallace. Automating biomedical evi- dence synthesis: Robotreviewer. In Proceedings of the Association for Computationnal Linguistics, volume 2017, pp. 7, 2017.
Mani Maybury. Advances in automatic text summarization. MIT press, 1999.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factu- ality abstractive summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1906-1919, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.0526)) 2022.
Diego Mollá and María Elena Santiago-Martinez. summarisation. The Australasian medical journal, 5(9):503, 2012.
Martínez. Creation of a corpus for evidence based medicine
Feng Nan, Cícero Nogueira dos Santos, Henghui Zhu, Patrick Ng. Kathleen McKeown, Ramesh Nallapati. Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. Improving factual consistency of abstractive summarization via question answering. ArXiv, abs/2105.04625 2021a.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cícero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. Entity-level factual consistency of abstractive text summarization. ArXiv, 1bs/2102.09930, 2021b.
Ani Nenkova and Kathleen McKeown. Automatic summarization, Now Publishers Inc, 201
Timothy Niven and Hung- Yu Kao. Probing neural network comprehension of natural language arguments. arXiv preprint 1rXiv:1907.000 2019.

---

Under review as a conference paper at ICLR 2023
Nadav Oved and Ran Levy. PASS: Perturb-and-See summarizer for product reviews. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 351-365, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.act-long.30 URL http:pp:/aacllaacoooooo..oore/ee
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL), pp. 4812-482222 4829, 2021.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Associa- tion for Computational Linguistics (ACL'05), pp. 115-124, Ann Arbor, Michigan, June 2005.
Association for Computational Linguistics. doi: 10.3115/1219840 //aclanthology.. wttpo/oooy..r///511111111
URL https:
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint erXiv:/909.000 00066, 2019.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910.10683, 2020.
Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. Questeval: Summarization asks for fact-based evaluation. In EMNLP, 2021.
Darsh Shah, Lili Yu, Tao Lei, and Regina Barzilay. Nutri-bullets hybrid: Consensual multi- document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 5213- 5222, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2011 ma1.l-main.411. URL httppsss/aaaaatthooooo.o org. 202l
411
Darsh J Shah, Lili Yu, Tao Lei, and Regina Barzilay. Nutri-bullets: Summarizing health studies by composing segments. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 17780-13788, 2021b.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In EMNLP, 2013.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference Learning Representations, 2019. htttthtts:///poeeeee
wttp:/ewwwwee..trree forum?
Cd-SSZSgnRcKX
Kapil Thadani and Kathleen McKeown. Supervised sentence fusion with single-stage inference. In Proceedings of the Sixth International Joint Conference Natural Language Processing, pp-
1410-1418, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural se- quence models. arXiv preprint arXiv:1610.02424 2016.
Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and lain James Marshall. Generating (fac- tual?) narrative summaries of rets: Experiments with neural multi-document summarization. ArXiv, his/2008.1199) 2020.
12

---

Under review as a conference paper at ICLR 2023
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In ACL, 2020a.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5008-5020, 2020b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference
Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: /www a acl www.wwew......m. www.orrg//aaotooooooooooooo
2020. emn em200.emmmp-demss.
Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Dagan, How "multi" is mulli--dcuumett summa- rization?, 2022. URL https:://axxiiiii org/ http//bb/2220.... /2t..221...... 12688
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. Primer: Pyramid-based masked sentence pre-training for multi-document summarization. ArXiv, abs/2110.004999 2021.
Jiacheng Xu and Greg Durrett. Dissecting generation modes for abstractive summarization models via ablation and attribution. arXiv preprint arXiv:2106.015/8, 2021.
Jiacheng Xu, Shrey Desai, and Greg Durrett. Understanding neural abstractive summarization mod- cls via uncertainty. arXiv preprint arXiv:2010.07882, 2020a.
Xinnuo Xu, Ondiej Dušek, Jingyi Li, Verena Rieser, and Ioannis Konstas. Fact-based content weighting for evaluating abstractive summarisation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5071-5081, 2020b.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre- tre-training with extracted gap-sentences for abstractive summarization. ArXiv, abs/1912.087777, 2020.
13

---

Under review as a conference paper at ICLR 2023
A
NOTATION
Variable
Definition
X;
The 1 set of input documents, corresponding instance
The target summary of the jth instance
94
The g token of target V: A generated summary of the hth instance
Vi.t
The 5th input document for instance i
A particular linearization of the imput documents X;
0
Model & parameters
po
Probabily under parameters 0
1 1..e-1111 ax))
Standard auto-regressive prediction of the next token given an input and partial summary
Latent property (sentiment, significance finding) of Zij
2.
Aggregated latent property (sentiment, significance finding) of X;
2.g
Latent property measured over summary ý
G
Aggregation function over latent properties Zij. yields 2; Auxillary function to measure latent property =ij of Tij. or Zig of A weight for žij
9
Oci)
Table 6: Notation.
B ADDITIONAL RELATED WORK
Shah et al. a created "nutri-bullets" system for generating consensus-based summaries of health and nutrition related content. They assume a low-supervision setting in which one has a set of tuples extracted from texts with which to train a content extractor, and where one can design heuristic rule-based aggregation strategies on top of extracted tuples mapping onto discrete categories like "consensus". By contrast, we assume more typical supervised summarization setting and are interested in continuous aggregation of latent attribute of interest, and we do not assume (or have access to) relational tuples over inputs. Indeed, recent work Wolhandler et al. (2022) has shown that systematic reviews are categorically different than news summarization, and that relational tuple extractors do not perform well in the medical domain.
First, Shah et al. (2021a)'s 's focus primarily settings in which training data is (severely) limited, and motivate their pipeline approach on the basis of this limited supervision assumption. For this reason they define separate modules: The first performs content selection (tuple extraction; this does require manual annotations of tuples on a subset of texts to train such an extractor); The second applies (manually composed) deterministic aggregation rules over these extracted tuples combine them; a final module then generates a "surface realization" conditioned on the aggregated result.
We have investigated more typical supervised settings (with thousands of input and summary pairs), and we are training modern end-to-end transformer-bas based summarization models. We have empiri- cally assessed the extent which model outputs in this typical training regime are consistent with the continuous synthesis result anticipated. We do not have annotated tuples on our inputs (as would be required to use the [Saah et al. (2021a) approach, as it assumes trained content extractor mod- ule). And while applying discrete (manually composed) aggregation operators over inputs makes sense some settings, we are explicitly interested in the ability of models to aggregate variables of interest continuously, for example producing "very positive" summaries when movie reviews are overwhelmingly positive, and merely "positive" "summaries when they are only mostly positive.
In sum, the approach proposed by Shah et al. (202 is appropriate in, and designed for, low- supervision settings (which we do not consider here) where there are natural "tuples" to be extracted from inputs and supervision for this sub-task (which we do not have) and where discrete aggregation categories of inputs is natural (whereas we are interested in continuous aggregation, e.g., mean sentiment).
Wolhandler et al. (2022) attempts to measure how challenging multi multi-document summarization is, as function of the unique knowledge (represented as relational tuples) required to produce summary.
14

---

Under review as a conference paper at ICLR 2023
Model
Hugginglace Checkponni
Opumizer
Senedule
Warmup
Senoodhing
allesabteS-hoe-
Paominkderay
steps
15
t5-base
Lineos
stops
PRIMERA
allesaoPRMMRR
Adam
Linese
stepp
D
Pegasus
Adaffanncr
Linesr
500 0.1
Table 7: Model hyperparameters. We used optimizers, schedulers, weight decay, and label smooth- ing as best according to examples from source implementations (where available). Optimizer warmup was arbitrarily chosen. Non-specified parameters were the Huggingface defaults.
This work measures how many new tuples each input document might add in contrast to subsets of other inputs. By greedily building subsets of imputs function of new information added, they find that standard multiple document summarization datasets merely need to select two to four documents from inputs of up ten, whereas their approach breaks down in the case of systematic reviews. They find that due to both technical constraints for relation extraction, as well as the inability to model contradiction, relational extraction and aggregation methods insufficient for producing evidence syntheses.
Oved & Levy 2021 introduce the Perturb and Selection Summarizer (PASS) system for summariz- ing Amazon product reviews. It works by perturbing model inputs (i.e. keep random subsets of the input), generating summary for each perturbation (via standard beam search), and then selecting amongst outputs (via a ranker) to produce a coherent, self ooff-consistent, and fluent summary.
PASS similar our work in that it generates multiple outputs and selects amongst them. However it differs in several key respects. The key conceptual difference between PASS and our work is that PASS's target summary's s coffcconsittency (a product review might contradict itself on some aspect, e.g. simultaneously discussing product fitting well in addition to the product running a size small), whereas our target continuous fact derived from the inputs as a whole (e.g. aggregate sentiment or effect sizes). PASS is designed to produce summaries that are plausible, as opposed (and complementary) to summaries that reflect inherent contradiction in the input data. As PASS produces summaries from subsets of cach instance's s input, it cannot perform explicit synthesis on its own, as opposed to our work, wherein each summary was produced with access to the whole of each instance's input.
C MODELS
We train all models using modified Huggingface Transformers library Wolf et al. (2020). For the Pegasus model, we use a distilled version provided by Huggingface (Table 7). All models were trained using their default hyperparameters, except for batch size, optimization steps, learning rates, and any parameters specified in Table 7 We fix our batch size to 16, using gradient accumulation over single instances at time, with floating point 16 (fp16) precision (due to data size), and perform an approximate (subject to resource constraints) grid search over learning rates and training steps (Tables 8 keeping the model highlighted in bold. Earlier experimentation with Longformer and T5 models only; we found that while lower numbers of steps could perform sensitive to hyperparameter changes than longer runs. All training was performed on 48G NVIDIA RTX8000 GPUs, most models are unable to fit single instance gradient information into fewer than 40G, even reduced precision.
performed ad- id-hoc
well, they had high variance and were
D DETAILED RESULTS
Measure Validation As results rely using proxy metrics, measure the quality of these proxies. See Figure g for movie meta-review sentiment correlation with human results, and Table for how well the automatic significance measures correlate with the underlying truth.
Diversity Sampling. We include detailed results for the importance of diversity sampling: the di- versity sampling procedure produces better metrics in every dimension (Table II top left vs. bottom left.). In the systematic reviews, most metrics drop slightly and abstention increases substantially (Table 12
15

---

Under review as a conference paper at ICLR 2023
model
steps
rougel
rouge2
rougel.
led
1000
le-5
25.09
8.95
20.14
led
1000
le-6
25.25
8.33
19.48
led
1000
3e-5
25.33
8.65
19.87
led
1000
5e-5
25.14
8.35
19.89
led
5000
le-5
25.46
8.63
19.76
led
5000
le-6
25.31
8.76
20.12
led
5000
3e-5
24.50
7.49
19.02
led
5000
5e-5
23.99
7.07
17.61
led
10000
le-5
24.28
7.60
19.25
led
10000
le-6
25.58
8.64
20.32
led
10000
3e-5
25.60
7.97
19.59
pegasus
1000
le-3
23.49
7.25
17.93
pegasus
1000
le4
22.25
7.23
17.67
pegasus
1000
le-5
18.95
4.23
14.34
pegasus
1000
le-6
18.28
3.10
13.08
pegasus
2500
le-3
26.44
9.30
20.45
pegasus
2500
le4
26.50
10.81
20.91
pegasus
2500
le-5
24.98
10.17
19.75
pegasus
2500
le-6
23.02
7.92
18.23
pegasus
5000
le-3
24.05
7.75
18.98
pegasus
5000
lc-4
27.41
10.26
21.72
pegasus
5000
le-5
25.67
9.86
20.28
pegasus
5000
le-6
23.57
8.74
18.77
pegasus
10000
le-3
23.18
7.17
17.19
pegasus
10000
lc-4
27.42
9.53
21.05
pegasus
10000
le-5
25.85
10.25
20.41
pegasus
10000
le-6
24.41
9.88
19.72
primera
2500
le-4
23.32
7.02
18.10
primera
2500
le-5
25.12
8.39
19.52
primera
2500
le-6
24.92
8.48
19.93
primera
5000
le-4
24.35
7.40
18,49
primera
5000
le-5
25.42
8.44
19.81
primera
5000
le-6
25.32
8.75
20.06
primera
10000
le-4
23.57
7.24
17.89
primera
10000
le-5
24.27
7.59
18.55
primera
10000
le-6
25.39
8.66
20.12
5
T000
le-4
25.24
9.13
19.67
1000
le-5
24.31
7.87
19.30
t5
1000
le-6
22.39
6.62
17.90
1000
5e-5
25.06
8.65
19.96
t5
2500
1e-4
25.82
8.46
19.59
2500
le-5
24.94
8.36
19.61
2500
le-6
23.82
7.59
19.09
15
2500
5e-5
25.57
8.47
19.71
5000
le-4
25.11
8.17
19.84
15
5000
le-5
25.07
8.57
19.55
5000
le-6
23.87
7.99
19.36
15
5000
5e-5
24.47
8.40
19.53
7500
le4
24.33
7.58
18.93
15
7500
le-5
25.67
8.75
19.92
7500
le-6
24.17
7.73
19.43
15
7500
5e-5
25.66
8.64
19.59
10000
le-4
24.41
7.78
18.89
15 10000
1e-5
25.73
8.91
20.10
15
10000
1e-6
24.29
7.98
19.31
15
10000
5e-5
25.09
7.89
19.19
Table 8: Movie Reviews dev training results, best models bolded.
16

---

Under review as a conference paper at ICLR 2023
Model
steps
1r
ROUGEI
ROUGE2
ROUGEL
led
250
5e-5
24.30
6.70
18.68
led
500
5e-5
23.99
6.91
17.09
led
1000
5e-5
25.21
7.31
18.54
led
2500
5e-5
26.05
6.52
17.65
led
5000
5e-5
30.96
8.65
20.33
pegasus
250
le-4
20.22
5.64
15.86
pegasus
500
le-4
21.66
6.71
16.92
pegasus
1000
le-4
21.87
6.67
16.79
pegasus
2500
le-4
22.44
6.71
17.24
pegasus
5000
le-4
22.66
5.69
16.87
primera
250
le-4
23.21
7.68
17.96
primera
500
le-4
22.80
5.72
16.39
primera
1000
le-4
26.08
6.70
16.77
primera
2500
le-4
27.60
6.99
18.16
primera
5000
lc-4
27.70
6.96
18.02
primera
250
5c-5
22.52
6.90
17.98
primera
500
5e-5
24.22
6.11
17.53
primera
1000
5e-5
24.53
6.71
17.01
primera
2500
5e-5
27.64
6.89
18.79
primera
5000
5e-5
28.46
6.59
18.08
t5
250
5e-5
23.80
7.09
18.58
t5
500
5e-5
22.77
6.99
18.30
55
1000
5e-5
0.00
0.00
0.00
15
2500
5e-5
0.00
0.00
0.00
Table 9: Systematic Reviews dev training results, best models bolded. We experimented with other parameters (in particular learning rates), and found that total number of steps was more important.
2.0
LED
PRIMERA
5.2
Sentment
3.6
75
Pegasus
3.2
Predicted Metareview Sentiment
Figure 8: Actual sentiment vs. predicted sentiments on model outputs.
E ROUGE RESULTS
We report mean differences in ROUGE outputs for both datasets in Figure 10 Ideally, these would have all mass zero.
17

---

Under review as a conference paper at ICLR 2023
Precision
Recall
Fl-score
Support
No significant difference
0.726
0.870
0.792
247
Significant difference
0.500
0.283
0.362
113
Accuracy
0.686
360
Macro avg
0.613
0.577
0.577
360
Table 10: Systematic review significance validation results.
R²
MSE
Pearson's r
RI
R
MSE
Pearson's s
R1
LED
0.656
0.032
0.821
0.229
LED
0.711
0.027
0.847
0.240
Pegasus
0.694
0.029
0.835
0.229
Pegasus PRIMERA
0.705
0.028
0.840
0.247
PRIMERA
0.749
0.024
0.880
0.240
0.731
0.025
0.857
0.255
TS
0.721
0.026
0.856
0.231
TS
0.669
0.031
0.819
0,253
Reference
0.697
0.023
0.836
Reference
0.697
0.023
0.836
R²
MSE
Pearson's
R1
R²
MSE
Pearson's is
R1
LED
0.653
0.033
0.815
0.241
LED
0.763
0.022
0.878
0.227
PEGASUS
0.649
0.033
0.809
0.248
Pegasus
0.799
0.019
0.894
0,232
PRIMERA
0.685
0.029
0.833
0.254
PRIMERA
0,890
0.011
0.948
0,240
TS
0.615
0.036
0.786
0.252
TS Reference
0.876
0.012
0.938
0.230
Reference
0.697
0.023
0.836
0.697
0.023
0.836
Table 11: Movie Reviews. Top left: Generate 5 diverse movie meta-reviews and then choose among them using an approximate target sentiment. Top right: Generate 25 diverse movie meta-reviews and then choose among them using approximate target sentiment; this was accidentally referenced in an earlier version this work. Bottom left: Generate movie meta-reviews using standard beam search and choose among them using approximate target sentiment. Bottom right: Generate diverse movie meta-reviews and select amongst them using the oracle sentiment. In all cases R. refers to ROUGEI.
FI
Abstention
ROUGEI
Abstention-
ROUGE1 -Oracle
LED
0.521
0.503
0.258
0.358
0.263
PRIMERA
0,551
0.464
0,256
0.342
0.248
T5
0.546
0.422
0.204
0.328
0.211
Pegasus
0.589
0.469
0.211
0.281
0.222
Reference 0.577
Table 12: Systematic reviews results with multiple gemerate-then-selle predictions, this time using the top-5 results from standard beam-search. Fl is a macro-averaged Fl on the set of returned results. We abstain when output matches the expected synthesis result.
18

---

Under review as a conference paper at ICLR 2023
1.00
Pegasus
1.80
LED
Pegasus
0.50
0.50
0.25
0.25
0.00
1.00
PRIMERA
TS
Density
PRIMERA
T5
0.75
0.75
0.25
0.25
0.00
0.00
Average Predicted Effect Entropy Significant
Entropy Average Predicted Effect
Figure 9: Entropy of instance predictions. Broken out by whether or not the underlying truth not significant (left); or the whole dataset (right)
LED
PRIMERA
0.4
LED
Pegasus
0.2
0.2
0.1
0.1
Densty
Pegasus
T5
0.3
0.2
0.2
Mean ROUGE] Difference
ROUGE1 Difference from Mean
Figure 10: ROUGE1 deviations from instance means for movie reviews (left) and systematic reviews (right).
F EXAMPLES OF DIVERSE MOVIE SUMMARIES
Summary
Sentiment
The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships.
0.800731
The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships.
0.800731
With strong cast and Robin Wright Penn's sharp performance, The Private Lives Pippa Lee
0.809596
succeeds both witty tribute lost characters and showcase for Robin Wright Penn.
With strong cast and Robin Wright Penn's empathetic direction, The Private Lives Pippa Lee succeeds as both humorous look at domestic issues and poignant look relationships.
0.80908
The Private Lives Pippa Lee benefits from Robin Wright Penn's superb per- formance, as well as strong ensemble cast that includes Keanu Reeves, and Faye Dunaway.
0.845693
The Private Lives of Pippa Lee has an affecting ensemble cast and Robin Wright Penn delivers a noteworthy performance, although the film is bit too episodic.
0.654905
Table 13: Different meta-review: of "The Private Lives of Pippa Lee" and corresponding sentiments. The target sentiment for this meta-review is 70%, generating diverse candidates helps find a meta- review closer the target.
19

---

Under review as a conference paper at ICLR 2023
Summary
Sentiment
You Don't I Mess With the Zohan's s handful of laughs are almost enough to com-
0.242698
pensate for its inconsistent tone and stale, obvious jokes.
You Don't Mess with the Zohan has a handful crotch thrusts, but not enough 0.429654
of them land.
You Don't t Mess With the Zohan's 's handful of laughs are almost enough to com-
0.287896
pensate for its aimless, crass script.
You Don't Mess with Zohan has its moments, but not all of them and the 0.434442
jokes embarrassingly crass and often crude.
You Don't 't' Mess with the Zohan has its moments, but not all of them and the 0.406172
jokes are embarrassingly crass and often crude. The script
Table 14:
Different meta-reviews for YYo Don't 't Mess With The Zohan"; relatively panned movie with a target meta-review sentiment of 37%.
G EXAMPLES OF DIVERSE SYSTEMATIC REVIEWS
Generated
Effect significant difference
Ketanserin versus placebo in the Raynaud's phenomenon is neither ef- fective nor safe. The Raynaud's phenomenon associated with signif- icant adverse effects including dizziness and pain. The effectiveness of ketanserin for the Raynaud's phenomenon is unknown.
Ketanserin versus placebo in the Raynaud's phenomenon is neither effective nor safe. The Raynaud's phenomenon is associated with sig- nificant adverse effects including dizziness and pain. Ketanserin and serotonin receptor antagonists in the Raynaud's phe- nomenon treatment of systemic seleroderma reduce the incidence of ischaemic ulcers and may reduce the frequency of adverse events. The Raynaud's 's phenomenon is associated with small number of ad- verse effects when administered orally to patients with Raynaud's phe- nomenon. The frequency of Raynaud's phenomenon is similar to that of other drugs. However, there is little evidence to aid the treatment of Raynaud's 's phenomenon.
no significant difference
significant difference
significant difference
The Raynaud's s phenomenon is associated with a small number of ad- verse effects when administered orally to patients with Raynaud's 's phe- nomenon. The frequency of Raynaud's phenomenon is similar to that of other drugs.
no significant difference
Table 15:
An instance where generating multiple reviews allows our models to find a candidate summary reporting significant difference (the target).
20

---

Under review as a conference paper at ICLR 2023
Generated
Effect
The overall evidence supports the use of topical antibiotics in surgical patients who have undergone minor surgery, compared to no treat- ment. The effect other outcomes, other than infection rate, is con- sistent. The safety profile of topical antibioties also concern. Further well-designed RCTs are needed to assess effectiveness of top- ical antibiotics in surgical patients.
no significant difference
single application of topical antibiotics in surgical site wounds re- duces the risk of infection, and the risk of other complications, includ-
no significant difference
ing wound dehiscence. The risk of infection recurrence is low. The use of topical antibiotics outside of surgery should be restricted to sur- gical site wounds.
single application of topical antibiotics in surgical site wounds re- duces the risk of infection, and the risk of other complications, includ- ing wound dehiscence. The risk of infection recurrence is low. The overall evidence supports the use of topical antibiotics in surgical patients to reduce the risk of infection, and the risk of other compli- cations, especially in high-risk patients. There is lack of evidence in low-risk patients to support the use of topical antibiotics in this set- ting-
significant difference
significant difference
single application of topical antibiotics in surgical site infection pre- vention has been demonstrated to reduce the risk of infection in pa- tients who have undergone surgery. The number of patients who have been treated with topical antibiotics has been small but this is due risk of bias in the trials. Ointment use should be limited to patients whose primary wound is irradiated.
significantly difference
An instance where generating multiple reviews allows our models to find a candidate summary reporting significant difference (the target).
Table 16:
21


Reviews:
[{"review_id": "e1Ux9uU4UhG", "reviewer": "Reviewer_Z9mN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the synthesis problem in multi-document summarization. Then, this paper proposes a simple method: Given a set of documents, the proposed method generates diverse candidates using the diverse beam search and selects one candidate that aligns with the expected aggregate property of inputs.", "summary_of_the_review": "The paper shows insights about the synthesis of the existing multi-document summarizers. However, the experiments lack to prove the superiority of the proposed method.", "strength_and_weaknesses": "Strengths:\n\n- This paper tackles one of the important issues in multi-document summarization: the existing multi-document summarizers can synthesize inputs properly.\n- This paper includes extensive analysis with statistical results on two data sets.\n\nWeaknesses:\n\n- The main criticism is that the diverse beam search (DBS) does not generate diverse candidates with various ``properties''. The DBS generates a set of candidates by penalizing the generation of strings similar to candidates on beams in other groups. In DBS, the similarity is only considered at the token level, not the property of interest level. DBS is not guaranteed to generate diverse candidates with various properties of interest.\n- The proposed strategy 'generate-diverse-then-select' should be compared with the strategy 'generate-then-select.' This paper has no empirical studies to show the superiority of the 'generate-diverse-then-select.'\n- There is no report of the ROUGE1 score on movie review experiments (Table 4). The main goal of the proposed strategy is to select a summarization that aligns with the expected property of inputs and the target summary. Only showing $R^2$, MSE, and Pearson's r results cannot prove the superiority of the proposed method because it selects the wrong summarization but aligns with the expected property.\n- The proposed method can do a cautious summarization, but there is no evaluation of the cautionary of the summaries. Human evaluations or correlations of automatic factuality metrics should be included.\n- There are two versions to predict sentiment on movie data set: BERT trained on the SST data set, and BERT trained on the IMDB data set. It is a wonder why this paper adopted different models.\n- Some evaluation settings are missing, like the number of beams. Furthermore, selecting the true sentiment in Table 4 is unclear. Please add the details to the appendix.\n- As mentioned in the introduction, the problem considered in this paper is only for some applications, such as movies or medical, not all applications for multi-document summarization, like news. The reviewer thinks the title is too generalized for what the paper considers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bF6UNP1VjMq", "reviewer": "Reviewer_YA3t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to tackle the problem of multi-document summarization through synthesis evaluation. Do summarization models synthesize large and varied inputs? The paper studies this question from two angles: movie reviews, where the meta-review is human-written and scores are given (roughly equivalent to sentiment scores); and biomedical systematic reviews of treatments, where multiple systematic reviews summarize clinical trials, resulting in statistical meta-analysis results that can be used as scores. The authors make a good choice of pre-trained models to analyze and show that they do not synthesize a large body of input as well. Then they further analyze how the models fare when we switch the input ordering, or the balance of the input (by ablation). Finally, they propose their own post-hoc method consisting of generating a diverse set of output candidates, and then select the set that best corresponds to an expected synthesis result obtained using an aggregate score predicted by an external pre-trained model.", "summary_of_the_review": "While the authors propose an empirically-supported method to improve synthesis in summarization models, their analysis is lacking as they have evaluated models against input ordering and re-balancing, which the models could be trained for but were not. The result is a good contribution, but the analysis suggests that a training-time method can improve the scores further. The high rate of abstention is another flaw that hints at a required change in training rather than inference.", "strength_and_weaknesses": "Strengths:\n- The paper tackles an important problem and proposes a simple post-hoc (inference time) method for improving synthesis in summarization models\n- The metrics provided are automatic and rely on human-written summaries, making the evaluation strong/fair\n\nWeaknesses:\n- The ordering analysis in section 5.2 could result in different results if we augmented the training data by switching the order of the reviews for example.\n- Likewise, the analysis in section 5.3 addresses something that the models were not trained for. The results could see a shift if there is data augmentation with the expected result as the label.\n- The abstention percentage is between 30 and 40%, which is quite high and shows the need for a change in training rather than inference.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VSGPLjiTdq", "reviewer": "Reviewer_ACAC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the capacity of summarization models to synthesize (potentially conflicting) information from multiple documents. It defines a model-based metric for the aggregate latent aspect of interest, in this case, the sentiment of movie reviews (Rotten Tomatoes dataset) and the treatment efficacy of medical trials. This metric is applied to reference summaries as well as summaries generated by Longformer, PEGASUS, PRIMERA, and T5. Then, the results are compared to gold labels from the datasets so that a higher correlation would be an indicator for better synthesis capacity. The experimental results indicate inconsistent synthesis for off-the-shelf summarization models. The authors then propose a technique for improving model synthesis by ranking candidate summaries according to the expected aggregated aspect.", "summary_of_the_review": "This work presents a novel way to evaluate multi-document summarization and its experimental methods are sound. I would recommend the authors to address the experimental improvements listed above to make the claims stronger.", "strength_and_weaknesses": "Strengths:\n1. The paper presents an interesting way to evaluate summarization that complements traditional ROUGE scores.\n2. The experiments are conducted in two quite different domains.\n\nWeaknesses:\n1. It is not clear why the authors decide not to fine-tune the measurement model on the datasets, especially for Rotten Tomatoes. The BERT and RobotViewer models were trained on human-written inputs and are now used on machine generated summaries, which might detrimental for its performance (and correlations to gold sentiments).\n2. The summarizers are the \"base\" models and the gap in quality with respect to the large versions could change some of the conclusions. At least one experiment with a large version (PEGASUS, for instance) could make the empirical evidence much stronger.\n3. Citation missing: Oved and Levy (2021) also used a strategy to generate several summaries and rank them according to a desired criteria. https://aclanthology.org/2021.acl-long.30/\n\nMinor comments:\n1. Figure 10 in the Appendix is cut.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ax5S4x_kJZ", "reviewer": "Reviewer_18KJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the synthesizing capabilities of multi-document summarization models and presents a method for increasing those capabilities in summarization. \nSynthesizing capabilities are measured with respect to an application-dependent latent factor, the model is required to generate outputs that align with that factor in the input. Two cases have been studied: movie reviews with the latent factor of sentiment, and medical study results with the latent factor of significant effects for interventions. The investigated models proved to be lacking in synthesizing capabilities, but the proposed method raised these to human level (with regard to the metrics).", "summary_of_the_review": "The paper addresses an important issues for multi-document summarization, and show empirical evidence that this issue persists in SoA summarization models. Conceptualisation and experiments are generally sound. It misses a detailed discussion of one highly relevant work, and it's inclusion in the comparison. The paper is generally reproducible from the given details, inclusion of source code would make more so. ", "strength_and_weaknesses": "Strengths:\n[S1] The paper extends the conceptualization of desiderate for multi-document summaries to synthetization.\n[S2] The paper is well written and methodological sound.\n[S3] The proposed measures, given that factor information is available, could be used as evaluation metric for MDS tasks.\n\nWeaknesses: \n[W1] The novelty of the method is limited. \n[W2] Some questions in the design and results of the experiment remain open. \n[W3] A stronger baseline could have been used. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject