Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the capacity of summarization models to synthesize (potentially conflicting) information from multiple documents. It defines a model-based metric for the aggregate latent aspect of interest, in this case, the sentiment of movie reviews (Rotten Tomatoes dataset) and the treatment efficacy of medical trials. This metric is applied to reference summaries as well as summaries generated by Longformer, PEGASUS, PRIMERA, and T5. Then, the results are compared to gold labels from the datasets so that a higher correlation would be an indicator for better synthesis capacity. The experimental results indicate inconsistent synthesis for off-the-shelf summarization models. The authors then propose a technique for improving model synthesis by ranking candidate summaries according to the expected aggregated aspect.
strength_and_weaknesses: Strengths:
1. The paper presents an interesting way to evaluate summarization that complements traditional ROUGE scores.
2. The experiments are conducted in two quite different domains.

Weaknesses:
1. It is not clear why the authors decide not to fine-tune the measurement model on the datasets, especially for Rotten Tomatoes. The BERT and RobotViewer models were trained on human-written inputs and are now used on machine generated summaries, which might detrimental for its performance (and correlations to gold sentiments).
2. The summarizers are the "base" models and the gap in quality with respect to the large versions could change some of the conclusions. At least one experiment with a large version (PEGASUS, for instance) could make the empirical evidence much stronger.
3. Citation missing: Oved and Levy (2021) also used a strategy to generate several summaries and rank them according to a desired criteria. https://aclanthology.org/2021.acl-long.30/

Minor comments:
1. Figure 10 in the Appendix is cut.
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and easy to follow. Experimental details and hyperparameters are well-defined for reproducibility. The proposed evaluation approach presents novelty but the idea of generating several candidate summaries is already explored in previous research.
summary_of_the_review: This work presents a novel way to evaluate multi-document summarization and its experimental methods are sound. I would recommend the authors to address the experimental improvements listed above to make the claims stronger.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the synthesis problem in multi-document summarization. Then, this paper proposes a simple method: Given a set of documents, the proposed method generates diverse candidates using the diverse beam search and selects one candidate that aligns with the expected aggregate property of inputs.
strength_and_weaknesses: Strengths:

- This paper tackles one of the important issues in multi-document summarization: the existing multi-document summarizers can synthesize inputs properly.
- This paper includes extensive analysis with statistical results on two data sets.

Weaknesses:

- The main criticism is that the diverse beam search (DBS) does not generate diverse candidates with various ``properties''. The DBS generates a set of candidates by penalizing the generation of strings similar to candidates on beams in other groups. In DBS, the similarity is only considered at the token level, not the property of interest level. DBS is not guaranteed to generate diverse candidates with various properties of interest.
- The proposed strategy 'generate-diverse-then-select' should be compared with the strategy 'generate-then-select.' This paper has no empirical studies to show the superiority of the 'generate-diverse-then-select.'
- There is no report of the ROUGE1 score on movie review experiments (Table 4). The main goal of the proposed strategy is to select a summarization that aligns with the expected property of inputs and the target summary. Only showing $R^2$, MSE, and Pearson's r results cannot prove the superiority of the proposed method because it selects the wrong summarization but aligns with the expected property.
- The proposed method can do a cautious summarization, but there is no evaluation of the cautionary of the summaries. Human evaluations or correlations of automatic factuality metrics should be included.
- There are two versions to predict sentiment on movie data set: BERT trained on the SST data set, and BERT trained on the IMDB data set. It is a wonder why this paper adopted different models.
- Some evaluation settings are missing, like the number of beams. Furthermore, selecting the true sentiment in Table 4 is unclear. Please add the details to the appendix.
- As mentioned in the introduction, the problem considered in this paper is only for some applications, such as movies or medical, not all applications for multi-document summarization, like news. The reviewer thinks the title is too generalized for what the paper considers.
clarity,_quality,_novelty_and_reproducibility: This paper contains originality because this paper is the first study to investigate synthesis in multi-document summarization.
summary_of_the_review: The paper shows insights about the synthesis of the existing multi-document summarizers. However, the experiments lack to prove the superiority of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the synthesizing capabilities of multi-document summarization models and presents a method for increasing those capabilities in summarization. 
Synthesizing capabilities are measured with respect to an application-dependent latent factor, the model is required to generate outputs that align with that factor in the input. Two cases have been studied: movie reviews with the latent factor of sentiment, and medical study results with the latent factor of significant effects for interventions. The investigated models proved to be lacking in synthesizing capabilities, but the proposed method raised these to human level (with regard to the metrics).
strength_and_weaknesses: Strengths:
[S1] The paper extends the conceptualization of desiderate for multi-document summaries to synthetization.
[S2] The paper is well written and methodological sound.
[S3] The proposed measures, given that factor information is available, could be used as evaluation metric for MDS tasks.

Weaknesses: 
[W1] The novelty of the method is limited. 
[W2] Some questions in the design and results of the experiment remain open. 
[W3] A stronger baseline could have been used. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. I especially liked the technical accuracy, being neither too complex nor sparse. The level of detail for the implementation is solid and generally seems reproducible. The authors don't provide the source code, which would greatly improve reproducibility (and speed up subsequent research).
The novelty is limited, the solution to address the weakness of the models is an application of existing methods.

Baseline/Related Work
- Shah et al. (2021a) solves a similar task, in a different application area. The authors need to state the technical relation to this paper in more detail. It seems to be a valid, stronger baseline than SoA summarization models not focusing on factors, and I am wondering why it hasn’t been used in the experiments (source code is available).
 
Experiments / Open Questions
- How the cases when the model abstained from a decision (Table 5) counted in the evaluation (e.g., Figure 7)?
- If the related work cited in footnote 2 is complementary, a remark on why that’s the case and why it doesn’t serve as a baseline would be interesting. Otherwise, this raises the question of whether the right baseline has been chosen.
summary_of_the_review: The paper addresses an important issues for multi-document summarization, and show empirical evidence that this issue persists in SoA summarization models. Conceptualisation and experiments are generally sound. It misses a detailed discussion of one highly relevant work, and it's inclusion in the comparison. The paper is generally reproducible from the given details, inclusion of source code would make more so.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to tackle the problem of multi-document summarization through synthesis evaluation. Do summarization models synthesize large and varied inputs? The paper studies this question from two angles: movie reviews, where the meta-review is human-written and scores are given (roughly equivalent to sentiment scores); and biomedical systematic reviews of treatments, where multiple systematic reviews summarize clinical trials, resulting in statistical meta-analysis results that can be used as scores. The authors make a good choice of pre-trained models to analyze and show that they do not synthesize a large body of input as well. Then they further analyze how the models fare when we switch the input ordering, or the balance of the input (by ablation). Finally, they propose their own post-hoc method consisting of generating a diverse set of output candidates, and then select the set that best corresponds to an expected synthesis result obtained using an aggregate score predicted by an external pre-trained model.
strength_and_weaknesses: Strengths:
- The paper tackles an important problem and proposes a simple post-hoc (inference time) method for improving synthesis in summarization models
- The metrics provided are automatic and rely on human-written summaries, making the evaluation strong/fair

Weaknesses:
- The ordering analysis in section 5.2 could result in different results if we augmented the training data by switching the order of the reviews for example.
- Likewise, the analysis in section 5.3 addresses something that the models were not trained for. The results could see a shift if there is data augmentation with the expected result as the label.
- The abstention percentage is between 30 and 40%, which is quite high and shows the need for a change in training rather than inference.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well organized. The paper can be reproduced based on the proposed method is somewhat novel and empirically supported. The quality of analysis could be improved.
summary_of_the_review: While the authors propose an empirically-supported method to improve synthesis in summarization models, their analysis is lacking as they have evaluated models against input ordering and re-balancing, which the models could be trained for but were not. The result is a good contribution, but the analysis suggests that a training-time method can improve the scores further. The high rate of abstention is another flaw that hints at a required change in training rather than inference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates whether multi-document summarization models can effectively synthesize conflicting or diverse information from multiple documents, and further introduces a new technique for improving model synthesis.  The strengths lie in the unique way to evaluate summarization models, the focus of an important aspect of multi-document summarization, with solid experiments conducted in very different domains. The main criticisms include the baseline choices, evaluation details, and the approach not being able to generate diverse candidates, as well as the vague generalization of this method on other general multi-document summarization. Taking into account these major concerns and after reviewing the authors’ responses, I’d encourage the authors to further improve their work for a stronger future submission.
Decision: Reject