Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses two common problems in pseudo-label (or oracle extract) generation for extractive summarization: 1) greedy or beam search does not always lead to a globally optimal solution, and 2) a pseudo-label is a deterministic training target that lacks calibration. 
The authors then propose to leverage a series of pseudo-summaries generated from beam search to generate sentence-wise soft labels used for extractive summarization training.

strength_and_weaknesses: **Strengths**
- The identified problem is significant in extractive summarization, and the motivation is clear.
- The idea of using soft labels for sentence-level labels is novel.
- Extensive experiments and thorough analyses demonstrate the superiority of the proposed method in many summarization scenarios across domains and languages in supervised and zero-shot settings, with 
- A theoretical proof is provided, showing that maximizing oracle expectation is equivalent to the proposed non-deterministic objective for extractive summarization.

**Weaknesses**
- The performance improvements look marginal in some settings, and more sophisticated baselines can be considered.

clarity,_quality,_novelty_and_reproducibility: The paper is well organized and presented. Though soft labels are widely used in modern deep-learning models, incorporating them in a sentence labeling process is a novel design. The authors also provide a theoretical explanation of the learning objective involving soft labels. In addition to experimenting with the labeling method in various text summarization scenarios, the analyses on the connection between the upper bound of different oracles and the actual performance (as well as the proposed attainable summary knowledge) are inspirational. The authors release their code for reproducibility.

A minor concern is that performance improvements over baselines or deterministic beam/greedy oracle seem marginal in some cases. It's weird to mark the results of OREO in Table 4 in bold since MatchSum is the most performance model. The authors claim that MatchSum has an extra reranking process, and OREO can potentially serve as a foundation for those more complex methods. I would like to know whether OREO can further enhance performance over MatchSum. Meanwhile,  the idea of using reinforcement learning over evaluation metrics or exploiting supervision signals from multiple target summaries can be widely found in abstract summarization efforts (e.g., [1][2]), and some of them may not be hard to adapt to extractive summarization. Comparing these works can make the evaluation more solid. 

I'm also curious why the evaluation metric considers ROUGE-1 and ROUGE-2 but not ROUGE-L. Will the experimental conclusions still hold if using the mean of ROUGE-1, 2, and L?

[1] Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization. EMNLP 2019

[2] BRIO: Bringing Order to Abstractive Summarization. ACL 2022

summary_of_the_review: This paper presents a sentence labeling algorithm creating soft labels for extractive text summarization, with clear motivation, novel technical design, extensive evaluation,  and theoretical analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel way of extracting silver-standard extractive summaries (Oracle) from gold-standard abstractive summaries which they call “OREO”. Earlier methods mostly relied on the Greedy labelling method, which is sub-optimal and deterministic. The authors propose to instead generate the final oracle summaries by soft labelling so that multiple possible extractive oracles (obtained by beam-search) are considered according to their probability distribution; which leads to the final oracle summary being non-deterministic. 
The authors have then performed several experiments to show the robustness of their model in different settings - supervised, zero-shot cross-domain, zero-shot cross-lingual, supervised abstractive, etc. The authors have experimented on several datasets, using the BERTSum model (and GSum model for abstractive setting). They show that models trained on OREO perform better than greedy and simple-beam search labels in most cases and generalize better.
strength_and_weaknesses: Strengths
The problem statement is very interesting.
Proposed method is simple to implement and doesn’t require architectural changes for existing models
Extensive set of experiments on different settings and on multiple datasets performed, showing the method leads to better-performing models
Weaknesses and suggestions
Only the BERTSum model was used for comparison between labelling strategies. Performing similar experiments with some other models would be better, and these models need not be very complex (even some older architectures such as SummaRuNNer [Nallapati et al., 2017] may be used). 
Another simple labelling strategy that may be a baseline may be to just get the top-K ROUGE sentences from the document individually for every sentence in the original reference summary. [Cheng & Lapata (2016)]
In Zero shot multi-lingual setting XLM-base has been used when the given SOTA uses XLM-Large, but then the authors claim the SOTA performs better because of the more complex architecture.
Some equations are a bit difficult to understand, and it will be good if the terms are defined properly and described in simpler text. 
The purpose of Section 5.5 is not very clear, and the results do not seem too interesting either. It is also not clear how exactly the ASK metric measures the extent to which a model can attain summary relevant knowledge. This space may be better used for discussing experiments using other models.
In many of the results, only a very small improvement in ROUGE scores have been achieved over other labelling strategies (which is kind of expected with ROUGE scores). It is also known that ROUGE scores are probably not a very good way of measuring quality of summaries. Thus it is not very evident if the OREO strategy actually leads to better summaries. It would perhaps be better if some survey analyses were performed with the outputs from models trained on different labelling strategies.

Other Comments:
There is some discrepancy with the caption and data in Table 1, it seems the authors talk about sentences 1 & 4 (not 2 & 4) but then the OREO does not provide very high score (relatively) to sentence 1.
In line 8 of Algorithm 1, the l’ on the right hand side will probably be x.

clarity,_quality,_novelty_and_reproducibility: Clarity - The paper is mostly clear, except that some equations are a bit hard to understand
Quality - Quality needs improvement in terms of more experiments with different models
Novelty - Somewhat novel
Reproducibility - The work is reproducible and the codes are available.

summary_of_the_review: While the proposed method is simple to implement and is shown to be robust by several experiments, my main problem is that only BERTSum model was used to evaluate the different labelling strategies. Sections 5.5 and 5.6 are also questionable. Even though the method performs better than others, in most cases the increase in performance is quite small.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel labeling scheme for extractive text summarization. Extractive summarization datasets are most often derived from abstractive datasets using a greedy labeling scheme, providing a single extractive reference per document. Greedy multi-hot labels cause sparsity/under-fitting issues so the authors propose a soft-labeling scheme and show it is equivalent to incorporating multiple hypotheses in a non-autoregressive labeling model. The soft-labeling oracle is implemented using beam search. The authors then show improvements over greedy and beam labeling in several settings, although the model itself is not state-of-the-art.
strength_and_weaknesses: Overall, this paper is well written and well structured. Although the contribution is fairly simple, it is well theorized. Extensive experiments are also provided.

This work is likely to have little practical use given the additional complexity of the proposed scheme and its fairly limited experimental improvements. But it does provide interesting insights and may trigger further research.

Suggestions/Comments:
- can improvements that are statistically significant be added to the result tables? Many improvements seem limited; are they statistically significant?
- since the method does not yield state-of-the-art results, it would be nice to better explain summary reranking and MatchSum (e.g., how much more complicated/computationally expensive these are).
- could MatchSum be added as a baseline for the cross-domain experiments?
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is clear and well written.

Novelty: the work is novel to the best of my knowledge.

Quality: both theoretical and empirical aspects of the paper are good.

Reproducibility: the code and models are open-sourced (I could not check given broken anonymized link).


summary_of_the_review: The work is novel to the best of my knowledge but its significance is likely to be limited given:
- recent improvements to abstractive summarization 
- its complexity over labeling schemes that are simpler to implement
- it is not state-of-the-art

That said, the work is overall solid and well theorized.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors address two observed problems in generating training sets for extractive summarization: (1) greedy/beam search in sentence selection frequently doesn't lead to an 'optimal' summary (in a structured prediction sense) and (2) greedy/beam search leads to a deterministic output that doesn't have reliable confidence calibration. To solve this, they build on recent work for soft-labels in deep learning to generate sentence-wise soft labels that can generate multiple reference summaries incorporated into extractive summarization training; referred to as ORacle ExpectatiOn (OREO) labeling. Experiments are conducted on multiple widely-used benchmark summarization datasets against multiple illustrative and competitive baselines in 'standard' supervised extractive, zero-shot cross-domain, zero-shot cross-lingual, and supervised abstraction settings -- demonstrating solid performance (often SotA with improved generalization properties). 

The consensus strengths of this work include:
- They identify a systemic problem in extractive summarization and build on recent work regarding training on soft-labels in deep learning settings to develop an application-suitable soft-labeling scheme. The work is well-motivated and the approach is interesting and effective.
- The paper is well-written and well-contextualized wrt related work both conceptually and empirically.
- Both analytical and empirical results are provided to support the primary claims of the paper.
- The proposed approach ostensibly applies to any future extractive learning algorithm (that can support multi-reference) and displays promising results for abstractive cases (under specific approaches).
- The authors were able to address issues raised during rebuttal effectively; hopefully these will be integrated into the final version of the paper effectively.

The weaknesses discussed in the reviews (and my own reading) of this work include:
- Extractive summarization isn't as widely studied and specific use cases are likely to be more esoteric in the future (under current summarization use cases). Thus, the practical impact may be limited.
- The connections to offline RL are fairly strong; thus, some of the findings could be more closely related from a conceptual and analytical cases. Making these connections more rigorously would strengthen the paper.
- Soft-labeling is common in deep learning; the contributions are fairly specific to extractive summarization.

Overall, the consensus is that this is a well-executed and potentially important contribution to extractive text summarization. The proposed method (OREO) will likely improve most existing summarization techniques in terms of performance and robustness -- and may apply to other text generation problems with additional work. The primary concern is that this work centers around extractive summarization, which is less studied due to LLMs, but the strength of the contributions outweigh this limitation as it is still a widely used setting.

justification_for_why_not_higher_score: The contribution is focusing on extractive text summarization, which is currently "out of fashion" and it isn't clear will ever again have the prominence  of abstractive summarization work -- thus, despite being a well-motivated and well-executed paper, the impact and interest may be limited by the specific application.
justification_for_why_not_lower_score: The paper is of high technical quality; well-motivated, well-written, simple, and effective. While the contribution focuses on extractive summarization, it also may impact all extractive summarization methods and may inspire work on validation methods, etc. in other text generation settings.
Decision: Accept: poster