Data Selection Curriculum for Abstractive Text Summarization
Authors: Shichao Sun
Ruifeng Yuan
Jianfei He
Ziqiang Cao
Wenjie Li
Xiaohua Jia
Venue: EMNLP 2023 Findings (2023)
Keywords: Data Selection
Curriculum Learning
Abstractive Text Summarization
TL;DR: Select and sort data for training a better abstractive text summarization model.

Abstract: Abstractive Text Summarization (ATS) models are commonly trained using large-scale data that is randomly shuffled. However, the impact of data selection and data ordering on ATS models remains a relatively unexplored research area, where a significant challenge lies in accurately assessing the learning difficulty of each training instance. This study introduces a Data Selection Curriculum (DSC) scoring system that incorporates both the difficulty of improving ATS model via an instance and the expected performance on this instance. By selectively excluding excessively simple and overly complex instances, the training efficiency can be optimized. Furthermore, curriculum learning is integrated to accelerate convergence and improve performance by gradually increasing the learning difficulty, inspired by human learners. Experimental results on the CNN/DailyMail dataset demonstrate that our approach surpasses potent baselines, utilizing a mere 20\% of the available instances.

Data Selection Curriculum for Abstractive Text Summarization
Shichao Sun', Ruifeng Yuan', Jianfei Jianfeii He², Ziqiang Cao', Wenjie Li', Xiaohua Jia" 'The Hong Kong Polytechnic University, Citty University of Hong Kong, 'Soochow University [csssun, csryuan] ccomp polyu. edu hk, jianfeihe2 c00m cityu. edu. hk zqcaoθsuda edu. cn, cswjli comp polyu. edu hk, cs jiaecityy edu. hk
Abstract
This study aims explore the potential for opti- mizing ATS models through the strategic utilization of data selection and curriculum learning (Bengio et al., 2009) in conjunction with contrastive learn- ing techniques. It is important to note that not all data is equally valuable, and the presence of re- dundant even detrimental examples can impede the performance of ATS systems (Mohiuddin al., 2022). Furthermore, the ordering in which data is presented during model training, as emphasized by curriculum learning principles, can have a signifi- cant impact on both the efficiency and effectiveness of the learning process. Consequently, there is a fundamental necessity to develop a scoring system that can accurately assess the learning difficulty of individual samples and furtherly can be used for data selection and curriculum learning.
Abstractive Text Summarization (ATS) models are commonly trained using large-scale data that is randomly shuffled. However, the im- pact of data selection and data ordering on ATS models remains a relatively unexplored research area, where significant challenge lies in accurately assessing the learning dif- ficulty of each training instance. This study introduces a Data Selection Curriculum (DSC) scoring system that incorporates both the diffi- culty improving ATS model via an instance and the expected performance on this instance. By selectively excluding excessively simple and overly complex instances, the training effi- ciency can be optimized. Furthermore, curricu- learning is integrated to accelerate conver- gence and improve performance by gradually increasing the learning difficulty, inspired by human learners. Experimental results on the CNN/DailyMain dataset demonstrate that our approach surpasses potent baselines, utilizing a mere 20% of the available instances.
In this study, we introduce novel scoring sys- tem called the Data Selection Curriculum (DSC) score, which serves measure of learning dif- ficulty. The DSC score considers factors: the difficulty of enhancing the ATS model through a specific instance and the expected per- formance the model on that particular instance. Instances on which the current model already ex- hibits promising performance and stands to benefit significantly from slight adjustments are assigned lower learning difficulty. This is because these instances have the potential to contribute more the final performance. As for the first factor, we propose a novel Margin-aware List-wise Ranking (MLR) loss, which introduces a dynamic margin instead of constant margin, which the state-of-the- art BRIO (Liu et al., 2022) uses in the list-wise ranking loss for contrastive learning. It can bet- ter reflect the fine-grained misalignment between sequential likelihood and performance. Through the DSC score, we can effectively select and priori- tize instances during the training process, focusing on those that offer the greatest opportunitics for enhancing the ATS model's 'ss overall performance.
essential
1 Introduction
Abstractive Text Summarization (ATS) aims to gen- erate concise summaries while preserving essential content. Recent studies (Liu et al., 2023b; Zhang et al., 2023; Liu et al., 2023a) have shown that Large Language Models (LLMs), including GPT- 3.5 (Ouyang et al., 2022), can produce summaries more favored by human annotators compared ref- erence summaries from well-established datasets, such as CNN//ailyMail (Hermann et al., 2015). Additionally, Liu et al. (2023a) experimentally as- sert that contrastive learning based methods ap- plied to smaller summarization models, like BART (Lewis et al., 2020), can deliver performance on par with LLMs. This finding renews the importance of training smaller models via contrastive learning, as it offers the advantage of reducing computational costs. However, ATS models are commonly trained using large-scale data that randomly shuffled.

---

We empirically validated our method on the CNN/DailyMail dataset. The results suggest that our method surpasses several robust baselines us- ing fewer than 20% of instances. Additionally, experimental findings highlight that our MLR loss is superior to the BRIO loss.
pre-defined metric score when <. j. X is the mar- gin between two adjacent candidates. The score s(Y") considers normalizing the candidate's length by a penalty hyperparameter a. Po denotes the fine-tuned language model with parameter 6, like BART. y is the weight of the calibration loss, which prevents models from deviating significantly from their fine-tuned cross-entropy objective.
2 Method
2.1 Preliminary
2.2 MLR Loss
As evidenced in recent research, contrastive learn- ing has the capacity to achieve superior perfor- mance in text summarization (Liu et al., 2022; Zhang et al., 2022; Zhao et al., 2023b; Liu al., 2023a; Zhao et al., 2023a). These methodologies adopt novel three-stage training protocol, com- prising pre-training, fine-tuning, and calibrating stages. The calibrating stage employs diverse beam search (Vijayakumar et al., 201 8) to generate a va- riety of candidates, which are subsequently ranked based on their metric scores. A ranking loss tunes the parameters, aligning the sequence likelihood with the established order. List-wise rank loss Leel from BRIO (Liu al., 2022), along with cross- entropy loss farent, has been shown yield promis- ing results (Liu et al., 2022: Zhang et al., 2022: Zhao et al., 2023b; Liu et al., 2023a). The overall loss L is delineated as follows:
The calibration loss Leat previously discussed uses constant margin, denoted as X, for every pair of adjacent candidates. The utilization of a constant margin may be counter-intuitive since this method may demonstrate limitations certain scenarios, such as cases where different pairs of adjacent can- didates exhibit varying differences in metric scores. To rectify this, we propose a novel Margin-aware List-wise Ranking (MLR) loss that can serve more flexible replacement for Coat. The MLR loss defined follows:
Limr = ΣΣ
man(0, s(Y)) s(V:))
15J
Tij = B3x (m(Yy)) - m(Y)))
(2)
where m(+) denotes the metric function, which could be BERTScore (Zhang al., 2020) or ROUGE (Lin, 2004), and 3 is a hyperparameter responsible for scaling the difference between two candidates. Please note that the value of m(*) is within the range of 0 to 1, with higher m(+) score indicating superior performance. Consequently, this setup makes the calibration loss aware of the margin in metric scores, enabling more precise cal- ibration training. In essence, the fyui loss can better define the difficulty faced by the model transitioning to a more enhanced version pa- rameters, unlike the Leat. which fails to facilitate fine-grained changes due to its constant margin.
L = Lzent + 7hoot
Y
Lzent =
log pe (vel XX, y0 ... ,3t-1)
YYI
Loal =
max(O, Aaj + s(Y,) s(Y)))
i<j
Xij = X x i)
s(Y') - El'llog log po(vil X, Vo. ... 1
where X, Y' respectively denotes the input doc- ument, the ground truth summary, and candidate summaries from diverse beam search. The (candi- date or ground truth) summary Y consists of YY] tokens, i.e., Yo. "",".. U/yl, and |Y| is the length of the summary Y. yo is the pre-defined start token. Crent is the mean of the negative log-likelihood of all ground truth summary tokens. Yj and Y are two different candidate summaries and i, are their rank index in the ranked list. means that the quality of y/ is better than Yj in terms of the
2.3 Difficulty Metric
The difficulty metric, represented as D, incorpo- rates both the MLR loss and expected model per- formance. It is defined as follows:
exp(s(Y;))
wi Σ, exp((s(Y)))
Σ m(Y:)
(3)
D 1 - E Lmur

---

where the expected model performance & is defined the expected metric scores of all candidates that the model generates by diverse beam search.
In this study, we conducted an analysis of the training instances using the difficulty metric dis- cussed earlier (in Section 2.3). The results of this analysis are presented in Figure We observed that the distribution of the difficulty scores closely approximates Gaussian Distribution. The obser- vation provides strong support for our hypothesis in Section 2.4. This verification indicates that our DSC method can be effective in identifying and prioritizing training instances with the mean and standard deviation.
2.4 Data Selection Curriculum
We rank training data based on the difficulty met- ric discussed above. Inspired by the success of Mohiuddin et al. (2022), we discard data in- stances that are too easy or too hard to learn. As- suming a Gaussian Distribution of the training data, the data selection window can be defined as (u(D)-o((D) (D), ()))),(()))))) ((o)+ where ((D), o (D) represent the mean and standard deviation respec- tively, and is hyperparameter that is used to adjust the window size of data selection.
30333
<3333
80333
3 Experiment
20003
20003
3.1 Implementation Details
30000
20003
We implement method top of the BRIO open-source codebase and adhere to most of its experimental settings as defined by Liu al. (2022). Differences in settings and the reasons be- hind these will be discussed in this section. As our approach is model-agnostic, we initialize it us- ing the BRIO checkpoint to ensure quick conver- gence. To better understand the data effect, we use constant learning rate of le-6 instead of a learning schedule, thus reducing the optimizer's impact. Liu et al. (2022) found that the use of ROUGE scores in the metric function m(*) (referenced in Equations (2) and (3)) resulted in better perfor- mance than BERTScore. Therefore, we follow this approach, using the mean value of ROUGE- 1, ROUGE-2, and ROUGE-L as m(*) to ensure range from 0 to 1, the same as implemented in BRIO. 3 is set 0.1 for all experiments.
10033
1,26 dificc metic
Figure I: Difficulty metric histogram.
3.3 Baseline
For comparison, we choose several impressive models baselines. BART (Lewis al., 2020), large pre-trained Seq2Seq model, is fine-tuned on CNN/DailyMaii.. BRIO (Liu et al., 2022) is a promising model based on contrastive learning. which is initialized by BART. BRIO-Loop an iterative version of BRIO, trained on the candi- dates generated by BRIO. MLR substitutes for the calibration loss of BRIO-Loop. Finally, DSC represents proposed method which uses data selection curriculum along with the MLR loss.
3.2
Data
4
Results
We conduct experiments the CNNi dataset (Hermann al., 2015), large-scale news dataset with 287K/13K/1 IK instances for train- ing/validattionttesinng respectively. As per previous work (Nallapati al., 2016; See et al., 2017; 7; Liu et al., 2022), we use the news articles (averaging 791.6 words) as the source documents and the asso- ciated highlights (averaging 3.75 sentences or 55.6 words) as summaries.
4.1 Overall Evaluation
Table 1 shows the evaluation results of various methods on CNll Notably, our pro- posed DSC method outperforms the strong baseline models in terms of ROUGE scores and BERTScore. This improvement demonstrates the effectiveness of the DSC method in enhancing the abstractive text summarization task. Moreover, MLR achieves higher ROUGE scores and BERTScore than BRIO- Loop. This finding underscores the effectiveness of dynamic margin calibration in sequential likelihood calibration, as opposed to a constant margin.
'https://github http:/hwww...cmmiiiiii////////te hhttss://buggigggaat Beee/Vale--Lie brio-cnndn--uncased Phttos://cs.ng: /yy. edt.ee/uuuuuueuueeeeemooe

---

Method
R-1
R-2
R-L
BS
degradation in performance. Moreover, we ob- served that the training process tends to favor casier instances, as optimal performance was achieved with less than 20% of the available instances.
BART*
44.29
21.17
41.09
27.38
BRIO
47.78
23.55
44.57
32.11
BRIO-Loop"
48.01
23.80
44.67
MLR
48.17t
23.95t
44.92t
35.25t
DSC
48.62
24.14
45.31
35.90
35.75
Table I: Overall evaluation results. significantly
35.50
better than the baseline model (p < 0.01). results reported in the original papers. R-1/2/L ROUGE- 1/2/L F2 scores. BS denotes BERTScore.
BERTSCORe
35,00
34.75
4.2 Coefficients of the Data Selection
DSC
The hyperparameter d can control the size of the data selection window. A larger å results in larger window for selected data, including more easy and hard instances for training. We explore the impact different hyperparameter 5 values on model per- formance. Our experiments, reported Table 2, reveal that including more data is not always ben- eficial. Performance declines with larger 8 values. Conversely, too few instances also lead to poor performance when is small.
10%
20%
40%
50%
Figure 2: DSC performance with different data ratios.
We validate our methods on XSum (Narayan et al., 2018) from the same start point of BRIO. We initialize our model using "google/pegasus- xsum"* "4 and use the selected 20% instances to train. The BERTScore of our model (52.88) beats BRIO (41.08) by large margin, so the summary from our model is more similar to the reference than BRIO.
8
R-1
R-2
R-L.
BS
4.4
Ablation Study
0.50
48.26
23.92
45.00
35,42
0.75
48.44
24.09
45.15
35.73
1.00
48.46
24.11
45.22
35.75
Method
R-1
R-2
R-L
BS
1.25
48.62
24.14
45.31
35.90
BRIO
47.78
23.55
44.57
32.11
1,50
48.59
24.14
45.24
35.71
MLR
48.17
23.95
44.921
35.25t
1.75
48.51
24.13
45.24
35.64
+ MLR DS
48.36
24.06
45.04
35.54
2.00
48.41
24.12
45.22
35.59
+ MLR DS CL
48.62
24.14
45.31
35.90t
Table 2: Model performance with different å coeffi- cients. R-1/2/L are ROUGE-1/2/L F, scores. BS de- notes BERTScore.
Table 3: Ablation Study. 1; significantly better than the
baseline model (p < 0.01). ": results reported in the original papers. R-1/2/L are ROUGE-1/241 F, scores. BS denotes BERTScore.
4.3 Selected Data Ratio
We also conducted separate experiments to eval- each sub-model. MIR represents our novel contrastive ranking loss. DS shorthand for data selection, which refers to training the model using selected data instead all instances. CL denotes curriculum learning, which trains the model in the from-easy-to-hee ordering. Table 3 shows that all methods we proposed contribute the improve- ment of summarization performance.
To investigate the impact of the number of selected instances, we conducted experiments using varying ratios of selected data from the dataset. The model was trained on the selected data for one epoch, and without early stopping, the latest checkpoint was used for testing. Alternatively, with early stopping, the checkpoint with the highest ROUGE scores was selected. The results, depicted in Figure 2, indicate that the quantity of data does not necessarily cor- relate with improved performance. Surprisingly, an excessive number of instances may even lead to
Btttps:/hhugee i haccace.cmm core.co//ooo ee/ begasus-xxee Xmum

---

5 Conclusion
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke 7ettlemoyer. 2020. BART: Denoising sequencc-to-ssqquenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computaaional Länguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
This paper investigates the impact of data on text summarization. We propose learning difficulty metric for data selection and curriculum learning. This score incorporates a Margin-aware List-wise Ranking (MLR) loss, further enhancing contrastive learning due to its dynamic margin. Experimental results indicate that our data selection curriculum can outperform strong baselines using fewer than 20% instunces from the CNN/DailyMail dataset.
Chin-Yer Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Ouf, pages 74-81, 1, Barcelona, Spain, Association for Computational Linguisties.
6 Limitations
Yixin Liu, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023a. On learning to summarize with large language models references. arXiv preprint arXiv:2305. 14239..
We acknowledge that the conclusions presented in the paper are derived from analyzing single dataset specifically focused on English news arti- cles. While this dataset is widely accepted as a benchmark for summarization tasks, it is important to recognize that the results obtained may not be applicable to other languages or domains. There- fore, caution should be exercised when generaliz- ing these findings beyond the scope of the dataset. Despite the limitation, the findings of this study are valuable in terms of providing insights into the potential enhancements that can be achieved abstractive summarization by employing strate- gic data selection and curriculum learning tech- niques. These insights contribute to the ongoing exploration of improving summarization methods.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. 2023b. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Grabam Neubig. 2022, BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Tasnim Mohiuddin, Philipp Koehn, Vishrav Chaudhary, James Cross, Shruti Bhosale, and Shafiq Joty. 2022. Data selection curriculum for neural machine trans- lation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 1569-1582, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
A Acknowledgements
The work described in this paper was supported by Research Grants Council of Hong Kong (PolyU/15003617 and Po111/5210919), National Natural Science Foundation of China (61672445, 62076212, 62106165).
Ramesh Nallapati. Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequeeee RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
References
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Confer- ence Machine Learning, ICML '09, page 41 48, New York, NY, USA. Association for Computing Machinery.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th Interna- rional Conference on Neural Information Processing Systems - Volume 1, NIPS' 15, page 1693-1701, Cam- bridge, MA, USA. MIT Press.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,

---

Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Proceedings of the 55th An-
Ashwin Vijayakumar, Michael Cogswell, Ramprasatth Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse Beam Search for Improved Description of Complex Scenes. Proceed- ings of the AAAI Conference on Artificial Intelligence. 32(1).
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.138
Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. 2022. Momentum calibration for text generation. arXiv preprint urXiv:2212.04257.
Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. 2023a. Slic-hf: Sequence likelihood calibration with human feed- back. arXiv preprint 2rxiv:2355.1042 10425.
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter Liu. 2023b. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations.


Reviews:
[{"review_id": "jYXZmUn7Gf", "reviewer": "Reviewer_HkHS", "summary": "", "questions": "Could you report the results of the using different percentages of the training data against ROUGE L?\nDid you try to repeat these experiments on XSum? Were there issues that caused not reporting the results?\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our submission and for your valuable feedback on our work!\n\n\n\nWe try to repeat our experiments on XSum from the same start point of BRIO. We initialize our model using \"google/pegasus-xsum\" using the selected 20% instances to train. The results are in the following table. The performance of ROUGE-2 and ROUGE-L are competitive. Besides, the BERTScore of our model beats BRIO, so the summary from our model is more similar to the reference than the summary from BRIO in semantics.\n\n|      | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore |\n| ---- | ------- | ------- | ------- | --------- |\n| BRIO | 49.07   | 25.59   | 40.40   | 41.08     |\n| DSC  | 47.95   | 25.00   | 39.81   | **52.88** |\n\n\n\n\n\nThe conclusion is consistent that 20% of data is needed to achieve the highest performance. Here is the result of ROUGE-L.\n\n|                    | 10%   | 20%   | 30%   | 40%   | 50%   |\n| ------------------ | ----- | ----- | ----- | ----- | ----- |\n| DSC w/o early stop | 45.02 | 45.09 | 45.0  | 44.71 | 44.69 |\n| DSC                | 45.28 | 45.31 | 45.31 | 45.31 | 45.31 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper touches on the interesting issue of data selection by measuring  the difficulty of each input instance. They show that this technique improves summarization performance. They authors further show that only by training using 20% of the training data the model achieves its highest performance in terms of BertScore. The paper does summarize the recent literature on abstractive summarization in an insightful way. ", "reasons_to_reject": "-The proposed method has been tested on a single dataset, CNN/DM. Therefore, the claims on the impact of data selection may not generalize to other datasets for Abstractive Summarization.\n-The analysis that shows only 20% of data is needed to achieve the highest performance is only based on BertScore, however, it is very important to validate this claim also in terms of ROUGE metrics. This is due to the fact that BertScore in principle does assign high scores to semantically related sentences even if they are not exactly expressing the same stances. \n-The combination of the two above issues make the claims of the paper much weaker.\n-They use only a checkpoint of BRIO (which is BART-based) to initialize the model, however, if they had used also a BART checkpoint and showed similar performance it would have been helpful.\n\n ", "questions_to_authors": "Could you report the results of the using different percentages of the training data against ROUGE L?\nDid you try to repeat these experiments on XSum? Were there issues that caused not reporting the results?\n\n", "paper_topic_and_main_contributions": "This paper studies the effect of data on text summarization in the sense of order and instance selection. They propose a learning difficulty metric for data selection and curriculum learning. This score incorporates a Margin-aware List-wise Ranking loss based on the notion of contrastive learning.\n\nThe authors show through experiments on the CNN/DM dataset that these engineering techniques improve summarization performance in terms ROUGE metrics as well as BertScore.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aiQsfUnlfv", "reviewer": "Reviewer_qyJh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your response.\n\n\n\nWe try to repeat our experiments on XSum from the same start point of BRIO. We initialize our model using \"google/pegasus-xsum\" using the selected 20% instances to train. The results are in the following table. The performance of ROUGE-2 and ROUGE-L are competitive. Besides, the BERTScore of our model beats BRIO, so the summary from our model is more similar to the reference than the summary from BRIO in semantics.\n\n|      | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore |\n| ---- | ------- | ------- | ------- | --------- |\n| BRIO | 49.07   | 25.59   | 40.40   | 41.08     |\n| DSC  | 47.95   | 25.00   | 39.81   | **52.88** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. simple but efficient methods are provided.\n2. could significantly reduce the amount of data required to train the Abstractive Text Summarization (ATS) models, and have better model performance.", "reasons_to_reject": "Needs more experiments on other datasets to show whether the method is general.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In order to explore the impact of data selection and ordering in abstractive text summarization models, this paper firstly provides a Margin-aware List-wise Ranking(MLR) loss, which introduces a dynamic margin to improve the original loss from the BRIO. Meanwhile, get the difficulty metric mainly based on the MLR loss. Then, based on the difficulty metric value of each instance, simple and complex instances are discarded, and curriculum learning is used to training models. Experiments on the CNN/DailyMail dataset showed promising results for those improvements.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ByFGHk7byB", "reviewer": "Reviewer_r3Wo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed comments.\n\n**Weakness 1 & 2:**\n\n\nWe try to repeat our experiments on XSum from the same start point of BRIO. We initialize our model using \"google/pegasus-xsum\" using the selected 20% instances to train. The results are in the following table. The performance of ROUGE-2 and ROUGE-L are competitive. Besides, the BERTScore of our model beats BRIO, so the summary from our model is more similar to the reference than the summary from BRIO in semantics.\n\n|      | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore |\n| ---- | ------- | ------- | ------- | --------- |\n| BRIO | 49.07   | 25.59   | 40.40   | 41.08     |\n| DSC  | 47.95   | 25.00   | 39.81   | **52.88** |\n\n\n\n**Weakness 3:**\n\nOur motivation is to select the most favorable part of the model data for training. We hope the selected data is the most favorable. Since we also consider the model to calculate the difficulty score, it is more possible to select the suitable data for the current model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper is easy to read. Although the length of this paper is small, it provides the necessary background and explanation.\n2.\tBased on BRIO model, the authors change the margin between two adjacent candidates of contrastive loss from a fixed distance to a dynamic metric-based calculation.\n", "reasons_to_reject": "1.\tThe author has only conducted experiments on CNN/DailyMail dataset and obtained good results. Can it be generalized to other datasets?  \n2.\tThe overall workload is less. The authors should provide the result on the XSum dataset or other datasets. This needs to be verified.\n3.\tOn the CNN/DailyMail dataset, the authors utilized a data selection curriculum that can outperform baselines using 20% instances on the CNN/DailyMail dataset. Is it possible that this improvement is caused by selecting the most favorable part of the model data for training?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors investigate the impact of data on text summarization; they proposed a learning difficulty metric for data selection and curriculum learning. Due to its dynamic margin, this score incorporates a margin-aware list-wise ranking loss to enhance contrastive learning. By selectively excluding straightforward and overly complex instances, the CNN/DailyMail dataset get better results. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pXXgloqoR4", "reviewer": "Reviewer_pCuk", "summary": "", "questions": "Q1: β is a key parameter. Are there any experiments that suggest it should take a value of 0.1?\n\nQ2: In Figure 1, the authors show the difficulty metric of all training data,  I would like to know how the metric is calculated. And what are the parameters \\theta of the model?\n\nQ3: CL trains the models in order from easiest to hardest. I know the process of sorting the data, but not how they add the data to the training set.\n\nQ4: In Eq(3), the second sub-equation, I wonder what is the sum operation.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for the positive assessment and highly detailed and constructive feedback on our paper.\n\n**Weakness:** Explanation why the difficulty measure is designed as mentioned in section 2.3.\n\nWe design a difficulty measure metric for an instance by considering the expected performance ($\\mathcal{E}$) of the current model in this instance and the effort of transitioning the current model to a stronger version. \n\nIf the current model can obtain the promising performance for this instance, it means that the current model ''knows'' this instance well and the current model can easily learn this instance. We consider the generation process as sampling so we can obtain some sampled summaries $Y$ and their probability-based scores $s(Y)$.  We can define the expected performance as the weighted sum of the summary metric score $m(Y)$. The weight is the softmax scores of probability-based scores $s(Y)$. A larger expected score ($\\mathcal{E}$) means easier to learn, so we use   (1-$\\mathcal{E}$) to represent the difficulty. The score  (1-$\\mathcal{E}$) is large so the instance is hard to learn. \n\nAs mentioned in section 2.2, mlr loss can reflect the effort that the current model needs to take for achieving stronger performance. \n\nTherefore, it is intuitive to define this difficulty measure score as the combination of the above two scores.\n\n\n\n**Q1:** β is a key parameter to balance the order of magnitude between the model-predicted scores and the reference-based scores (like ROUGE or BERTScore). If this value is too small, it means that the margin between different summaries is similar (very small, like 1e-6 and 2e-6), and this method may take a slight effect. We obtain this value 0.1 on CNNDM by observing some model-predicted scores and their ROUGE scores. To further validate the effect of this parameter, we conduct experiments on XSum. We found the β is set to 1, we can obtain the best ROUGE performance 49.27/25.75/40.40 against 49.01/25.41/40.18 when β is set to 0.01. \n\n\n\n**Q2:** The difficulty score in Figure 1 is calculated as mentioned in section 2.3. And the model parameter is our initialized model ''Yale-LILY/brio-cnndm-uncased'' from huggingface.\n\n\n\n**Q3:** We calculate the difficulty scores as mentioned in section 2.3, and then we use this score to sort all training instances. Please note that the data distribution closely approximates a Gaussian Distribution as shown in Fig 1, so we can use the mean value and standard deviation to find suitable data. We test different $\\delta$ values in Table 2 and select the best one.\n\n\n\n**Q4:** As mentioned in the response of weakness, the second sub-equation is defined as the expected value of the sampling summaries score (ROUGE or BERTScore) with their probability-based score. After softmax, the weight can be viewed as the probability of generating each summary from the model. So, this weighted sum can be viewed as the expected summaries score. It can reflect the average performance of the current model with sampling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. See the above part.\n\n2. The motivation is clearly described. They combine the loss of model to evaluate the difficulty of the instance, it's reasonable.\n\n3. Experimental results demonstrate the method proposed in this paper is effective.\n\nIt is well-written and easy to follow.", "reasons_to_reject": "1. CL strategy proposed in this paper is model-agnostic, but it is designed for contrastive learning models, not for the Abstractive Text Summarization task.\n\n2. The design of their difficulty measure is not clearly explained, i.e., why the formula in section 2.3 is designed the way it is; the difficulty is a very important part of curriculum learning, and readers would be more likely to understand it if there were a more out explanation of it.\n\n3. There are some details that are not clearly described, see Questions For The Authors.", "questions_to_authors": "Q1: β is a key parameter. Are there any experiments that suggest it should take a value of 0.1?\n\nQ2: In Figure 1, the authors show the difficulty metric of all training data,  I would like to know how the metric is calculated. And what are the parameters \\theta of the model?\n\nQ3: CL trains the models in order from easiest to hardest. I know the process of sorting the data, but not how they add the data to the training set.\n\nQ4: In Eq(3), the second sub-equation, I wonder what is the sum operation.\n", "paper_topic_and_main_contributions": "In this paper, they design a Data Selection Curriculum for Abstractive Text Summarization. They propose a data selection method for the contrastive learning based approach. \n\n1.  they introduce a dynamic margin instead of a constant margin to improve the loss function proposed in BRIO, called MLR loss.\n\n2. they use both MLR and expected model performance to design a difficulty metric.\n\n3. they use difficulty scores to select data for training.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a learning difficulty metric based on margin-aware list-wise ranking loss and curriculum learning for summarization. The incorporation of a dynamic margin to modify the original loss from the BRIO is a significant step. While the paper showcases promising results on the CNN/DM dataset with respect to both ROUGE metrics and BertScore, the validity of these findings could benefit from a broader array of tests, including other datasets and different pre-trained checkpoints. In addition, one of the reviewers is not convinced by the main conclusion, claiming optimal results in less than 20% of the instances, and believes this statement lacks sufficient rigor. 

Quality: 
- Limited to a single dataset (CNN/DM) for testing, raising questions about the method's generalizability.
- More validation is needed to substantiate the claim regarding the optimal amount of data for training to achieve the highest performance.

Clarity:
- While the paper has been acknowledged for its clear presentation and readability, there is a consensus among reviewers regarding the insufficient detail in explaining the core concepts, notably the difficulty metric and its utilization in curriculum learning. 

Originality:
- The introduction of a dynamic margin in the loss function is somewhat novel for summarization. 
- However, the curriculum learning appears more tailored for contrastive learning models, as indicated by one reviewer, requiring further justification of its applicability for abstractive text summarization.

Significance: 
- The technique proposed holds promise as evidenced by the experimental results, with potential implications for improving summarization performance while reducing the amount of data necessary for training. 
- Yet, the work's significance is somewhat constrained by the limited experimentation and the lack of validation across various datasets.

The paper explores an interesting research topic and integrates recent developments from other areas into text summarization effectively. However, there are significant issues with the paper: it needs clearer explanations of its methods and broader testing on various data sets to strengthen its arguments (one reviewer is concerned with over-claim). While the paper shows promising results, it requires more experiments to fully realize its potential. I am leaning towards accepting it with some reservations, but the overall soundness is not fully satisfactory (even the reviewer gave 4 for soundness indicated in the reasons for rejections many points about the claims are not backed up by enough experiments).
Decision: Accept-Findings