On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study
Authors: Polina Zablotskaia
Du Phan
Joshua Maynez
Shashi Narayan
Jie Ren
Jeremiah Zhe Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Uncertainty
calibration
ensembles
generation
summarisation

Abstract: Modern deep models for summarization attains impressive benchmark performance, but they are prone to generating miscalibrated predictive uncertainty. This means that they assign high confidence to low-quality predictions, leading to compromised reliability and trustworthiness in real-world applications. Probabilistic deep learning methods are common solutions to the miscalibration problem. However, their relative effectiveness in complex autoregressive summarization tasks are not well-understood. In this work, we thoroughly investigate different state-of-the-art probabilistic methods' effectiveness in improving the uncertainty quality of the neural summarization models, across three large-scale benchmarks with varying difficulty using our newly introduced evaluation protocol. We show that the probabilistic methods consistently improve the model's generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. We also reveal notable failure patterns of  probabilistic methods widely-adopted in NLP community (e.g., Deep Ensemble and Monte Carlo Dropout), cautioning the importance of choosing appropriate method for the data setting.

On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study
Polina Zablotskaia'
Du Phan Joshua Maynez Shashi Narayan
Jie Ren Jeremiah Liu Google Research
phandd, p shaasiinrrayo..rjreeeeellooeeeee jtw.seereeeel/egggglee.oooo
Abstract
is mis-aligned with its prediction quality. For ex- ample, in classification tasks, a model said be miscalibrated for all test examples where it predicts with probability 0.9, the model's actual accuracy for these examples deviates far from 90% (Guo et al., 2017; Gneiting et al., 2007). Despite its practical importance, this notion of uncertainty calibration has received much less attention in the summarization literature until recently, with the proposed techniques mostly focusing on training deterministic models (Cao and Wang, 2021; Sun and Li, 2021; Zhao et al., 2022; Liu et al., 2022; Jung et al., 2021).
Modern deep models for summarization at- tains impressive benchmark performance, but they are prone to generating miscalibrated predictive uncertainty. This means that they assign high confidence to low-quality pre- dictions, leading to compromised reliability and trustworthiness in real-world applications. Probabilistic deep learning methods are com- mon solutions to the miscalibration prob- lem. However, their relative effectiveness in complex autoregressive summarization tasks are not well-understood. this work, we thoroughly investigate different state-of-the- art probabilistic methods' effectiveness in im- proving the uncertainty quality of the neu- ral summarization models, across three large- scale benchmarks with varying difficulty us- ing our newly introduced evaluation protocol. We show that the probabilistic methods con- sistently improve model's generation and uncertainty quality, leading to improved selec- tive generation performance (i.e., abstaining from low-quality summaries) in practice. We also reveal notable failure patterns of proba- bilistic methods adopted in NLP com- munity (e-g., Deep Ensemble and Monte Carlo Dropout), cautioning the importance of choos- ing appropriate method for the data setting.
In the uncertainty literature, probabilistic deep learning has emerged as a principled approach to tackle model miscalibration while maintain- ing prediction quality (Nado et al., 2021). In- tuitively, probabilistic DNNs generate multiple plausible predictions from its posterior predictive Pmo (m) = M ΣM Pm. (y] and report the aver- age, thereby mitigating the overconfidence of the individual model prediction. Although well-tested in classification tasks, the effectiveness of differ- ent state-of-ar probabilistic methods in improving neural summarization models' uncertainty quality has been less explored. The existing study mostly focuses on a particular classic method (e.g., Monte Carlo Dropout or MCD) and tested on relatively simple datasets that doesn't fully capture the realis- tic usage (Gidiotis and Tsoumakas, 2022).
1
1 Introduction
In recent years, autoregressive deep models for text summarization have achieved impressive per- formance. However, despite their success, these models often suffer from a critical flaw: they gen- erate prediction with high confidence even when the quality of the summary is low (Xu et al., 2022). This can severely compromise the reliability and trustworthiness of the generated summaries in real- world applications. In the probabilistic forecast literature, such issue is known under the term mis- calibration. i.c., the model's predictive confidence
this work, we address this by conducting a comprehensive investigation of the relative ef- fectiveness of state-of-the-art probabilistic meth- ods in improving the uncertainty quality of neu- ral summarization models. We interrogate both classic approaches such as Monte Carlo Dropout (MCD) and Deep Ensemble (DE), and more recent state-of-art methods such as Batch Ensemble (BB) nermaiald Gaussian process (SNGP) and their combinations that address the latency and quality caveats of the classic methods (Gal and
"Equal contribution.

---

Ghahramani, 2016; Lakshminarayanan al., 2017; Liu et al., 2020; Wen al.. 2020). Furthermore, we evaluate method performance across multiple benchmarks of varying difficulty to ensure the prac- tical relevance of our result, and to uncover poten- tial failure patterns of different approaches. Our contributions are:
rett (2020) noted that calibration methods can be used to improve the accuracy of pre-trained lan- guage models. Jung et al. (2021) proposed a novel training approach to improve calibration by mini- mizing a combined loss of cross-entropy and cali- bration. In the summarization literature, (Cao and Wang. 2021; Xu et al., 2022: Sun and Li, 2021; Zhao et al., 2022; Liu et al., 2022) explored cali- brating model probability using contrastive learn- ing approaches. Most of these techniques focus on deterministic models. They are orthogonal to and can combined with the probabilistic approaches we explore in this work.
We adapt the various probabilistic deep learning methods to the Pre-trained Language Model (PLM) setup and conduct an extensive study on their ef- fect on both uncertainty and prediction aspects of model performance.
We propose a new evaluation protocol to measure the uncertainty calibration performance of summa- rization models, tailored toward domain-specifi quality scores (e.g., ROUGE).
3 Methods
Probabilistic methods have been adopted to in- crease the reliability of pre-trained language mod- els. Plex paper (Tran et al., 2022) provided a nice survey on the robustness of uncertainty methods on text classification tasks. In this study, we opted for the methods that are widely recognized and used. Section A.7 contains in-depth details, here provide general overview:
We show that using probabilistic methods gen- erally leads to improved summarization and cali- bration performance, and consequently improved selective generation. We also discuss the failure patterns of the popular methods such Deep Ensem- bles (Lakshminarayanan et al., 2017) and Monte Carlo Dropout (Gal and Ghahramani, 2016).
2 Related work
Single-model Methods:
Deterministic Baseline we use the base T5 model (Raffel et al., 2020) the baseline model. Monte Carlo Dropout (MCD) (Gal and Ghahra- mani, 2016) which estimates uncertainty using the Monte Carlo average of 10 dropout samples. Those samples are generated using the same model and parameters but with different random seeds at dropout.
Probabilitistic Learning for Seq2seq Models. Developed primarily the context of discrim- inative models, the state-of-art probabilistic ap- proaches can be applied to large neural models without sacrificing performance (Gal and Ghahra- mani, 2016; Lakshminarayanaan et al., 2017; Wen et al., 2020; Liu et al., 2020). Recently, however, initial investigations into unsupervised uncertainty estimation for structured prediction have appeared, primarily focusing on more basic approaches such as Monte Carlo Dropout (MCD) or Deep Ensemble (DE) (Xiao and Wang. 2019; Wang et al., 2019; Fomicheva et al., 2020; Malinin and Gales, 2021; Lin et al.. 2022), with few work looking into summarization tasks (Xu et al., 2020; Gidiotis and Tsoumakas, 2022). In comparison, this work fo- cuses on an unbiased evaluation of wide range of state-of-the-ar probabilistic methods on tasks with varying difficulty, and reveals failure patterns of classic approaches such as MCD and DE.
Batch Ensemble (BE) (Wen et al., 2020) an ensemble method which has much lower computa- tional costs comparing MC Dropout and Deep Ensemble. We replace the last transformer's MLP block by batch ensemble block with ensemble size be 5.2
Spectral-normallzed Neural Gaussian Pro- cess (SNGP) (Liu et al., 2020) a recent state- of-the-art approach which improves uncertainty quality by transforming neural network into an approximate Gaussian process model. The Gaus- sian Process last layer is able reflect the distance between a test example and the training set, hence potentially be helpful in improving calibration. SNGP+MCD which is the MC Dropout on top of an SNGP model;
Calibration Technique in Language Processing. Guo et al. (2017) proposed improving calibration of document classifier using of temperature scal- ing. Müller et al. (2019) and Wang et al. (2020a) explored improving calibration in neural machine translation using label smoothing. Desai and Dur-
1 'All methods can applied to larger models. "BF requires more memory on single machine, so we keep the ensemble size below 10.

---

Multi-model Methods:
ing them with the baseline deterministic model. We use ROUGE-1/2/L (Lin, 2004) to measure general summarization quality. As shown in Table 1, we observe the consistent improvement of the ROUGE scores in probabilistic models compared to base- lines. For single model methods, SNGP achieves the highest average ROUGE scores over the three datasets. Other probabilistic methods also show promising performance: SNGP+MCD is ranked the second top regarding ROUGE-I, and BE is ranked the second top regarding ROUGE-2 and the top regarding ROUGE-L. For multiple model methods, SNGP+DE improves over the determinis- tic DE. Comparing multiple model methods with single model methods, DE and SNGP+DE gener- ally have higher ROUGE scores than single model methods.
Deep Ensemble (DE) (Lakshminarayanan et al., 2017) which trains 10 deterministic models indi- vidually and averages all. We use the same model architecture but changing the initial seeds.
Gaussian Process Ensemble (SNGP+DE) is the combination of deep ensemble and SNGP.
For all methods, we use the official base T5 checkpoint, which are pretrained on a large cor- pus like C4 (Raffel et al., 2020). We then fine- tune the parameters on summarization tasks. To generate prediction from the model posterior in all, we perform beam inference with respect to the model's conditional posterior mean probabil- ity, i.e., P(ur .²) = M Em= Pm (vl,,et,, z),, where M- = 10 is the number of samples from model posterior (for the deterministic baseline and SNGP-only method M = 1). To quantify model uncertainty, we consider the Sength-normalize pre- dicted log-probabillties following previous work, i.e., x(3(a)) := + E'n p(yelyee, (Wu et al.. 2016; Liu et al., 2022) where x is the input se- quence, y is the output sequence, yt is the t-th token of y, and T is the length of the sequence, i.e. the number of tokens in y.
M
ROUGE-11
Method
XSUM
CNN/DM
Roddit
Average
Average Rank]
Base
40.83
41.19
26.14
36.05
2.67
Bose muclous
40.03
40.66
24.54
35.08
5.33
SNGP
41.76
26.08
36.21
MCD
40.31
40.68
24.27
35,09
5.00
SNGP-MCD
40.90
41.49
24.60
35.66
2.35
DE
41.21
41.22
23.36
35,26
3.33
DE
41.51
41.20
26.65
36.45
1.67
SNGP-DE
42.14
41.99
26.57
36.90
ROUGE-21
Method
XSUM
CNN/DM
Reddit
Average"
Average Rink]
4 Experiments
Base
19.14
19.77
7.76
15:56
2.67
Basc muclous
18.15
18.89
6.47
14.50
6.00
SNGP
18.91
20.19
7.76
15,62
2.00
For the first time, we benchmark the probabilis- tic uncertainty calibration methods. We use our proposed evaluation protocol, consisting of assess- ing the ROUGE quality improvements, measur- ing uncertainty calibration, which includes our newly proposed sequence-level Expected Calibra- tion Error (ECE), assessing rank correlation and finally analysing selective generation via absten- tion as a measure of language model calibration. We evaluate all methods three datasets: XSUM (Narayan et al., 2018), CNN//ailyMail (Hermann et al., 2015; See al., 2017) and ReddiCTIFU- long (Kim et al., 2019) due to their diversity in abstractiveness, lengths, domain and style (see Sec- tion A.6 for additional details). For all experiments we use beam search decoding, which was adapted to work with ensemble generation. We have also adapted SNGP and BE algorithms to work with the sequence generation and the corresponding loss (see Section A.7 for more details).
MCD
18.63
19.78
7.12
15.18
4.33
SNGP-MCD
18.91
20.33
7.00
15.41
3.00
BE
19.41
19.81
7.31
15.51
2.33
DE SNGP-DE
19.84
19.77
8.21
19,81
1.67
20.35
20.49
7.77
16.20
1,33
ROUGE-L
Method
XSUM
CNN/DM
Reddit
Averigge"
Average Runk]
Base
33.76
38.54
21.31
31,20
2.33
Base nucleus
38.13
19.79
30.32
SNGP
33.53
39.12
21.17
31.27
2.67
MCD
33.21
38.09
20.06
30.45
5.00
SNGP-MCD
33.59
38.97
20.00
30.85
3.00
DE 34.06
38.50
20.81
31.12
2.00
34.38
38.54
21.72
31.55
1,33
SNGP-DE
34.92
36.55
21.18
30.88
1.67
Table I: ROUGE scores and ranking of different probabilistic methods across all datasets. Probablistic methods consistently outperform base model, and SNGP-family family models generally lead to strong performance. For this experiment we addi- tionally conduct evaluation of the Base model with nucleus sampling (p = 0.5), which sometimes improves model perfor- mance by truncating the less reliable tail of the distribution. however it doesn't 't change the model calibration. The best and second-best results are denoted by bold and underlined formats, respectively.
4.2 Measuring Uncertainty Calibration in Summarization
4.1 ROUGE with Probabilistic Methods
We first study the effectiveness of different proba- bilistic methods on summary prediction by compar-
We now study model's uncertainty calibration qual- ity. We consider both the classic metric Expected

---

XSum
CNN/DailyMell
Reddit Tifu
57.5
50
32
55.0
ROUGE-1
52.5
49
30
50.0
DE
46
SNCIPDDE
45.0
42
40.0
24
0.6
0.4
Abstention
Abstention
Figure I: ROUGE vs Abstention Curve for different probahilistic methods. Abstention rate denotes the percentage of examples that excluded, after ranking according log probabilities. For single model methods (solid lines), SNGP+MCD models have generally higher ROUGE scores in CNN/DM, and regions of a 0.6 in XSUM and Reddit. For multi-model methods, SNGP+DE generally outperforms DE in all the three datasets. See Figure for the results other ROUGE scores.
Calibration Error (ECE), and also the uncertainty score's Spearman's 's rank correlation with domain- specific quality scores tailored for summarization (e.g., ROUGE).
As shown in Table 2, across all methods, SNGP+MCD and SNGP+DE generally leads lower ECE in single model and multi- multi-model meth- ods respectively, suggesting SNGP helps to reduce ECE.
ECE. In order to evaluate whether the model es-
timated probabilities have been more calibrated assess the difference in expectation between confi- dence and accuracy using ECE metric (Nacini et al., 2015):
Seqpenee-lve
Average
Averge
7.71
Table 2: ECE on sequence and token levels of different
probabilistic methods across all datasets. SNGP+MCD and SNGP+DE generally leads to lower ECE in single model and multi-model methods, respectively.
Bul (conf(Ba) - (x)(Bx)..
ECE Σ A=1
where we split the interval (0, 1] into K equal- size bins and define B& to be the set contain- ing the indices of examples which have pre- dicted probability lie in the k-th bin: Bg {ilpe E (#+,, }1}. where the average accuracy and confidence within each bin are defined acc(Bx)
Rank Correlation with Quality Score. We in- vestigate how the Spearman's 's rank correlation be- tween the log-probabilities and ROUGE changes with calibration. Overall we see a general trend demonstrating the calibration increases the corre- lation, shown in Figure 2. For the ROC-AUC scores please refer to the section A.I.
Be DieB. I vi) and (Bnf(Bx)= J Eien, Pi-
In auto-regressive prediction, Ús can be a se- quence or a token, which corresponds to two dif- ferent metrics sequence-lewe ECE and token-level ECE respectively. When computing the sequence- level ECE, we cast the problem into a binary classi- fication task, where the probability á of predicted sequence is the production of probabilities of all tokens of that sequence. Regarding the token-level ECE, other work (e.g. Wang al. (2020b)) uses translation edit rate (Snover et al., 2006) relax the condition that the tokens under consideration need to be at the same position. In our work, we say that a predicted token is correct if it matches the target token at the same position in the target sequence.
ROUGE1
50
25
24
10
And
Figure 2: Spearman's rank correlation between the length- normalized log probabilities and the ROUGE-L I. We compute the hars using bootstrap standard deviation technique. The best second-bes results are denoted by bold and underlined formats, respectively.
4.3 Selective Generation via Abstention
Selective generation refers the procedure se- lectively generate higher-quality outputs while ab- stain the low-quality outputs (Ren et al., 2022). It evaluates the models' uncertainty calibration, since
DDuring evaluation, we compute token probabilities from the highest scoring heam sequence.

---

well calibrated model is expected to have high uncertainty for low-quality outputs such it can be used to select examples to abstain. We use the ROUGE vs Abstention Curve to compare meth- ods: specifically, at a given abstention rate a, we remove the lowest a-fraction uncertain examples and compute the average quality of the remaining examples, as shown in Figure 1. For single model methods (solid lines), SNGP+MCD models have generally higher ROUGE scores CNN/DM, and in regions of > 0.6 in XSUM and Reddit. For multi-model methods, SNGP+DE generally outper- forms DE in all the datasets.
with SNGP as an approach to language model calibration, since they are effective in terms of reducing the ECE and improving the quality of summarization.
Even when calibration methods appear be effective in reducing ECE it may not neces- sarily suggest that they will be effective improving ROUGE or other language genera- tion quality metrics.
We hope to see our efforts making a significant contribution to the improvements of the reliability of PLMs, enabling future researchers to effectively leverage probabilistic calibration methods in the development and analysis of these models.
Failure Patterns. When comparing multi-model methods with single model methods, we observe that XSUM and Reddit both have multi-model methods outperforming single model methods, but CNN/DM does not benefit from using multi-model methods. This difference can be explained by the fact that CNN/DM is simpler task that is more extractive in nature, and single model already per- forms well and relatively calibrated. In this case, using a deep ensemble can in fact lead to under- confidence (Rahaman et al., 2021). Furthermore, in Reddit dataset, MCD-family methods seem to lead to severe degradation of summarization quality, Note that Reddit is more challenging task with much greater linguistic diversity when compared XSUM and CNN/DaillMail, cautioning the of MCD method in challenging test environments where single model does not perform well.
Limitations
In our paper we investigated the effect of most com- mon and widely used probabilistic deep learning methods on Pre-trained Language Models. We ob- served positive effect of calibration on the variety of metrics, however it is important address the limitations of this work. Even though observed the improvements on multiple metrics, we believe more work needs be done to fully understand- ing the interplay between classic probabilistic deep learning methods that were traditionally applied in the context of classification and the unique set up the language models pose: autoregressive gen- eration and mismatch between the learning and inference. Above mentioned unique properties of PLMs make harder align the model predic- tive probability with the distribution of the data. In our work we have done very basic adaptation of the existing methods to the PLM setting, but future more work needs to be done to address these differences.
5 Conclusion
We have adapted the most popular probabilistic cal- ibration methods to the PLMs use-case and have conducted novel and extensive study of the cali- bration effect on PLMs uncertainty and summariza- tion quality. We proposed novel evaluation pro- tocol for the uncertainty estimation in PLMs. We demonstrated that probabilistic calibration meth- ods can have a positive impact on the quality of generated summaries and increase reliability of the models. Our main findings can be summarized in two points:
Ethical impact
Our work directly contributes to the topic reli- able deep learning. We believe our work should positively affect the scientific community, since we address one of the main problems that often occurs in the machine learning research: how to make the models more reliable and trustworthy. We hope that in long run we can arrive standard- ized benchmark set of techniques that can help the NLP community develop PLMs that are universally trustworthy.
When there is time and memory con- straints, our results point out that it is best to choose Deep Ensembles in combination
"Different from the performance vs data retention curves in Filos et al. (2019), we employ log probability rather than predictive entropy as the metric for data retention.

---

References
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 25 2519-2531, Min- neapolis, Minnesota. Association for Computational Linguisties.
Shuyang Cao and Lu Wang. 2021. Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649
Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 295-302, Online. Association for Computational Linguisties.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Mi- lad Alizadeh, Arooud De Kroon, and Yarin Gal. 2019. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks. arXiv preprint 101401.
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74- 74-81, Barcelona, Spain. Association for Computational Linguisties.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the As- sociation for Computational Linguistics, 8:539-555.
Zi Lin, Jeremiah Liu, and Jingbo Shang. 2022. Neural- symbolic inference for robust autoregressive graph parsing via compositional uncertainty quantification. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 4759-4776.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uneer- tainty in deep learning. In international conference on machine learning, pages 1050-1059, PMLR.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. 2020. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. Advances in Neural Information Processing Systems, 33:7498-7512.
Alexios Gidiotis and Grigorios Tsoumakas. 2022. Should we trust this summary? bayesian abstractive summarization to the rescue. In Findings of the As- sociation for Computational Linguisties: ACL 2022,
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203. 16804.
pages 4119 4131.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. 2007. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Soci- ety: Series (Statistical Methodology), 69(2):233 268.
Andrey Malinin and Mark John Francis Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Länguistics, pages 1906-1919.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. International conference on machine learning, pages 1321-1330 PMLR.
Karl Moritz Hermann. Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Rafael Müller, Simon Korbblith, and Geoffrey E Hin- ton. 2019. When does label smoothing help? Ad- vances in neural information processing systems, 32.
Zachary Nado, Neil Band, Mark Collier, Josip Djo- longa, Michael W Dusenberry, Sebastian Far- quhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, et al. 2021. Uncertainty base- lines: Benchmarks for uncertainty & robustness in deep learning. arXiv preprint 1xXxx:2106.0401.
Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. 2020. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7):67-78.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated prob- abilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 29.
Christopher Jung, Changhwa Lee, Mallesh Pai, Aaron Roth, and Rakesh Vohra. 2021. Moment multicali- bration for uncertainty estimation. In Conference on Learning Theory. pages 2634-2678. PMLR

---

Shashi Narayan, Shay Cohen, and Maria Lapata. 2018. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In 2018 Conference on Empirical Methods in Natural Language Processing. pages 1797-1807. Association for Computational Linguis- tics.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving back-translation confidence estimation. In
with uncertainty-basee Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IICNIP), pages 791- 802.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(11:5485-555 -5551.
Shuo Wang. Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020a. On the inference calibration of neural ma- chine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3070-3079, Online. Association for Computational Linguistics.
Rahul Rahaman et al. 2021. Uncertainty quantification and deep ensembles. Advances in Neural Informa- tion Processing Systems, 34:20063-2008
Shuo Wang. Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020b. On the inference calibration of neural ma- chine translation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Lin- guistics, pages 3070-3079.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mo- hammad Salch, Balaji Lakshminarayanann, and Pe- detection and se-
ter J Liu. 2022. lective generation for conditional language models. arXiv preprint 2209: 15558.
Yeming Wen, Dustin Tran, and Jimmy Ba. 2020. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin. Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni. Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lec- Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Shepard, Noah Fiedel, Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203. 17189.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey. Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between hu- man and machine translation. arXiv preprint arXiv:1609.000 0609.08144.
Yijun Xiao and William Yang Wang. 2019. Quanti- fying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on ar- rificial intelligence, volume 33, pages 7322-7329.
Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020. Un- derstanding neural abstractive summarization mod- els via uncertainty. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 6275-6281.
Abigail See. Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1073- 1083.
Shusheng Xu, Xingxing Zhang. Yi Wu, and Furu Wei. 2022. Sequence level contrastive learning for text summarization. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11556-11565.
Matthew Snover, Bonnic Dorr. Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of
translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Associa- tion for Machine Translation in the Americas: Tech- nical Papers,, pages 223-231.
Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.0004
Shichao Sun and Wenjie Li. 2021. Alleviating expo- sure bias via contrastive learning for abstractive text summarization. arXiv preprint arXiv:2108. 1846.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. 2022. Plex: Towards reliability using pretrained large model ex- tensions. arXiv preprint (xXiv:2207.07411

---

A Appendix
A.1 ROC-AUC scores
Marric Method Type
Michod
XSUM
CNNDaal-Mail
Rudii Tife
Rase
73
60.35
60.8
ROCOBI
angle-model
MCD
75.06
62:29
09.58
SNGP+MCD
75.14
62.05
67.M0
69.45
59.97
Molti-mudiel (10)
SNGI+116
75 19
59.25
63.00
Bise
73.00
60.22
59.33
SNGP
73.46
ROUGE-2
Single model
SNGP-MCD
74.58
63.22
63.58
BE
72.82
$9.90
63.23
(10)
Rise
71.25
39.35
61.34
SNCP
72.86
59.06
19
ROUGEL
Singlo-mode
MCD
SNGP-MOD
74.61
61.35
BB
72.45
60035
64.42
Mutti model (10)
7285
56.78
65.0
Table 3: We measure the Area Under Curve values, when using the log-probabilities as signal for "good/bad" summaries. Good and bad summaries are defined by threshold 0 we impose on the metric, i.e. when metric is above certain @ then we treat the summary as good and when it below we treat a bad summary. We used the following I for ROUGEI, ROUGE-2 and ROUGE-L correspondingly: 40, 15, and 30
A.2 Spearman's rank correlation ROUGE-2 and ROUGE-L Spearman's 's rank correlation for the rest of the metries can be found on Figure 3.
ROUGE-2
40
30.0
30
Spearnan
27.5
SMD+-DD
50
25.0
20
22.5
10
XSUM
20.0
CNN/DallyMail ROUGE-L
Reddit Tifu
30
Spearnnn %
55
25
50
20
20
CNN/DallyMail Figure 3: Spearman's rank correlation between the length-normalized prg-prbbabilities and the ROUGE-2 and ROUGE-L.
XSUM
Reddit TIfu
A.3 Abstention plots
We demonstrate the abstention plots for the rest of the metrics on Figure 4.
A.4
Experimental details
We ran all the experiments on the T5 base model (220 million parameters) using open-sourced T5X framework (Roberts al., 2022). We used the TPU v3 chips (Jouppi et al., 2020) for all experiments. Reported metric results are collected from a single evaluation run, unless error bars are provided or
"The values were selected as round numbers on the SoTA performance circa 2020, when NI.G fluency was human-like. https://ppperswithcode.cmmm

---

Mall
13
BASE
32
BASE
SHOP
SHOP
ShOP
35
MCD
30
MCD
MCD
SNCP-MICD
SNCF-NCC
SPGPP MMCD
11
28
ROUGE-2
26
25
24
22
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Abstention rate
Abstention rate
Abstention rate
CNN//DailyMail
Reddit Tifu
BASE SNCP
BAGE
27
BAST SNGP
MCD
46
SHOP
MCD
26
NCD
25
ROUGE-L
45
42
40
22
21
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6 0.8
Abstention rate
Abstention rate
Abstention rate
Figure 4: ROUGE-2 and ROUGE-L abstention plots.
stated otherwise. To select each model checkpoint we ran a hyperparameter sweep find the best of parameters. Parameters we sweeped over were: checkpoint step, leaning rate, SNGP mean field factor, number checkpoints for the ensembles and number training steps.
In all the experiments we use beam search (except where we stated "nucleus sampling") as a decoding method and we use beam_ size= 3. For the MCD we used dropout_rate rate = 0.1 everywhere. Covariance matrix momentum in SNGP was set to 0.999. For the XSum the best mean field factor 10-4 and for CNN and Reddit it was 10-6.

---

A.5
Qualitative results
Base model ROUGE 1 scooee
%
70
SNGP+DE model ROUGE 1 score
Figure 5: Scatter plot for ROUGE scores of SNGP+DE and Base models on RedditTifu task. Detailed contents and symbols can be founded in Tables 4 and respectively.
A.6 Dataset
XSUM (Narayan et al., 2018) consists of 227k BBC articles from 2010 to 2017 with a single sentence highly abstractive summary. The average length each input document is 687 words, whereas the summary is 53 words long. Sometimes the summary contains information not present in the article (Maynez et al., 2020).
MNNiDailyMaiil (Hermann al., 2015; See al., 2017) contains 313k articles from the CNN and Daily Mail newspapers with bullet point summaries. The summaries are average 3-4 sentences and relatively extractive. The average length of each input document is 375 words, whereas the summary is 21 words long.
RedditTIFU-long (Kim et al., 2019) contains 42k posts of informal stories from sub-reddit TIFU from 2013-Jan to 2018-Mar with author written summaries. The style and length of the summaries are very diverse. The average length of each input document 433 words, whereas the summary is 23 words long.
A.7 Methods details
In this section, we delve into the methods, offering a comprehensive explanation of their individual characteristics and how their combined properties can best help in model calibration. Deep Ensembles, Monte Carlo dropout, SNGP, and Batch Ensemble are all techniques used in the field of machine learning improve the performance and robustness of neural network models. While they share the goal of calibrating the model predictions, they differ in their underlying principles and methodologies. Let's discuss each method and highlight their fundamental differences.
The main idea behind Monte Carlo Dropout, Deep Ensembles and Batch Ensembles is to get an approximation of the predictive posterior model via computing a mean of token probabilities from several unique models. Once we have access M predictions we can compute the predictive posterior the

---

our porch, having some drinks and horderves before giving dad his father's 's day gift, some point my mom notices one our neighbors walking down street. she points out that hes wearing these funny shoes and turn around to see him shuffling his feet while wearing these giant slippers that look like part of a goofy costume. as he's crossing the street to check his daughter playing at her friends house my dad yells "love those shoes!" and the guy waves and responds "thunks, they't 're from her." while pointing his doughter.
Input: summarize: my family
him and his daughter exchange a few words and he turns around to walk back home. you have imagine this guy moving his body side to side while walking, dragging his one foot and swinging the other around, making it look like the slippers were heavy (since they were quite large).
he passes in front of our house again, decide to poke little fun the way he walked in the slippers and yelled with chuckle, "hey y k'now you walk little funny those!" which he replied, "heh, dont have much of choice," and continues walking with his head down. my brain, was saying "there's no other way to walk in these damn things." so laughed audibly so could hear i appreciated the response and attitude about the silly present.
once he's inside, my mom uums
and says. "yknow he used to weigh like 300 pounds, he's lost over 120. be also has cerebral palsy he's always kinda walked with limp."
Target: thought a guy's shoes
the reason he walked funny, turns out he has cerebral palsy.
Base model: made fun of guy walking in giant slippers father's day.
SNGP+DE model: laughed at the way a guy walked in his slippers, turns out he has cerebral palsy.
Table 4: An example where SNGP+DE model gives better ROUGE1 score than Base model. This is annotated by symbol in Figure 5.
following way:
M
Faylar, D) = Eot (p(y))p((),,,)))) 0)] R
Σ
Pm (mla (lm)), ((m) ~ g(0) ~ P(ODD)
(I)
M
Here, are the model parameters, y is the model prediction and the model input (for simplicity we consider y to be a whole sentence instead of an individual token, that are dependent on the previous tokens). (0) is the parameters prior distribution, in our case we used standard normal distribution to
initialize the model parameters. Once we have an approximation of the model's predictive posterior we can estimate the expected uncertainty:
D)) Hyp(y/x D)] = -Bp(lleD) In pnyly, D)] = Σ DEY
In ((
(2)
The uncertainty is essentially the Shannon entropy, and therefore can be easily calculated, once we have the values and probabilities across the whole space of possible outputs V, in practice we compute everything at the token level, and y just becomes the models' vocabulary.
Monte Carlo Dropout (MCD) (Gal and Ghahramani, 2016) is technique that leverages dropout regularization during training inference. Dropout is regularization technique that randomly sets a fraction of neural network units to zero during training. During inference, MCD involves performing multiple forward passes with dropout enabled (but with different random seeds) and averaging the predictions (more precisely, the probabilities of all tokens in the vocabulary). By sampling multiple predictions, MCD provides a measure of model uncertainty or confidence in its predictions. In our case MCD estimates uncertainty using the Monte Carlo average of 10 dropout samples.
Batch Ensemble (BE) (Wen et al., 2020) an ensemble method which has much lower computational costs comparing to MC Dropout and Deep Ensemble. Batch Ensemble is technique that involves training multiple neural network models simultancously within a single batch. Each model in the ensemble receives a different subset of the data batch, and the models share their weights the end of each training iteration. By training models on different data subsets, Batch Ensemble encourages diversity and reduces overfitting. In the experiments, we replaced the last transformer's ML. block and the last dense layer by batch ensemble blocks with ensemble size be 5.

---

Input: summarize: ilf] have small hairs on my lip und waxing doesn't seem work because of how tiny and thin they are, shaving doesn't help very much since it causes farther irritation. when i was feeling extra self concious after trimming the hairs on valentines day, my fiuncé brought up trying the cream hair removal.
went on amazon and bought the veet hair removal cream. last night, got it in the mail and read the precautions.
not it the face but like idiot, thought to ignore it (us does every story about the hair removal cream). totally did the trick and my lip hairless. felt a of burning and irritation on the lip after but went away after using bit of bio oil. didn't think about all night.
this morning on the other hand woke up with more pain that you would feel after a burn, there was small patch of skin breakdown and irritation to the lett of my lip and bit of redness my upper lip but nothing more. covered it up with makeup and it seemed have done the trick.
fast forward about 8hrs. now have small pin sized seabs across my upper lip and pain. look like 15 year old boy who docsn't know how to shave or someone with uncontrollable herpes cold sores. ontop of that got a venus razors ud while write this on my smurt phone to rub it in some more that i should 've used razor. the next week will be ugly.
Target: used hair remover cream on face, now have chemical burn the upper lip
Base model: used veet hair removal cream my lip and now have small scabs all my upper lip.
SNGP+DE model: veet huir removal cream made me look like an idiot.
Table 5: An example where SNGP+DE model gives worse ROUGEI score than Base model. This is annotated by symbol in Figure 5.
Deep Ensemble (DE) (Lakshminarayanan et al., 2017) which trains 10 deterministic models individ- ually and averages all. Deep Ensembles involve training multiple neural network models independently and then combining their predictions to make a final prediction. Each model in the ensemble typically trained with different initialization, architectures, or subsets of the training data. The main idea is that the diverse models capture different aspects of the data and, when combined, produce more accurate and robust predictions. Deep Ensembles are computationally expensive since they require training and storing multiple models.
The above mentioned methods very good at increasing the representation diversity, i.c. learning the multiple modes of the underlying data distribution. However, when dealing with uncertainty it is also important that model has distance awareness, i.e. the property of the model that allows quantify the distance of Fest and X train in the input space ||arest Xtrain ||=- Typical Neural Networks are not distance aware. Spectral- normalized Ncural Gaussian Process was proposed as a solution for this problem:
SNGP (Liu et al., 2020) a recent state-of tfte--of-the-an art approach which improves uncertainty quality by transforming a neural network into an approximate Gaussian process model. The Gaussian Process last layer is able to reflect the distance between test example and the training set, hence potentially be helpful in improving calibration. The model architecture includes neural network that maps inputs a power-dimensional feature space, and Gaussian process layer that models the target variable using the extracted features. We adapted SNGP to the sequence generation by letting all pre-logits (i.e. inputs of the last dense layer) in a sequence go through the same GP layer.
Gaussian process
Spectral Noraaizzdd DNN
X
h(x)
Random Features
p(y|x)
Laplace Approximation
Neural Network
Figure 6: Architecture changes for the SNGP model. Spectral normalization enforces bi-Lipschitz smoothness. which discourages inputs that are far away in input space get mapped close in hidden space. For GP layer confi- dence is a function of distance from the training data.

---

In summary, the fundamental differences among these techniques lie in their approaches ensemble learning, uncertainty estimation, and distance awareness. Deep Ensembles train multiple models inde- pendently and combine their predictions, Monte Carlo dropout incorporates dropout during training and testing, Batch Ensemble trains multiple models within a single batch and shares their weights and SNGP combines neural networks with Gaussian processes. Each method offers unique advantages and can be employed based on the specific requirements of the problem at hand, our experiments showed that the best results can be achieved by combining the SNGP with Deep Ensembles, which gives us the best of both worlds, i.e. representation diversity and distance awareness. See Table 6 for the summaries of different methods properties.
Distance awareness
Number of models
Number of inference runs
Simplicity of Representation implementation
Method
diversity
MCD BE
X
10
X
1
X
SNGP
SNGP+MCD
V
10
DE
X
10
10
SNGP+DE
/
10
10
Table 6: Comparing studied calibration techniques against selected properties.
Finally, in order compare the algorithms it is helpful to look the total complexity:
SNGP, BE: time complexity O(T). space complexity O(S).
MCD, SNGP+MCD: time complexity OMM x T), space complexity O(S),
DE, SN:P+DE: time complexity ((M x T), space complexity O(M x 5),
where T and S are time and space complexity the base model and M the ensemble size. To reduce the memory required in DE models, we compute the predictions (i.e. the probabilities of all tokens in the vocabulary) of each model sequentially before averaging them. For that reason, the DE models have a time complexity ((M x T). The actual time and memory required for each method depend on the TPU topology, and they roughly align with the time and space complexity mentioned above.


Reviews:
[{"review_id": "k35il7zGoS", "reviewer": "Reviewer_X5Gn", "summary": "", "questions": "A. Why isn't the AESLC dataset used for this study? \n\nB.  The \"quality vs abstention curve\" is essentially the same as the \"performance vs data retention curve\" as proposed by Filos et al. 2019, isn't it?  If so, it's better to cite and include the reference:\n\nAngelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G.J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud De Kroon, and Yarin Gal. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks, 2019. ISSN 23318422.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helpful comments and pointing out the typos. We will incorporate the suggestions into the camera ready version. We hope that our responses below will address your concerns.\n\n\n**Average rank for table 2**\n\nWe agree with the reviewer that having an average rank in Table 2 would make the results more consistent and clear, therefore we will add the average rank metric to Table 2 for the camera ready version.\n\n\n**Evaluation novelty**\n\nAlthough we acknowledge the reviewer's observation that the individual calibration approaches are not inherently groundbreaking, we believe the novelty of our paper lies in implementing and adapting these calibration techniques inside the LLM framework and proposing the evaluation metrics for these methods within the context of the LLM domain.\n\n\n**Studying the AESLC dataset**\n\nWe are originally interested in standard tasks without too brief summarization. Summaries in the AESLC dataset are the subjects of emails, which are a bit short and vague. We agree that studying uncertainty methods on this dataset is a worthwhile task. We want to leave it open for further investigation.\n\n**Relationship between “quality vs abstention curve” and “performance vs data retention curve”**\n\nWe appreciate the reviewer bringing up the paper by Filos et al. 2019. In Filos et al. 2019 data is retained based on predictive entropy. In our paper, we employ log probability as the metric for data retention. In our camera-ready version, we will properly cite the Filos et al. 2019 paper and explain this distinction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper performs systematic evaluation of the probabilistic methods in uncertainty calibration in neural summarization models. The evaluation is performed on classic and state-of-the-art methods, on three datasets, and with evaluation metrics of ROUGE, ECE, correlation between quality and predictive confidence, and quality vs abstention rate.", "reasons_to_reject": "1. It is not quite reassuring to see that the authors use the \"average\" performances from the three datasets (as in Tables 1 and 2) to prove the effectiveness of the probabilistic models. The average rank as in Table 1 is more reasonable. And yet, the average rank is not used in Table 2, which needs explanation.\n\n2. The evaluation is systematic, covering quality (with ROUGE), calibration (sequence-level and token-level ECE), quality vs predictive confidence, and quality vs abstention rate.  However it is a bit farfetched to say it's novel.", "questions_to_authors": "A. Why isn't the AESLC dataset used for this study? \n\nB.  The \"quality vs abstention curve\" is essentially the same as the \"performance vs data retention curve\" as proposed by Filos et al. 2019, isn't it?  If so, it's better to cite and include the reference:\n\nAngelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G.J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud De Kroon, and Yarin Gal. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks, 2019. ISSN 23318422.", "paper_topic_and_main_contributions": "The paper addresses the research gap that uncertainty calibration has not been well studied in summarization, especially with probabilistic neural summarization methods.  The paper investigates the effectiveness of the classic approaches and the state-of-the-art methods of probabilistic methods in improving the uncertainty quality of the neural summarization models. The paper also proposes the evaluation protocol including: ROUGE for quality improvement, sequence-level Expected Calibration Error (ECE) for calibration improvement, rank correlation for quality and predictive confidence as they changed with calibration, and quality vs abstention curve that measures change in quality as certain fraction of uncertain prediction examples are removed.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Shouldn't the \"access\" as in lines 208 and 247 be \"assess\"?\n\nAbove line 272, caption of Figure 2: \"standart\" should be \"standard\".\n\nLine 081: \"latecy\" ?\n\nLine 202: \"we\" should be \"We\".\n\nNotes need to be added to Table 1 and 2 about the meaning of the \"bolded\" and the \"underlined\" format."}}, {"review_id": "4GVSu1VX3L", "reviewer": "Reviewer_o7C1", "summary": "", "questions": "1. L244: What is \"beam decoding\"? Is it beam search or beam sampling? You should explain it clearly.\n2. L542: \"Sometimes the summary contains information not present in the article.\" Who proved it?\n3. What is the criterion with which you have selected the following values? 40, 15 and 30 (caption of Table 3).\n4. Why did you choose only the T5-base model for the experimental phase?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the thoughtful comments and suggestions. We will fix the typos in the camera-ready version. We hope that our response below will address your concerns.\n\n**Evaluate multiple language models of different sizes**\n\nWhile we appreciate the reviewer feedback, we believe evaluation of the model on such an extensive set of checkpoints goes beyond the scope of the short paper expectation. Additionally we believe that evaluation across multiple architectures should be motivated by finding new information about how the calibration methods work in LLMs. While it would be interesting to see if the model size or instruction tuning can affect the calibration method’s performance, we are not addressing these questions in this work, but rather focus on a general assessment. Finally, we would like to point out that T5 is one of the most widely used transformer based model architectures, so we believe our finding would be generally interesting to the community.\n\n\n**Include a long document summarization dataset**\n\n\nWe thank the reviewer for the suggestions and the idea to evaluate the uncertainty calibration in the long document summarization setting. We believe it would be a great research study, and we will definitely consider looking into it in a follow up work. In this paper our aim was to study the application of uncertainty calibration probabilistic deep learning methods to LLMs in a general setting, therefore we have chosen datasets that are in line with the standard benchmark datasets that are usually used in summarization work.\n\n\n**Novel methods to address the problem of miscalibration**\n\n\nWe appreciate the reviewer pointing this out. We presume that there might have been a level of confusion, that we would like to correct by explaining the main contributions. We wish to clarify that we don’t claim a novel method that addresses the problem of miscalibration, we do however claim that we have adapted the standard probabilistic calibration methods (that are generally used in the classification tasks) to the LLMs for a sequence generation task and benchmarked them for the first time using newly proposed set of evaluation metrics. We will make sure this is completely clear in the camera ready version. \n\n**T5-base is far from being an LLM**\n\nTo the best of our knowledge there is no formal widely accepted definition of the LLM term. LLMs and PLMs have been used interchangeably, T5 is a group of pretrained LMs, there are currently more modern LLMs that have sizes similar to T5 models. In several published and academically recognised works T5-base checkpoint is regarded as an LLM and we share the same understanding.\nIt would definitely be interesting to perform the study using LLMs like GPT-3. More infrastructure work will be required to employ uncertainty methods considered in the paper to those LLMs. We leave it open for future study.\n\n\n**The dataset statistics**\n\nWe thank the reviewer for the idea to include this information into the paper. For now we provide the following statistics on the word account and we will later add it into the paper (together with the information about the number of sentences):\n - The CNN/DailyMail dataset has on average 687 words in the input document and around 53 words on average in the summary. \n - XSum has 375 words on average in the input document and 21 words in the summary. \n - Reddit Tifu has 433 words on average per input document and 23 words per summary.\n\n**Time and space complexity of each method**\n\nThank you for the question! We will clarify the complexity of each method compared to the time complexity $T$ and space complexity $M$ of the base model as below.\n\n - SNGP, BE: time complexity $O(T)$, space complexity $O(M)$\n - MCD, SNGP+MCD: time complexity $O({num \\textunderscore mc \\textunderscore  samples}  \\times  T)$, space complexity $O(M)$.\n - DE, SNGP+DE: time complexity $O(ensemble \\textunderscore size \\times T)$, space complexity $O(ensemble \\textunderscore size \\times M)$.\n\n\n**What is \"beam decoding\"?**\n\nWe apologize that the term was not clear and we will fix this in the camera ready version, and refer to the term as \"beam search decoding\", however we would like to point out that it is not uncommon to refer to \"beam search decoding\" algorithm as \"beam decoding\", for example see \"Beam decoding with controlled patience\" by\nJ Kasai, K Sakaguchi, RL Bras, D Radev, Y Choi, NA Smith.\n\n\n**In XSUM dataset, the summary might contain information not present in the article**\n\nWe thank the reviewer for pointing this out, we agree that the sentence can be phrased differently and we can provide an example from XSum that confirms the claim, which we promise to include this into the final version of the paper:\n\n- Input document:\n\"Sustainable Shetland, a group opposed to the development, has announced it intends to seek a judicial review of the Scottish government's decision to approve the development. The wind farm would be the third biggest in Scotland, run by community company Viking Energy. Energy Minister Fergus Ewing granted consent for the scheme in April. Protesters claim the development is too big and would blight the landscape. Supporters argue it would raise money for the islands, create jobs and help meet renewable energy targets. The 370MW wind farm is aimed at powering more than 175,000 homes despite Shetland having a population of about 22,000. It is estimated the wind farm could bring about  £30m annual income for the local community.\"\n\n- Summary:  \n\"Controversial plans to build a **103-turbine** wind farm in the centre of Shetland could face a legal challenge.\"\n\n**\"103-turbine\"** only appears in the summary and not in the text.\n\nPlease also refer to the paper named  \"On Faithfulness and Factuality in Abstractive Summarization\" by\nJoshua Maynez, Shashi Narayan, Bernd Bohnet, Ryan McDonald, where authors discuss this phenomenon.\n\n**Criterion for selecting the values 40, 15 and 30 in the Table 3**\n\nThe values were selected as round numbers on the SoTA performance circa 2020 when NLG fluency was human-like: https://paperswithcode.com/sota/text-summarization-on-x-sum. \n\n**Why only the T5-base model is used for the experimental phase**\n\nOur codebase is based on the open-sourced T5X framework. Since we must load different sets of the model’ parameters into acceleration devices for uncertainty approaches like Deep Ensemble, the memory requirement is substantially higher than for the base model. As a result, we decided to use the T5-base, which works smoothly across methods and tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important topic regarding how to make the output of neural language models more reliable and trustworthy.\n2. The problem of miscalibration is understudied in text summarization, meaning that this work could open new research directions toward this topic.", "reasons_to_reject": "1. The experimental setup is poor. The authors compared several probabilistic methods over a single language model (T5-base). They should evaluate multiple language models of different sizes (e.g., BART-base, BART-large, FLAN-T5-XXL). Furthermore, they should include a long document summarization dataset (and thus a long sequence model such as LED or PEGASUS-X) to make the evaluation analysis more robust.\n2. There is not a novel method to address the problem of miscalibration. \n3. The authors state to experiment with large language models (LLMs), but in practice, they use T5-base, which is far from being an LLM (like GPT-3).\n4. The dataset statistics (i.e., number of source/target words and sentences) should be included to show and demonstrate the difference between the benchmarked datasets.\n5. The paper contains many typos (see \"Typos Grammar Style And Presentation Improvements\" for a non-exhaustive list), meaning the document's drafting has not been curated enough.\n6. The authors did not mention how costly each benchmarked method is regarding time and space.", "questions_to_authors": "1. L244: What is \"beam decoding\"? Is it beam search or beam sampling? You should explain it clearly.\n2. L542: \"Sometimes the summary contains information not present in the article.\" Who proved it?\n3. What is the criterion with which you have selected the following values? 40, 15 and 30 (caption of Table 3).\n4. Why did you choose only the T5-base model for the experimental phase?", "paper_topic_and_main_contributions": "The authors proposed a benchmark study about the effectiveness of probabilistic methods in improving the uncertainty quality of the text summarization models, i.e., addressing the problem of miscalibration. \nThey tested different methods over a T5-base model on three summarization datasets, indicating that Spectral-normalized Neural Gaussian Process combined with Deep Ensembles obtains the best results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. L090: \"Large Language Model(LLM)\" -> adding a space\n2. L097: \"ROGUE\" -> \"ROUGE\"\n3. L168: \"5^2 .\" -> put the footnote reference after the dot\n4. L191: \"all , we\" -> \"all, we\"\n5. L202: \"we for the first time benchmark ...\" -> \"For the first time, we benchmark ...\"\n6. L218-219: \"We first study how well different probabilistic methods on summary prediction\" -> incomplete sentence\n7. L221-222: \"summarizaiton\" -> \"summarization\"\n8. L243: \"ROGUE\" -> \"ROGUE\" \n9. L258: \"token^3\" -> put the footnote reference after the comma\n10. Table 1 should be put on top of the page for better positioning."}}, {"review_id": "l4usVikbCH", "reviewer": "Reviewer_Mqji", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer finds our paper interesting and informative. We thank the reviewer for pointing out the typos in the submission. Corrections to the typos will be made in the camera-ready version. We hope the answers we provide here may address your remaining worries.\n\n**The significance and insights from experiment 4.1 are not fully clear**\n\nThe main takeaway from Table 1 is that using a single model probabilistic approach might not necessarily lead to the improvement of the ROUGE scores across every dataset, even if the ECE went down. However in those cases where we observe an improvement it is typically the SNGP method. When it comes to using multiple model probabilistic approaches we observe consistent improvement of ROUGE scores using Deep Ensembles. Deep ensembles are known to be very effective in model calibration and we have demonstrated in our paper that in LLM setting this still holds true. The main conclusion are:\n- Under no time and memory constraints choose Deep Ensembles since they are effective in terms of reducing the ECE and improving the quality of summarization.\n- Calibration methods that are known to be effective in reducing ECE in classification models are not necessarily as effective when adapted to LLMs.\n\nWe promise to make these takeaways more clear in the camera-ready version and we thank the reviewer for asking this question.\n\n**Novelty given the advancements in probabilistic methods for classification tasks.**\n\nWe agree with the reviewer's observation that our contribution does not lie in introducing novel probabilistic methods. Nevertheless, it's crucial to emphasize that the primary novelty of this paper is to adapt the probabilistic calibration methods from classification to the LLM setting with autoregressive generation and conduct a large benchmarking of the variety of probabilistic calibration methods across different datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well written paper, and interesting and informative to read\n- Investigation of state-of-the-art probabilistic methods in improving uncertainty calibration in neural summarization models.\n-  Clear demonstration of the positive impact of probabilistic methods on summary quality and calibration performance.\n- In-depth analysis of failure patterns in widely-adopted probabilistic methods, providing insights for choosing appropriate methods based on data settings.\n", "reasons_to_reject": "- The significance and insights from experiment 4.1 are not fully clear, requiring more detailed explanations.\n- Although the paper provides good experiments and analysis, it is not entirely novel given the advancements in probabilistic methods for classification tasks.\n", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The paper addresses the challenge of miscalibrated predictive uncertainty in summarization models, which can compromise their reliability in real-world applications. To rectify this issue, the authors perform a study of these methods, analyzing their impact on both uncertainty and prediction aspects of model performance. Additionally, the authors introduce a evaluation protocol tailored to measure uncertainty calibration performance, utilizing domain-specific quality scores like ROGUE. The contributions of the paper include - the adaptation of probabilistic methods to the LLM setup, the evaluation protocol, and the analysis in the experiments. The paper also delves into the identification of failure patterns in well-established probabilistic methods such as Deep Ensembles and Monte Carlo Dropout.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 202\n218-220\nsummarization spelling in 222\nstandard spelling in figure 2 caption"}, "extra_text": {}}, {"review_id": "CmTZFRIeMS", "reviewer": "Reviewer_pqaw", "summary": "", "questions": "1. What are the definitions of all the terminologies?\n2. How are the sequence-level and token-level ECE calculated? Please justify with concrete examples.\n3. What is the meaning of the quality in Section 4.3?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer liked our paper. We hope that the explanations we provided below will help to answer the questions that you have.\n\n**Definitions of the terminologies at the end of Section 3**\n\nIn the formula $u(y|x) := \\frac{1}{T} \\sum_{t=1}^T \\bar{p}(y_t|y_{<t},x)$, $x$ is the input sequence, $y$ is the output sequence, $y_t$ is the $t$-th token of $y$, $T$ is the length of the sequence $y$, i.e. the number of tokens in $y$. We will clarify those terminologies in the camera-ready version. \n\n**How sequence-level ECE is calculated**\n\nThe sequence-level ECE is computed by casting the problem into a binary classification task. If the prediction is “i made fun of a guy walking in giant slippers on father’s day” and the ground truth is “thought a guy’s shoes were the reason he walked funny, turns out he has cerebral palsy.”, we say that the prediction is wrong. The probability $\\hat{p}$ of a predicted sequence is the production of probabilities of all tokens of that sequence.\n\n**How token-level ECE is calculated**\n\nTokens make up the space of predictions in token-level ECE. For each input data, we pad or truncate the target sequence such that it has the same length as the predicted sequence. We then say that a predicted token is correct if it matches the target token at the same position in the target sequence. Wang, Shuo, et al. 2020 uses translation error rate to relax the condition that the tokens under consideration need to be at the same position. We will clarify this distinction in the final version.\n\nWang, S., Tu, Z., Shi, S., & Liu, Y. (2020, July). On the Inference Calibration of Neural Machine Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 3070-3079).\n\n**Clarifying the meaning of the quality in the y-axis of Figure 1**\n\nThis is a great suggestion! Yes, the quality in the figure is the ROUGE-1 score. We agree that using 'ROUGE score vs Abstention' is clearer. We will make the change accordingly and move the ROUGE-1 title to the x-axis of that figure. We also included ROUGE-2 and ROUGE-L results in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- In-depth Analysis: The paper provides a comprehensive comparison of various calibration techniques, including Monte Carlo Dropout, Deep Ensembles, Batch Ensemble, and Spectral-normalized Neural Gaussian Process (SNGP), and adapts these probabilistic methods to LLM setup for summarization task. This detailed analysis includes both generation quality assessment for calibrated model and quality of its uncertainty (length-normalized log-probabilities), which would be invaluable for future research in the field of NLP.\n- Clear Presentation: The overall logic flow of the paper is fluent and easy to follow. Both figures and tables are nicely drawn with subtle explanations.", "reasons_to_reject": "Overall, I like this paper. My main concern is that some details seems missing and need further explanations:\n- For the main method (section 3), formal definitions (preliminaries) are necessary before showing equations. For example in Line 186-200, what are the meanings of x, y, t, T? Is u(y|x) a vector or a number (I expect it to be a vector because it is a distribution among the vocabulary space)? Otherwise, readers may find it hard to follow the theoretical part.\n- For the evaluation (section 4.2), how is the sequence-level ECE calculated? For example, in line 256-257, if the predicted sequence is “i made fun of a guy walking in giant slippers on father’s day” and the ground truth is “thought a guy’s shoes were the reason he walked funny, turns out he has cerebral palsy.”, does y^ equal y and what is p^? Lack of concrete examples makes it difficult to understand how the accuracy of a prediction (sequence) and the confidence are calculated.\n- For the abstention (section 4.3), what is the meaning of the quality in the y-axis of Figure 1? If it is ROUGE, then make it 'ROUGE score vs Abstention', instead of introducing an undefined new term.\n", "questions_to_authors": "1. What are the definitions of all the terminologies?\n2. How are the sequence-level and token-level ECE calculated? Please justify with concrete examples.\n3. What is the meaning of the quality in Section 4.3?\n", "paper_topic_and_main_contributions": "This paper seeks to improve the calibration of language models, specifically focusing on abstractive text summarization tasks. It explores several calibration methods, including Monte Carlo Dropout, Deep Ensembles, Batch Ensemble, and Spectral-normalized Neural Gaussian Process (SNGP), analyzing their ROUGE performance on three public datasets and sequence-level ECE of their calibrated uncertainty. The paper also delves into the failure patterns of probabilistic methods in NLP community to unveil the significance of selecting appropriate methods based on task setting.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors studied the problem of uncertainty calibration on the summaries from PLM. This is an interesting problem to NLP community. 
The authors investigate different probabilistic methods in addressing the problem. The experiments were on T5-base and results showed that these methods can improve the uncertainty quality. 
A weakness of this paper is that evaluation was carried out on T5-base. It is not an LLM in light of the current progress. Nevertheless, the results on T5-base look thorough.
Decision: Accept-Findings