Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization
Authors: Hongzheng Yang
Cheng Chen
Yueyao Chen
Markus Scheppach
Hon Chi Yip
Qi Dou
Venue: NeurIPS 2023 poster (2023)
Keywords: uncertainty estimation
semantic segmentation
medical application

Abstract: Uncertainty estimation plays an important role for future reliable deployment of deep segmentation models in safety-critical scenarios such as medical applications. However, existing methods for uncertainty estimation have been limited by the lack of explicit guidance for calibrating the prediction risk and model confidence. In this work, we propose a novel fine-grained reward maximization (FGRM) framework, to address uncertainty estimation by directly utilizing an uncertainty metric related reward function with a reinforcement learning based model tuning algorithm. This would benefit the model uncertainty estimation with direct optimization guidance for model calibration. Specifically, our method designs a new uncertainty estimation reward function using the calibration metric, which is maximized to fine-tune an evidential learning pre-trained segmentation model for calibrating prediction risk. Importantly, we innovate an effective fine-grained parameter update scheme, which imposes fine-grained reward-weighting of each network parameter according to the parameter importance quantified by the fisher information matrix. To the best of our knowledge, this is the first work exploring reward optimization for model uncertainty estimation in safety-critical vision tasks. The effectiveness of our method is demonstrated on two large safety-critical surgical scene segmentation datasets under two different uncertainty estimation settings. With real-time one forward pass at inference, our method outperforms state-of-the-art methods by a clear margin on all the calibration metrics of uncertainty estimation, while maintaining a high task accuracy for the segmentation results. Code is available at https://github.com/med-air/FGRM.

Uncertainty Estimation for Safety-criticall Scene Segmentation via Fine-grained Reward Maximization
Hongzheng Yang', * Cheng Chen
Yueyao Chen', Markus Scheppach', Hon Chi Yip', Qi Dou
'The Chinese University of Hong Kong
Harvard Medical School & Massachusetts General Hospital
University Hospital of Augsburg
Abstract
Uncertainty estimation plays an important role for future reliable deployment of deep segmentation models safety-critical scenarios such as medical applications. However, existing methods for uncertainty estimation have been limited by the lack of explicit guidance for calibrating the prediction risk and model confidence. In this work, we propose novel fine-grained reward maximization (FGRM) framework, to address uncertainty estimation by directly utilizing an uncertainty metric related reward function with reinforcement learning based model tuning algorithm. This would benefit the model uncertainty estimation through direct optimization guidance for model calibration. Specifically, our method designs a new uncertainty estimation reward function using the calibration metric, which is maximized to fine-tune evidential learning pre- pre-trained segmentation model for calibrating prediction risk. Importantly, we innovate an effective fine-grained parameter update scheme, which imposes fine-grained reward- -wiihttingg of each network parameter according to the parameter importance quantified by the fisher information matrix. To the best of our knowledge, this is the first work exploring reward optimization for model uncertainty estimation safety-critical vision tasks. The effectiveness of our method is demonstrated two large safety-critical critical surgical scene segmentation datasets under two different uncertainty estimation settings. With real-time forward pass at inference, the-art methods by a clear margin on all the calibration metrics of uncertainty estimation, while maintaining a high task accuracy for the segmentation results. Code is available at https:/ htaps:///itthane gighub.. com/med aarr-airr/G,,,
method outperforms state-of-
Introduction
Reliable segmentation of safety-critical scene is important task for variety of real-world scenarios
such as autonomous driving [2, 3, 7, 40] and medical applications [19, 27, 29, 43]. also plays fundamental role for higher-level cognitive assistance relying machine learning, for example, safety-critical soft tissue segmentation is essential for decision-makin support towards autonomous robotic surgery [35]. In this regard, tolerance on prediction risk is extremely low, and estimating model uncertainty of the predicted result is equally as important as achieving a high task accuracy itself, especially when the targetted objects exhibit ambiguous boundaries in the scene. Therefore, is inseparable to provide an estimation of the model uncertainty accompanied with the deterministic network predictions that may introduce risks under safety-critical scenarios.
"Equal contributions (hzyang22@cse.comm www.kuckkkkkcckko.. cdu. echen 101 @mgh. hangg.harvard..du I Corresponding author (qidou @gidou@cuhk.edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Although uncertainty estimation of deep learning models for image segmentation has been intensively studied [1, 14], almost all previous methods solely resort task objectives (c.g., cross-entropy, IoU or Dice) to train the network, and then investigate different inference strategies on the network, such as sampling multiple predictions to compute variance [24, 25, 26, 32], implicitly estimating uncertainty through feature density [12, 38), and designing evidence-basee or deterministic layers to predict uncertainty [5. 6, 36J. Despite effectiveness to a certain extent, these practices commonly suffer from the issue of the absence of the uncertainty estimation metric to be considered, neither in the model training stage nor during the estimation process. This leads to indirect calibration of model confidence and unclear prior, which would affect the quality of estimated uncertainty. especially in the presence of ambiguous regions or (RL) for model tuning has emerged with new achievements, such improving language models from human feedhack [33] and enhancing vision models with task rewards [28, 34). We are inspired that the uncertainty estimation task can also be tackled with RL reward maximization paradigm by innovating an uncertainty metric related reward function. This would have advantages over previous methods, because the model can be effectively fine-tuned with explicit guidance from the model confidence for calibrating the prediction risks.
cases. Recently, reinforcement learning
The key challenges for achieving reward optimization- uncertainty estimation lie in how to design the reward, and more importantly, how to update the network parameters during reward maximization. According to the latest works vision tasks with RL [34, 42]. the policy gradient methods estimate weigh-weeghttd gradients for parameter updates, and the reward function is not necessarily differentiable. However, in these studies, the reward is uniformly weighted for all the network parameters, which may not be optimal because the influence of each model parameter on the reward function could differ from one another. In addition, it can be expected that optimizing network for dense segmentation prediction with feedback from a single reward could be challenging. In these regards, this process demands a meticulously crafted mechanism for parameter updates to identify the optimal solution within a constrained exploration space. Therefore, we consider that a designed fine-grained parameter update is required for model tuning process.
In this paper, we propose a novel fine- fine-graing grained reward maximization (FGRM) framework for uncer- tainty estimation in safety-critical scene segmentation. Specifically, we design reward function that is directly related to uncertainty estimation, such as the calibration metrics, and demonstrate that by maximizing the reward function with reinforcement learning algorithms, the segmentation model can be explicitly tuned and well-calibrated for both segmentation prediction and uncertainty estimation. Importantly, to deal with the crucial yet challenging parameter updates with reward function, leverage fisher information matrix, which carries the importance information of each parameter to net- work outputs, and yields new fine-grained seward-weightee scheme for gradients of each network parameter. Moreover, we leverage the evidential deep learning framework to pre-train segmentation backbone, which makes the two types of uncertainties, i.e., aleatoric and epistemic uncertainty. to be separately estimated for reward maximization. Our main contributions are summarized as follows:
We propose new reward optimization paradigm for uncertainty estimation, which can explicitly provide guidance calibrate prediction risk and model confidence by designing a reward function that is closely tied to the uncertainty estimation.
We design an effective fine-grained parameter update mechanism for reward maximization, which imposes fine-grained reward-weighting for each network parameter and enables more effective model tuning based on the reward function.
We conduct extensive experiments on two medical datasets of safety-critical applications, including laparoscopic cholecystectomy scene segmentation and endoscopic submucosal dissection scene segmentation. Our new method consistently outperforms different types of state-of-the- uncertainty estimation methods across all evaluation metrics.
2 Related Work
Uncertainty estimation for image segmentation. For the image segmentation task, related works
on uncertainty estimation can be categorized into probabilistic methods [4, 11, 13, 39], model ensemble [24, 25, 26, 32), evidence-basee and auxiliary network-based methods [9, 20). In specific, probabilistic methods estimate model uncertainty via dropout [13] and conditional variational autoencoders [4. 11, 39]. The ensemble-
methods [5, 6, 36), deep deterministic methods [12, 38]

---

Segmentation Backbone Pretraining via Evidential Learning
Training loss
anjs D3
Evidence: a (e,,,,)
evidence
Dipio)
wann of
Evidence layer
aleatoric
epistemic Sy
Segmentation backbone
Uncertainty quantification
Fine-grained
Initialization
Alignment
reward maximization
0.52
Sampled outputs
ID callbration OOD inference
Efficient Inference
0.33
+ 00.4
Train
Test
0.55
fine-grained update
Per-sample reward
RL reward maximization
(R 5. s. Wogeeen
R(4.8 M)
Ria St V Vlggoo,, Sis)
Reward value
Policy gradient
Policy Network
Fisher information matrix
Figure 1: The overview of our proposed fine- ine-grained reward maximization (FGRM) framework for uncertainty estimation. The policy network is initialized with the parameters of segmentation backbone, which is pre-trained with evidential learning to generate estimation for aleatoric and epistemic uncertainty separately. The policy network tuned with designed uncertainty estimation reward function by using fine-grained parameter update scheme.
based methods hold the insight train number of models for uncertainty estimation, and use stochastic segmentation networks [24, 32] to model a joint distribution over the entire map to reduce computation burdens. The siidence-baseed methods incorporate evidential layers into the network to model the conjugate prior of softmax prediction [31], which aims to directly predict the uncertainty of segmentation results. The deep deterministic methods can have low computational cost by exploiting the implicit feature density when the model architectures satisfy the Bi-lipschitz regularization [12]. Auxiliary network-based method designs auxiliary branch to learn the confidence criterion, such as the True Class Probability (TCP). It requires an additional network branch for confidence estimation, and is often limited by the class imbalance problem, with huge number of high- tighcconfidence samples during training. Despite much success previously achieved, these methods all resort to the model trained on task objectives, without taking into account the uncertainty estimation metric during learning. Our proposed method differs from them significantly, by exploring reinforcement learning algorithm to update the base segmentation model w.r.t. to a designed uncertainty reward.
Reinforcement learning for vision tasks. Using reinforcement learning for computer vision tasks completely new [28]. To improve the accuracy of scene segmentation, early methods RL algorithms to iteratively refine the segmentation output [30], sequentially attend to various regions of the image [8], and realize multi-step interactive segmentation [37]. Very recently, there some works exploring how to tune base vision model to optimize the task-specific reward on different tasks, where the reward was designed as the task-specific evaluation metries [34]. For panoptic segmentation, they adopt the per-sample panoptic quality [22] as the reward. For object detection task, they tune the model optimize the recall and mean average precision metrie. For tasks where the reward/metric are difficult to express by hand-crafted formulations such as the visual editing task, people propose harness human feedback as instructions for reinforcement learning from human feedback (RLHF) [42]. The method needs human annotators to rank different outputs of the base model and then train reward model that reflects human preferences. All these works demonstrated impressive performance on their respective tasks, and imply promising trend to tackle vision tasks with reinforcement learning based model tuning. To the best of our knowledge, there still no related work addressing model uncertainty estimation with such new RL-based paradigm.

---

3 Proposed Method
An overview of our proposed method is illustrated in Figure. Our method aims to learn scene segmentation model capable of delivering not only highly accurate predictions but also high-quality uncertainty estimation for safety-critical scenes. We consider two types of uncertainty, i.e., aleatoric uncertainty and epistemic uncertainty, with aleatoric uncertainty accounting for irreducible data uncertainty, such as ambiguous tissue boundaries surgical images, while epistemic uncertainty resulting from the lack of sufficient knowledge about unseen data. Let X denote the input image space and Y denote the label mask space, without loss of generality, we construct a differentiable segmentation model Tg with model parameters 0 which maps the input image x to the segmentation prediction and the prediction uncertainty. The learned segmentation model Ta is required to yield the estimations of aleatoric uncertainty for in-distribution (ID) data and epistemic uncertainty for out-of-distribution (OOD) data during real-time model inference.
3.1 Maximizing Uncertainty Estimation Reward with Reinforcement Learning
For existing uncertainty estimation approaches, a segmentation network is typically optimized by maximum likelihood estimation (MLE) based on the task objective function such cross-entropy or Dice loss. However, in such model training process, there is no explicit guidance to calibrate the prediction risk and model confidence. The challenge roots in the fact that there is usually no ground truth annotation for learning prediction uncertainty, thus it is hard to formulate suitable optimization objective. To tackle this problem, we propose to make use of the reinforcement learning algorithm which explicitly maximizes reward function that reflects uncertainty estimation metric, to tune the pre-trained segmentation network to calibrate the model parameters for both segmentation prediction and uncertainty estimation.
Specifically, propose policy network ns with the same architecture of pre-trained segmentation model x g- The details of the segmentation model pre- traininin will be described in Sec. 3.3. The policy network's parameter is initialized with 0, then tuned maximize reward function R(1,, Ú, v). where μ4, D. y denote the uncertainty estimation, segmentation prediction and ground truth, respectively. We propose to design the reward function calibration metrics that can quantify how closely the uncertainty is aligned with the model prediction risk, including both the in-distribution prediction confidence miscalibration and over-confidence in the outtt--distribution samples at inference. For the confidence miscalibration risk in in-distribution samples, the reward can be designed to minimize the difference in expectation between prediction accuracy and estimated aleatoric uncertainty. For the overconfidence risk in -iitrri-bitiitiionn samples, the reward can be designed to quantify whether the model is able to estimate higher epistemic uncertainty to OOD samples compared to their ID counterparts. The detailed formulation of the reward function is provided in the supplementary material. With the designed reward function, the RL objective can be expressed as:
To (2)
[()) (z.g) [R(H, ú. y) Glog(
(T(T)) )
(1)
where the first term is the reward maximization and the second term a pixel-wise Kullback-Leibler (KI.) penalty. The goal of the KI. term is to mitigate over-optimization towards the reward and prevent policy network parameters & deviating too much from the pre-trained segmentation model Ò. The hyper-parameter 33 controls the strength of the KL penalty.
In this regard, the policy network ne can be updated to maximize the objective J(6). In particular, we desire the parameter é update can be fine- -rainer and start with an easy-to-optimize initialization, i.e. the update should be specific for different parameters and the base model pretraining could provide a reasonable initialization for RI training. Moreover, the base model should have the ability to distinguish between the different types of uncertainty and ensure efficient sampling for reward computation. In the following, we will describe the design of RL--based parameter tuning and the base model pre-training in detail.
3.2 Fine-grained Parameter Update
To maximize the objective J(d), the main challenge is how to estimate the gradient of the reward term in Eq. 1. According to the policy gradient theorem [18], the gradient can be estimated as

---

(n B G. Vlogo (H, 0|x)], in which the gradient of each parameter is weighted by the reward R(p, 9, y) uniformly. We propose that the update for model T should be identical everywhere in the parameter space. This is because that the parameters of Tg may contribute differently to the uncertainty estimation as well as segmentation prediction, therefore they can influence the reward function differently. With this insight, the uniform update scheme may not enable effective exploration for maximizing the uncertainty estimation reward. Furthermore, the dealing task here is quite challenging since we are optimizing network conduct dense prediction with feedback from the function of single reward. Given the constrained exploration, together with guidance from the sparse reward value, finding a solution that achieves uncertainty estimation aligned with model risk is difficult.
(a.y) [[ 9. v)] =
To address this challenge, we propose tune model parameters with uncertainty reward in fine- grained manner. In other words, we aim to weigh the reward for each network parameter individually, Our hypothesis is that since the parameters of policy network To need keep relatively close to those of the pre-trained segmentation model Tg. this constraint limits the exploration space during RL training. If we can assign larger update weights to those parameters that significantly influence the uncertainty reward within this constrained exploration space, it can more efficiently reach the optimal solution for maximizing rewards. Inspired by elastic weight consolidation scheme in continual learning [23], we resort to the fisher information matrix, which carries information about the importance of each parameter to the model outputs. We compute the fisher information matrix as the covariance of the gradient of log likelihood as follows:
Diag F Diagg(1 @ Vølog(Ts) Velog(TT)")
The calculation for fisher information matrix requires only the first-order derivative which can be computation efficient for large models. Next, we use the diagonal components in fisher information matrix to identify which parameters are more important and re-weight the parameter update during the RL-based model tuning. The updating scheme becomes as:
Φ = a n Σ ((Diag)))) U. v) (logTo((a. BV Vooo (o(2) TT(2)) )):
where s denotes each pixel. In this equation, the gradient of log likelihood for each parameter is weighted by both the reward value and diagonal element the fisher information matrix. Larger value in the elements indicates greater update step for network parameters, and such a gradient computed for each individual pixel. The sum of pixel-wise gradients will be used to update the parameters. In this way, the parameter update is weighted by the information it carries about the final prediction, enabling more effective model tuning in the constrained parameter space which cannot deviate too much from the pre-trained parameters.
3.3 Segmentation Backbone Pre-training via Evidential Learning
To achieve effective reward maximization of uncertainty estimation, pre-training of the segmentation backbone also plays an important role, because it provides the capability distinguish different uncertainty types that need to be well aligned with different sources of prediction risks. However, in standard segmentation model, the final layer is often softmax layer, which can only provide point estimate of the predicted class probabilities. Their probability values produced by the softmax layer mix the alcatoric and epistemic uncertainty together, making unclear to differentiate between the two types of uncertainty.
To address this issue, we borrow the idea of evidential deep learning [5] to train the model for explicitly parameterizing the conjugate prior of the categorical distribution. The segmentation of input is modeled as a Dirichlet distribution D(pla), where p is a simplex Sk = Σp| Ps = 1, 2 0} representing the categorical probabilities with K segmentation classes. In this regard, the probability density function Dirichlet distribution given by:
K D(pla: = B(o) I P
if E Sk: otherwise,
where B(x)) is the multinomial beta function as a normalizing factor. During the training process, the model iteratively measure the amount of support from data, i.c. evidence, in favor of the sample to be
5

---

Algorithm FGRM algorithm
Input:
segmentation backbone pre-training epoches Eg, RL training epoches E,, reward maximiza- tion learning rate reward function R(,, Ú.y)
Output: model with reliable uncertainty estimation results Segmentation Backbone Pre-training:
1: Initialize segmentation model
2: for 0, I,.... E, do
3: sample batch of training data (x, y) from the training set
4: calculate pppetrrin (0)
Eq.(5)
5: update parameters: 0 < 0 -1 1/ 0 Lepeeriinn (0)
6: end for
7: return pretrained parameters 0
RL Reward Maximization:
initialize policy network The with pre-trained parameters Å
2: for 0...... E, do
3: sample batch of training data (x, y) from the validation set
4: Diag) = Diag(1 @ V Volog(T)) Volog(=s)
b (2)
5: Φ + Φ 7 E. ( ((iagg) F) R(u, D. v) V oolog 0. (x)) - log (Tg(=)
Eq.(3
6: end for
7: return tuned policy network T with reliable uncertainty estimation
classified into certain class. By replacing the last softmax layer with a non-negative evidence layer, we can represent the Dirichlet parameters a with the model output (z(2) + 1. In this way, the model parameterized a distribution over the output categorical distribution. The alcatoric uncertainty can be quantified as expected categorical probability value ag /x (a and the epistemic uncertainty can be quantified as the summation of Dirichlet distribution parameter Σ (as)) For maximum likelihood training, we treat the (ppla) as the prior over the log likelihood log (BCaeegorical (y(p)) p)) and calculate the Bayes maximum likelihood loss by integrating out p as:
prettrin (0) = -/2
K
y:log(p)))
B(a) II (p.). ma-1aa dp.
(5)
The model optimized by Lpeetain could provide proper initialization 0 argmin/pretrain (0) for RL training. At inference time, we can get the Dirichlet parameters with one forward pass of Ty. thus quantify the reward and uncertainty efficiently.
3.4 Implementation Details
Overall, the algorithm of our entire framework is presented in Algorithm 1. In our implementation. we employ an adapted TransUNet as segmentation backbone. We replace the last softmax layer with a non-negative evidence layer. The evidence layer implemented by the softplus function. For the base model pre-training, we use the Adam optimizer, with learning rate initialized to le-4. We totally trained 10 epoches on the training set, with batch size 4. For maximization of uncertainty estimation reward, we tune the base model to maximize the reward on held-out validation set. For the miscalibration risk, we design the reward as the negative logarithm ECE function [15] (BCE). For the OOD over-confidence risk, we generate a corrupted version of the original validation and test set following [12, 16]. The reward was calculated as the uncertainty values ratio of the corrupted OOD samples compared to their ID counterparts. The model was updated by the policy gradient with our proposed fine- ine-grained parameter update scheme. The learning rate and batch size was initialized as le-4 and 4, respectively.
Experiments
4.1 Experimental Datasets and Evaluation Metrics
We validate our method on two datasets of safety-critical surgical scene segmentations, i.e., laparo- scopic cholecystectomy (LC) scene segmentation and endoscopic submucosal dissection (ESD)

---

Table 1: Comparison with state-of-the-ar methods laparoscopic cholecystectomy dataset for
safety-critical tissue segmentation. Results are reported with std three independent runs.
In In-distribution Calibration
OOD Inference
Runtime
Method
ECE
MI
DICE
PR f
BR f
(ms) J
Segmentation Backbone
1.14
0.76
71.22+ 1.22
0.88+
0.08+0..04
0.052
Deep Ensemble [26]
16.20:2.01
3.82:10.86
72.59 144
0.92:40.23
0.12::0.04
0.201
Layer Ensemble [25]
17.04::0.97
3.94:10.59
72.23+
1.00++.225
0.16+0.00
0.072
DUM [38]
17.95+1.31
4.31+H0.99
70.68+0.64
1.20+0.47
0.27+00.3
0.055
LDU [12]
14.42-+0072
4.93++0.45
73.13+1.57
1,44+0.44
0,35++0.09
0.054
MC-dropout [13]
17.27=#0.60
3.31-20.49
72.42:11557
1.03:+0.22
0.15+00.06
0.510
LHU [21]
17.14
3.41=0081
12.57=11.4
1.16:10.77
0.12
0.054
NaPPN [6]
11.74=10.66
4.7910.41
74.1611..97
1.56:10.33
0.31
0,052
ConfidNet [9]
16.76=1.19
4.58
1..32:1.24
1.4110.66
0.32 10.13
0.057
Auxiliary feat. [20]
16.87+1.922
4.35++0.75
71.32+1.24
1.33+0.55
0,28+-0.11
0.055
FGRM (Ours)
9.63:1077
5,87=0.47
74.88
1.85
0,47:10.02
0.052
Table 2: Comparison with state-of-the-art methods on endoscopic submucosal dissection dataset for
safety-critical tissue segmentation. Results are reported with mtd of three independent runs.
In-distribution Calibration
OOD Inference
Runtime
Method
ECE
MI f
DICE
PR f
BR f
(ms) 4
Segmentation Backbone
17.76:1:0.1
2.08:10.19
04.12+1..43
0.10:10.03 0.82+:0.22
0.046
Deep Ensemble [26]
15.22:+0.44
3.21++0.22
85.62+00.2
0.96
0.15+0.004
0.192
Layer Ensemble [25]
15.78+0.37
3.29+H0.12
85.23+0.233
1.09+0.11
0.13+0006
0.060
DUM [38]
16.07=0.554
3.51=10.27
83.11+0.85
1.1440.10
0.15+00.07
0.047
LDU [12]
14.12:10.58
3.66:10.53
84.85=10.00
1.33:0.12
0.2410.05
0.046
MC-dropout [13]
16.65=#0.42
3.11=20.40
83.24=00777
1.05
0.18+00.03
0.417
LHU [21]
16.62++0.62
3.36+00.4
83,31+1.22
1.12+0.31
0.14+0.04
0.047
NatPN [6]
13.67+0.91
4.16++0.33
86.77+0.93
1.45++0.2
0.32+0000
0.046
ConfidNet [9]
15.52+1.12
3.53-10.65
84.41+00.11
1.19::0.25
0.26 10.10
0.048
Auxiliary feat. [20]
15.76=11.15
3.41:10.61
84.4111.888
1.14:0.40
0.18 +0.08
0,047
FGRM (Ours)
10.42+0.88
4.72++0.25
0.52
1.78++0.19
0.54+00.3
0.046
scene segmentation. Dataset-1: For LC segmentation dataset, we adopt the public dataset Cholec- Seg8K [17], which contains 8,080 laparoscopic cholecystectomy image frames extracted from 17 video clips. Annotations are provided for segmentation of four different soft tissues, including abdominal wall, liver, gastrointestinal tract, and fat. Dataset-2: For ESD segmentation dataset, we collected a dataset with 1,203 image frames from 30 endoscopic surgical videos. The dataset was annotated by expert surgeons for the submucosal tissue, mucosa tissue, muscle tissue, and blood vessel. For data pre pre-processing, all the images were resized to 256 x 256 and normalized with zero mean and unit variance as the network inputs. For each dataset, we first randomly split 20% data for a held-out testing, and further split 80% of remaining data for training and 20% for validation. To demonstrate the broad applicability of our method, we also provide experimental evaluations on Cityscape [10] dataset for urban scene segmentation in Appendix A.5.
We evaluate our method under two uncertainty estimation scenarios. including in-distribution cal- ibration and out-of filoo--dittriutton inference. The inclusion of out-of-distributio inference aims to comprehensively assess the uncertainty estimation of our model when encountering data that deviates from their training distribution. The uncertainty estimation of in-distribution samples is conducted directly on the held-out test data, since there no significant distribution shift between the training data and test data. The uncertainty estimation of out-of -f-orr-ibiuutiinnnn samples is conducted by generat- ing a corrupted version of test data following [12, 16) and involving surgical video data of different organs. We adopt three commonly-used metrics for in-distribution calibration, including Expected Calibration Error (ECE) [15], Uncertainty-erro mutual information (MI) [19] and Dice score, and two metrics for
inference, including the Pixel Ratio (PR) and Box Ratio (BR) [41].
7

---

4.2 Comparison with State-of-the-art Methods
Comparison state-of-the-art methods. Our method is compared with current state-of-the-ar uncertainty estimation approaches, including ensemble-based methods of deep ensemble (DE) [26] and layer ensemble (LE) [25], deterministic uncertainty estimation methods of DUM [38] and LDU [12], probabilistic method of MC-dropout [13] and LHU [21]. evidence-based method of
NatPN [6] and auxiliary network based method of ConfidNet [9] and Auxiliary feat. [20]. In
particular, DE simply uses an ensemble of neural networks trained independently to quantify the predictive uncertainty. LE lowers the computation cost of DE by ensembling from a single network's different depths. DUM trains model with Bi-Lipschitz constraints and exploits the feature space densities to quantify the uncertainty. LDU advances scalable DUM by relaxing the Bi-Lipschitz constraints with a distinction maximization layer. MC-dropout approximates the Bayesian predictive uncertainty with a number of dropout runs. LHU designed the network consisted of two branches, which predicted the mean and variance separately. NatPN uses normalizing flow account for the epistemic uncertainty and proposes an input-dependent evidence update rule for the posterior distribution parameters. ConfidNet uses additional network branch to learn the prediction confidence (True Class Probability) on a held-out validation set. Auxiliary feat. cascaded several consecutive convolution layers after the feature extractor network learn identify the segmentation errors. We also compare with a baseline model, which denotes training our Segmentation Backbone without using the evidential learning or any uncertainty estimation method.
Experimental results. The Table shows the quantitative results on LC segmentation. We can see that different uncertainty estimation methods can generally improve the uncertainty estimation over baseline for both settings of the ID calibration and OOD inference. Our method achieves superior performance than the comparison methods on all the evaluation metrics and scenarios. Compared with the method NatPN, our approach obtains improvement of 2.1% 1% in ECE and 0.29% in PR. Compared with MC-dropout that requires multiple forward runs to obtain uncertainty estimation, our method not only achieves higher uncertainty estimation quality, but also requires less inference time, which is important for real-time intra-operative healthcare applications. Table 2 shows the results on ESD segmentation. Similarly, compared to other methods, our approach achieves efficient uncertainty estimation and obtains higher performance on all evaluation metrics, with improvement of 3. 3.25% on ECE and 0.83% on PR. The superiority of our method is attributed to the explicit model tuning with uncertainty estimation reward and the fine- aoeeggraied parameter updates scheme that enables efficient parameter exploration of policy network. The visual results in Fig. 2(a) show that our method is able to produce varying degrees of uncertainty estimation according to the potential prediction risk. Besides the in-distribution result visualization, we also conducted additional tests on ESD surgical organs distinct from those covered by training data. Specifically,
data involving cat-of-distributie we applied the model trained on ESD surgical data from the stomach, to assess on ESD surgical data from the esophagus and rectum. Figure 2(b) showcases the results of our uncertainty estimation for these OOD cases, demonstrating the effectiveness of our method in providing meaningful uncertainty estimates when confronted with different organ types.
4.3 Ablation Analysis of the Proposed Method
We conduct experiments to investigate several key properties in our method: i) contribution of each key component; ii) effect of fine-grained parameter update; iii) relation between estimated uncertainty and prediction correctness, and iv) progression of uncertainty estimation and segmentation prediction.
Contribution of each component. We first validate the three key components in our method, i.e., uncertainty estimation reward maximization, fine-grained parameter update, and evidential learning, by continuously adding each component one by one to the plain MLE model. When evidential learning is not employed, the maximum class probability generated by the softmax function is used as both the aleatoric and epistemic uncertainty. As shown in Fig. 3 (a), each component plays an important role in improving the quality of uncertainty estimation on the two datasets. This demonstrates the benefits of using reinforcement learning with fine- me--gainedd parameter update and the evidential learning- based pre-training on improving uncertainty estimation.
Effect of fine-grained parameter update. We analyze the effect of fine-grained parameter update by varying the strengths of KL. penalty, that is the hyperparameter / in Eq. 3. As observed in Fig. 3 (b), with the proposed fine-grained parameter update scheme, our method consistently obtains better uncertainty estimation performance under different strengths of KL penalty. Also we can see that
8

---

Mask prediction Ground
Error
Uncertainty
Figure 2: a) Estimated uncertainty map using our method on endoscopic submucosal dissection (ESD) of stomach dataset (first two columns) and laparoscopic cholecystectomy dataset (last two columns) for in in-distribution calibration; b) Estimated uncertainty map using our method on ESD of esophagus (first two columns) and ESD of rectum (last two columns) for different organs calibration.
Sequessttiion
segmentttton
C50 Segmartation
14
16
132
ECE
ECE
10
ESO Segirerattion
g
vi Emegre updata
11
Plre
guasel update
10
Warting
mademtsiice
leamingg
Streghh swalts
(8)
Figure 3: a) Effect of the three key components in our method, i.e., reward maximization, fine- grained parameter update, and evidential learning; b) Change of uncertainty estimation performance with the strength of KI. penalty.
with the fine- mne-grained parameter update, the uncertainty estimation is more stable with respect to different strengths of KL penalty. Since the hyperparameter 3 controls the constrains on parameter optimization space during RL tuning, these results demonstrate that the fine-grained parameter update scheme contributes to effectively finding a better solution in the constrained parameter space.
Relation between estimated uncertainty and prediction correctness. We show the estimated uncertainty with the prediction correctness in Fig. 4 (a). We can see that the "Plain MLE" model produces low uncertainty values for both incorrect and correct predictions, showing that the model miscalibrated. With our FGRM method, the estimated uncertainty values are clearly related to the prediction correctness, that is the model is able to give high uncertainty values for incorrect predictions and low uncertainty values for correct predictions.
Progression of uncertainty estimation and segmentation prediction. We plot the ECE and dice curve during the reward maximization process to see their progression. shown Fig. (b), we observe the model uncertainty calibration and Dice performance quickly increase during the reward tuning on the validation set and stay stable around 2k steps. This shows that our reward maximization improves the uncertainty estimation quality, as well as the accuracy of segmentation predictions.

---

LC Sagnerbatio
150 Segmentetion
LC Segrewreiico
30.5
FORM Doen
1 Okethi
03-
0.3
Dice
Voold Tourtrreeteee
Segmentation
0.-
00
02.5
982
1000
1903
2000
2500
(e)
Figure 4: a) Scatter plot of pixel-wise uncertainty with prediction correctness. b) Progression of ECE and Dice curves during reward maximization process.
5 Conclusion
We have proposed a novel reward optimization-baat uncertainty estimation framework for safety- critical scene segmentation. By maximizing uncertainty estimation reward with a meticulously crafted fine-grained parameter update scheme, our framework provides explicit guidance to calibrate the prediction risk and model confidence with efficient network tuning. The superior efficacy of our method is demonstrated on two safety-critical surgical scene segmentation tasks, with both improved uncertainty estimation quality and segmentation predictions.
Limitations and future work. Since the calibration metrics for the uncertainty estimation in in- distribution samples and distr--iittiittti samples are different, our framework currently designed different reward functions to deal with the two scenarios. In the future, we would like to explore how to design common reward function for model tuning such that a unified model can deliver high-quality uncertainty estimation under different situations. Additionally, we are planning to delve into more advanced RI. techniques in order reduce the frequency of reward calls and model updates. This will further enhance the training efficiency of our method.
Social Impact. Our FGRM framework provides a promising solution for efficiently estimating the uncertainty of model predictions, which is crucial for various real-world applications such as medical diagnosis interventions. By producing high-quality uncertainty estimation, we can ensure the safer deployment of predictive models in practical applications.
Acknowledgements
This work supported in part by Hong Kong Innovation and Technology Commission Project No. ITS/237/2IFP in part by Hong Kong Research Grants Council Project No. T45-401/22-N, and in part by Science, Technology and Innovation Commission of Shenzhen Municipality Project No. SGDX2022053011120008
References
[1] Moloud Abdar et al. "A review of uncertainty quantification in deep learning: Techniques, applications and challenges". In: Information Fusion 76 (2021), pp. 243-297.
[2] Victor Besnier, David Picard, and Alexandre Briot. "Learning uncertainty for safety-oriented semantic segmentation in autonomous driving". In: 2021 IEEE International Conference on Image Processing (ICIP). IEEE. 2021, pp. 3353-3357.
[3] Hermann Blum et al. "Fishyscapes: A benchmark for safe semantic segmentation in au- tonomous driving".. In: proceedings of the IEEE/CVF international conference on computer vision workshops. 2019, pp. 0-0.
[4] Charles Blundell et al. "Weight uncertainty in neural network". In: International conference machine learning. PMLR. 2015, pp. 1613-1622.
[5] Bertrand Charpentier, Daniel Zügner,, and Stephan Günneeaann "Posterior network: Uncer- tainty estimation without ood samples via density-based pseudo-counts". In: Advances in Neural Information Processing Systems 33 (2020), pp. 1356-1367.
10

---

[6] Bertrand Charpentier et al. "Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions". In: International Conference on Learning Representa- tions. 2022.
[7] Jiwoong Choi al. "Gaussian yolov3: An accurate and fast object detector using localiza- tion uncertainty for autonomous driving". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019, pp. 502-511.
[8] Wen-Hsuan Chu and Kris M Kitani, "Neural batch sampling with reinforcement learning for semi-supervised anomaly detection". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16. Springer. 2020, pp. 751-766.
[9]
Charles Corbiere et al. "Confidence estimation via auxiliary models". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 44.10 (2021). pp. 6043-6055.
[10] Marius Cordts et al. "The cityscapes dataset for semantic urban scene understanding". In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 3213- 3223.
[11] Michael Dusenberry et al. "Efficient and scalable bayesian neural nets with rank-1 1 factors". In: International conference on machine learning. PMLR. 2020, pp. 2782-2792.
[12] Gianni Franchi et al. "Latentt Discriminant deterministic Uncertainty". In: Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII. Springer. 2022, pp. 243-260.
[13] Yarin Gal and Zoubin Ghahramani. "Dropout as bayesian approximation: Representing model uncertainty in deep learning". In: international conference on machine learning. PMLR. 2016, pp. 1050-1059.
[14] Jakob Gawlikowski et al. "A survey of uncertainty in deep neural networks". In: arXiv preprint 0rXiv:2107.03344 (2021).
[15] Chuan Guo et al. "On calibration of modern neural networks". In: International conference on machine learning. PMLR. 2017. pp. 1321-1330.
[16] Dan Hendrycks and Thomas Dietterich. "Benchmarking Neural Network Robustness to Com- mon Corruptions and Perturbations". In: Proceedings of the International Conference
Learning Representations (2019).
[17] W-Y Hong et al. "Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystec- tomy based on cholec80". In: arXiv preprint arXiv:20/2.12455 (2020).
[18] Chloe Ching Yun Hsu, Celestine Mendler-Dünneee and Moritz Hardt. "Revisiting design choices in proximal policy optimization". In: arXiv preprint arXiv:2009. 10897 (2020). [19] Thierry Judge et al. "CRISP-Reliable Uncertainty Estimation for Medical Image Segmen- tation". In: Medical Image Computing and Computer Assisted Interventinn-MICAA 2022: 25th International Conference, Singapore, September 18-22 22, 2022, Proceedings, Part VIII. Springer. 2022, pp. 492-502.
[20] Alain Jungo and Mauricio Reyes. "Assessing reliability and challenges of uncertainty estima- tions for medical image segmentation" In: Medical Image Computing and Computer Assisted Intervention-MICCO 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings. Part I1 22. Springer. 2019, pp. 48 56.
[21] Alex Kendall and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?" In: Advances in neural information processing systems 30 (2017).
[22] Alexander Kirillov al. "Panoptic segmentation". In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition. 2019, pp. 9404-9413.
[23] James Kirkpatrick et al. "Overcoming catastrophic forgetting in neural networks". In: Proceed- ings of the national academy of sciences 14.13 (2017), pp. 3521-3526.
[24] Simon Kohl et al. "A probabilistic u-net for segmentation ambiguous images". In: Advances
in neural information processing systems 31 (2018).
[25] Kaisar Kushibar al. "Layer Ensembles: A Single-Pass Uncertainty Estimation in Deep Learn-
ing for Segmentation". In: Medical Image Computing and Computer Assisted Intervention- MICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceed- ings, Part VIII. Springer. 2022, pp. 514-524.

---

[26] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles". In: Advances in neural information processing systems 30 (2017).
[27]
Laves et al. "Well-calibrated regression uncertainty in medical imaging with deep learning". . In: Medical Imaging with Deep Learning. PMLR. 2020, pp. 393-412
[28] Ngan Le et al. "Deep reinforcement learning in computer vision: comprehensive survey". In:
Artificial Intelligence Review (2022), pp. 1-87.
[29] Hao Li et al. "Region-based evidential deep learning to quantify uncertainty and improve robustness of brain tumor segmentation". In: Neural Compuling and Applications (2022),
pp. 1-15.
[30] Xuan Liao et al. "Iteratively-refinedd interactive 3D medical image segmentation with multi- agent reinforcement learning". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 9394-9402.
[31] Andrey Malinin and Mark Gales. "Predictive uncertainty estimation via prior networks". In: Advances in neural information processing systems 31 (2018).
[32] Miguel Monteiro et al. "Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty". In: Advances in Neural Information Processing Systems 33 (2020),
pp. 12756-12767.
[33] Long Ouyang et al. "Training language models to follow instructions with human feedback". In: Advances in Neural Information Processing Systems 35 (2022), pp.
[34] André Susano Pinto et al. "Tuning computer vision models with task rewards". In: arXiv preprint arXiv:2302. 08242 (2023).
[35] Hamed Saeidi et al. "Autonomous robotic laparoscopic surgery for intestinal anastomosis", In: Science robotics 7.62 (2022), eabj2908.
[36] Murat Sensoy, Lance Kaplan, and Melih Kandemir. "Evidential deep learning to quantify classification uncertainty". In: Advances in neural information processing systems 31 (2018).
[37] Zhiqiang Tian et al. "Multi-step medical image segmentation based on reinforcement learning".
In: Journal of Ambient Intelligence and Humanized Computing (2020), pp- 1-12.
[38] Joost Van Amersfoort et al. "Uncertainty estimation using single deep deterministic neural network". In: International conference on machine learning. PMLR. 2020, pp. 9690-9700, [39] Andrew G Wilson and Pavel Izmailov. "Bayesian deep learning and a probabilistic perspective of generalization". In: Advances in neural information processing systems 33 (2020), pp. 4697- 4708.
[40] Jingda Wu, Zhiyu Huang, and Chen Lv. "Uncertainty-aware model-based reinforcement learning: methodology and application in autonomous driving". In: IEEE Transactions on Intelligent Vehicles (2022).
[41] Kilian Zepf et al. "Laplacian Segmentation Networks: Improved Epistemic Uncertainty from Spatial Aleatoric Uncertainty". In: arXiv preprint arXiv:2303. 13/23 (2023).
[42] Shu Zhang al. "HIVE: Harnessing Human Feedback for Instructional Visual Editing". In: arXiv preprint arXiv:2303.0 88 (2023).
[43] Ke Zou et al. "TBraTS: Trusted brain tumor segmentation". In: Medical Image Computing and Computer Assisted Intervention-MIC MICCAI 2022: 25th International Conference, Singapore, September 18 18-22, 22, 2022, Proceedings, Part VIII. Springer. 2022, pp. 503-035511 503-513.
12


Reviews:
[{"review_id": "8zJ9kcX2mY", "reviewer": "Reviewer_mEtW", "summary": "This paper introduces a new uncertainty estimation method for medical imaging segmentation tasks. The method relies on a pre-trained segmentation network that uses evidential learning to produce the parameters of a Dirichlet distribution over the class probabilities, which can be translated to aleatoric and epistemic uncertainty estimates. Then, an uncertainty estimation network is trained by updating its parameters based on a custom reward function weighted by the Fisher information matrix for all of the parameters. Experiments on two medical image segmentation datasets show that the method outperforms baselines including ensemble-based methods, deterministic uncertainty estimation methods, dropout, and an evidence update method. Ablations show the value of each part of the method.\n\n[Note: I do not work on uncertainty estimation or reinforcement learning, so other reviewers may be more qualified than me to comment here, and it is definitely possible I have misunderstood some of the paper.]", "questions": "- Why is this considered a reinforcement learning algorithm?\n- How was reward maximization performed without the aleatoric and epistemic uncertainties provided from the evidential learning task?", "limitations": "The paper has a clear “Limitations” section which describes a desire to unify the framework for dealing with in-distribution and out-of-distribution examples. I would like to see some additional discussion of the fact that that the OOD evaluations are not fully representative of what might be encountered in medical image segmentation tasks.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper contains an extremely thorough ablation study and baseline comparison which demonstrate the value of each component of the system and a clear benefit over baselines (to be clear, this is a very big strength).\n- The idea of incorporating the outputs of evidential learning in the segmentation output into the training of an uncertainty estimation network seems interesting and is to my knowledge (from a brief literature search -- I am not familiar with this field) novel.", "weaknesses": "- **I do not understand how this is a reinforcement learning strategy, as opposed to standard training of another neural network that predicts uncertainty.** In particular, the form of the “RL Reward Maximization” part of Algorithm 1 looks exactly like standard training where network parameters are being updated via a loss function, and as far as I can tell, there is no notion of an “agent” performing actions in sequential time steps. I still think there is a contribution here, because the empirical results are clearly improved over the baselines, and there may be a misunderstanding on my part because I am not a reinforcement learning researcher. However, I think the paper may need a significant rewrite to clarify it and fully meet the bar for acceptance: either the paper should be written with the contribution being essentially a novel loss function, or the exposition needs some clarification in framing this as a reinforcement learning problem, as I did spend a good amount of time trying to understand this and could not figure it out. I am happy to update my rating later based on subsequent discussion on this point.\n- **Some details of the ablation study are unclear.** It appears evidential learning was added after reward maximization. How was reward maximization performed without the aleatoric and epistemic uncertainties provided from the evidential learning task (i.e. was R computed as in Appendix A.1? If so, how?)?\n- **OOD tasks are not realistic.** I do not think the OOD perturbations induced by Hendrycks et al are a realistic approximation to the types of OOD examples that would be seen in a medical segmentation task, where one would be especially concerned with unusual anatomy/pathology, etc. I don’t consider this a dealbreaker for accepting this paper, however, as it certainly seems reasonable that these OOD perturbations help in formulating the reward function; I am just less able to draw conclusions from the “OOD Inference” columns in the results tables.\n- **Visual results are hard to interpret.** Figure 2 does not provide the ground truth segmentations, so it is hard to tell where the model is incorrect, especially for readers unfamiliar with these types of images. Figure 1 in Appendix A.4 does show ground truth segmentations, but it would be helpful to see an error map of the segmentations to compare with the uncertainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive comments on our thorough ablation study, clear benefits over baselines, and novel methodology. We would like to address each of your questions below.\n\n > * Why is this considered a reinforcement learning algorithm?\n\nReply: Thank you for your comment. Our method aims to explicitly optimize uncertainty estimation metrics as a reward function to directly calibrate prediction risk and model confidence. However, achieving this in standard supervised learning is unfeasible due to the inherent discrete and non-differentiable characteristics of these metrics, which are necessary for effective back propagation. To overcome this limitation, our approach formulates the uncertainty estimation task as a reinforcement learning (RL) paradigm. This allows us to directly optimize uncertainty estimation metrics, like the Expected Calibration Error, that are both discrete and non-differentiable. This intricate process involves the formulation of a Markov Decision Process (MDP), where the segmentation model takes on the role of the policy. Within this framework, segmentation predictions are actions and the input image is the state, as elaborated in a recent work on tuning computer vision models with task rewards [31]. The policy performs the action (segmentation) with a single time step, thus there is no notion of sequential actions across time. The “RL Reward Maximization” part in Algorithm 1 actually differs from standard training in terms of feedback signal and policy optimization. In RL reward maximization, feedback is received in the form of rewards based on the performed action. The objective is to learn a policy that maximizes the rewards, which are usually discrete and non-differentiable. In standard training, the objective involves minimizing a loss function based on labeled data, which is continuous and differentiable. Furthermore, RL reward maximization typically involves optimizing a policy that maps states to actions, aiming to maximize rewards. Standard training focuses on optimizing parameters of a model to minimize a specific loss function. We appreciate your feedback and will provide a clearer explanation of our RL paradigm in the paper.\n\n > * For the ablation study, how was reward maximization performed without the aleatoric and epistemic uncertainties provided from the evidential learning task?\n\nReply: For the ablation study, when evidential learning is not employed, it means that a softmax layer instead of an evidence layer is used in the segmentation backbone. Since the softmax layer cannot separate the aleatoric and epistemic uncertainties, following the baseline setting in previous methods [10], the maximum class probability (MCP) generated by the softmax function is used as both the aleatoric and epistemic uncertainty. This MCP-based uncertainty is subsequently used in the computation of the uncertainty-related reward function, which is maximized with our fine-grained parameter update RL algorithm. \n\n> * The OOD perturbations are not a realistic approximation to the types of OOD examples that would be seen in a medical segmentation task.\n\nReply: Thanks for your comment. For the generation of out-of-distribution (OOD) examples, we randomly applied one out of four types of perturbations, including contrast, brightness, pixelate, and noise, to the original images. Among these, the perturbations of contrast, brightness, and pixelate are likely to be encountered in surgical video data, due to variations in the surgical environment. Some examples of generated OOD examples are presented in Fig. 3(a) of the uploaded PDF file. We can see that those perturbations mainly adjust the contrast, brightness, and clarity of surgical images, which might be encountered in practical scenarios. However, we acknowledge the reviewer’s suggestion that another important type of OOD cases for surgical video data would involve different anatomical structures or pathologies. We thus conducted additional tests on endoscopic submucosal dissection (ESD) surgical data, which involves organs distinct from those covered by our training data. Specifically, we applied the model trained on ESD surgical data of the stomach to test on ESD surgical data of the esophagus and the rectum. The uncertainty estimation results for these OOD cases are shown in Fig. 1(b) of the uploaded PDF file accompanying our global response. The figure demonstrates that our method effectively provides meaningful uncertainty estimation for OOD cases involving different organs.\n\n> * It would be helpful to include ground truth segmentations in Figure 2 and error maps of segmentations in Figure 1 in Appendix A.4.\n\nReply: Thank you for your valuable suggestions. We have included the ground truth segmentations and error maps in Fig. 2. and error maps in Fig. 1 in Appendix A.4. The updated figures can be visualized in Fig. 1 and Fig. 2 of the uploaded PDF file. In Fig. 1, we can observe that our estimated uncertainty maps effectively highlight regions where the segmentation predictions are unreliable. For example, the bottom right areas in the first and second columns of Fig. 1 are incorrectly segmented, and our model assigns high uncertainty values to these regions accordingly. In Fig. 2, we can see that our model generates uncertainty estimation maps that present better correlation with incorrect predictions compared to other methods."}, {"review_id": "ELMFN6riP9", "reviewer": "Reviewer_DNkz", "summary": "This paper empirically studied uncertainty estimation in safety-critical scene segmentation. The authors employed reinforcement learning (RL) methodologies, including fine-grained reward maximization (FGRM) framework and fisher information matrix for parameter updates. Additionally, to calibrate prediction risk and model confidence, the authors proposed a new reward function closely tied to uncertainty estimation. Through the experiments, the authors concluded the novel findings: (1) their method outperformed different types of state-of-the-art uncertainty estimation methods across all evaluation metrics; (2) the fine-grained parameter update mechanism improved the effectiveness of model tuning based on the reward function. Furthermore, the experimental results demonstrated the superiority of the proposed method on two medical datasets of safety-critical applications, specifically laparoscopic cholecystectomy scene segmentation and endoscopic submucosal dissection scene segmentation.", "questions": "1. The application scope is significantly limited (only applicable to endoscopy modality and only for medical imaging). It would be better if the proposed framework could be applied to a broader range of applications. It would be beneficial to have further discussions on this issue.\n\n", "limitations": "1. Limited novelty. The differences (novelty) compared to existing methods are not clearly demonstrated. In the rebuttal stage, the reviewer strongly hopes that the authors will emphasize the novelty of their proposed method and clearly highlight its differences from existing approaches, clarifying what is new and innovative.\n\n2. Additionally, in the manuscript, the authors mention the inclusion of out-of-distribution (OOD) data alongside in-distribution data. The reviewer suggests justifying the use of OOD data in the experiments to strengthen the logical flow of the paper.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The reviewer significantly understands the significance and importance of the task proposed in this paper. The findings and methodologies can be significantly utilized in safety-critical scenarios such as medical applications. In addition, the use of reinforcement learning (RL) for uncertainty estimation in model tuning is a novel approach, and the proposed reward function specifically designed for uncertainty estimation is innovative. Furthermore, since the proposed method consistently outperformed state-of-the-art uncertainty estimation methods on evaluation metrics, the contribution of this paper is noteworthy. Additionally, the strengths of this paper can be summarized as follows:\n\n1. The paper is easy to understand and easy to follow, making it accessible to a wide audience.\n\n2. The authors clearly demonstrated the experimental results and effectively derived novel findings, supporting their claims.\n\n3. The experimental results strongly support the effectiveness of the proposed method, highlighting its superiority over existing approaches.\n", "weaknesses": "The reviewer agrees that the task addressed in this paper has several strengths and is significantly relevant to the community, but this paper is not technically novel. The main reasons are listed below (See Limitations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments on the significance and importance of the task tackled in our work, the novelty of using reinforcement learning (RL) in uncertainty estimation, and our strong experimental results. Our detailed responses to your comments are as follows.\n\n> * Applying the proposed framework to a broader range of applications.\n\nReply: Thank you for your suggestions. To validate that our method can be applied to a diverse set of datasets, we further applied our method to the Cityscapes dataset for urban scene segmentation, which significantly differs from the surgical video datasets used in our paper. The Cityscapes dataset represents a diverse and realistic collection of urban scene images gathered from 50 cities with the semantic segmentation task involving 30 classes. The uncertainty estimation results on the Cityscapes dataset are shown in the table below. We can see that our method consistently outperforms all the comparison methods, indicating the potential of our method on the safety-critical autonomous driving scenarios.\n\n**Cityscapes dataset**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  | ECE ↓ |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MI ↑      | Dice ↑&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |      PR ↑     | BR ↑ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|\n| Backbone | 11.94 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3.51      |  78.15 |      0.70     | 0.06 |\n|    LayerEnsemble   |  10.40 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.73      |  79.04 |      1.15     | 0.11 |\n|    DeepEnsemble   |  10.66 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.91      |  79.10 |      1.37     | 0.26 |\n|    LDU   |  9.14 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5.75      |  78.69 |      1.55     | 0.35 |\n|    MC-dropout   |  11.15 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.09      |  78.46 |      1.23     | 0.19 |\n|    NatPN   |  9.32 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5.30      |  78.45 |      1.62     | 0.34 |\n|    FGRM   | 8.52 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6.97      |  79.23 |      1.77     | 0.48 |\n\n>  * The differences (novelty) compared to existing methods are not clearly demonstrated.\n\nReply: Thank you for your comments. We would like to highlight the novelty of our proposed method based on the following three key aspects.\n \ni. Our proposed FGRM framework is **the first work** to directly optimize uncertainty estimation reward function in a RL paradigm for effective uncertainty estimation. Existing uncertainty estimation methods are based on standard supervised learning with task-specific objectives, such as cross-entropy loss or Dice loss, which are not directly related to the uncertainty estimation task. Instead, our method leverages the RL paradigm, which enables us to optimize a reward function closely tied to uncertainty estimation metrics for explicit model tuning of uncertainty estimation. This is not feasible in standard supervised learning due to the discrete and non-differentiable nature of these metrics, which are essential for effective back propagation. Such framework novelty is also acknowledged by reviewer WcWT.  \n \nii. We design a novel fine-grained parameter update mechanism based on fisher information matrix for the reward maximization in RL. Existing works on RL in vision tasks typically adopt the policy gradient theorem to estimate the gradient of the reward term, in which the gradient of each parameter is weighted by the reward uniformly. Instead, our designed mechanism imposes fine-grained reward-weighting for each network parameter individually so as to enable more effective model tuning based on the reward function. Such fine-grained parameter update design novelty is also acknowledged by reviewer WcWT and reviewer iwgk.  \n \niii. We leverage evidential deep learning for the pre-training of the segmentation backbone. Evidential learning explicitly parameterizes the conjugate prior of the categorical distribution, enabling us to quantify the aleatoric uncertainty and epistemic uncertainty separately. This is in contrast to standard segmentation models, where the probability values from the softmax layer mix both types of uncertainty together. The novelty of incorporating evidential learning is also acknowledged by reviewer WcWT and reviewer mEtW.\n \nBased on the three points mentioned above, we believe that our work brings new insights and valuable contributions to the task of uncertainty estimation, which would be of great interest to the research community.\n\n > * Additionally, in the manuscript, the authors mention the inclusion of out-of-distribution (OOD) data alongside in-distribution data. The reviewer suggests justifying the use of OOD data in the experiments to strengthen the logical flow of the paper.\n\nReply: Thanks for your suggestion. The logic to include OOD data in our experiments is to provide a comprehensive evaluation of our proposed uncertainty estimation method. In practical deployment scenarios, segmentation models often encounter data that deviates from their training distribution, resulting in potentially inaccurate segmentation predictions. By including OOD data, we aim to simulate these real-world challenges and validate the effectiveness of our method when dealing with OOD data, which is crucial for reliable uncertainty estimation. \n"}, {"review_id": "LDpKdUydFd", "reviewer": "Reviewer_WcWT", "summary": "This paper proposes a novel method for uncertainty estimation. A segmentation network is first pre-trained by considering a generative model where the segmentation of an input $x$ is drawn from a Dirichlet distribution, which enables MLE. \n\nThe main contribution of the paper is then the reinforcement learning (RL) algorithm proposed whereby a novel reward function is used to maximise uncertainty estimation. Lastly, the authors posit that not all parameters in the network should be updated accordingly and use ideas from EWC in continual learning to learn parameter-specific update rules.\n\nTo learn aleatoric and epistemic uncertainty quickly, the authors assume a generative model where the segmentation is drawn from a Dirichlet distribution, thus optimising the MLE through integration of the conjugate prior and the likelihood function. The aleatoric and epistemic (calibrated) uncertainties can consequently be obtained through the learned $\\alpha$ parameter.", "questions": "* Did the authors consider also papers such as those which propose learning heteroscedastic uncertainty with MC dropout as a baseline such as in Kendal et al.(https://arxiv.org/abs/1703.04977). This method is used quite significantly and it would have been nice to see how it performs in comparison.\n\n* It would be nice to see uncertainty maps of other models to compare. Is this possible?\n\n* It is not entirely clear how the RL algorithm mitigates confidence miscalibration and OOD over confidence. This is only really mentioned in passing in Section 3.4 in the implementation details for Equation 1\n\n* In the experiments, did the authors consider a model where only evidential learning was used as another baseline to compare against vanilla MLE? It would be nice to understand how strong of a baseline that is and whether this could be used in isolation.", "limitations": "None", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* This is an excellent paper. It is well written, clear in its intentions. The methodology is clearly presented (Figure 1 is excellent), it is clear how the algorithm works and why each section of the method was developed. The results compared to various baselines help consolidate the strength of the paper.\n\n* There are many novelties in this paper, which in isolation might already be interesting contributions. Taken as a whole, the authors have presented a compelling piece of work. The idea to combine i) evidential learning (parameterising the segmentation as $s ~ D(p|\\alpha)$ for uncertainty ii) using EWC to learn parameter updates necessary in the RL algorithm for calibrating uncertainty and iii) the RL reward function  are interesting and novel. The results are subsequently impressive.", "weaknesses": "* I would have liked to have seen evidence of the calibration on a more diverse set of datasets to better show the applicability and performance of the algorithm. A diverse set such as Cityscapes (real scenes), a dataset such as BraTS (tumour segmentation from MRI scans) in addition to the surgical videos used in this paper would have strengthened the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and positive feedback regarding the recognition of “many novelties” in our paper, clear presentation of our methodology and strength of experimental evaluation. We would like to provide the necessary clarifications and improvements in response to your comments as follows.\n\n > * Evidence of the calibration on a more diverse set of datasets to better show the applicability and performance of the algorithm.\n\nReply: Thank you for your comment. Following your suggestion, we have conducted additional experiments on the Cityscapes dataset to evaluate the uncertainty estimation of segmentation on real urban scenes. The results are presented below, along with comparisons to various baselines. It is evident from the results that our method outperforms all the comparison methods for uncertainty estimation on the Cityscapes dataset. These results demonstrate the effectiveness of our method in model uncertainty calibration on diverse datasets, including both surgical videos in medical applications and urban scenes in natural images.\n\n**Cityscapes dataset**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method  | ECE ↓ |      &nbsp;&nbsp;MI ↑      | Dice ↑&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |      PR ↑     | BR ↑ |\n| Backbone | 11.94 |      &nbsp;&nbsp;3.51      |  78.15 |      0.70     | 0.06 |\n|    LayerEnsemble   |  10.40 |      &nbsp;&nbsp;4.73      |  79.04 |      1.15     | 0.11 |\n|    DeepEnsemble   |  10.66 |      &nbsp;&nbsp;4.91      |  79.10 |      1.37     | 0.26 |\n|    LDU   |  9.14 |      &nbsp;&nbsp;5.75      |  78.69 |      1.55     | 0.35 |\n|    MC-dropout   |  11.15 |      &nbsp;&nbsp;4.09      |  78.46 |      1.23     | 0.19 |\n|    NatPN   |  9.32 |      &nbsp;&nbsp;5.30      |  78.45 |      1.62     | 0.34 |\n|    FGRM   | 8.52 |      &nbsp;&nbsp;6.97      |  79.23 |      1.77     | 0.48 |\n\n > * Did the authors consider also papers such as those which propose learning heteroscedastic uncertainty with MC dropout as a baseline such as in Kendal et al.(https://arxiv.org/abs/1703.04977). This method is used quite significantly and it would have been nice to see how it performs in comparison.\n\nReply: Thanks for the comment. We would like to draw your attention to Table 1 and Table 2 in the paper, where we have compared our method with MC dropout. The results show that our method outperforms MC dropout by a large margin across all the evaluation metrics.\n\n > * It would be nice to see uncertainty maps of other models to compare. Is this possible?\n\nReply: Thanks for the comment. Please kindly note that we have visualized the uncertainty maps of other models in Appendix Fig. 1. For clearer illustration, we have also updated Appendix Fig. 1 by including error maps of segmentation predictions (please refer to Fig. 2 of the uploaded PDF file). From the figure, we can see that our model generates uncertainty estimation maps that present better correlation with incorrect predictions when compared to other methods.\n\n > * It is not entirely clear how the RL algorithm mitigates confidence miscalibration and OOD over confidence. This is only really mentioned in passing in Section 3.4 in the implementation details for Equation 1\n\nReply: Thank you for the comment. For in-distribution (ID) calibration, our reinforcement learning (RL) algorithm aims to maximize the reward of Expected Calibration Error (ECE) metric. According to the formula of ECE metric shown in Appendix A.1, ECE metric quantifies the average disparity between confidence and accuracy, that is it effectively evaluates whether predictions characterized by higher confidence levels (indicative of lower uncertainty) are more likely to be accurate. By maximizing this metric with the RL algorithm, the confidence miscalibration can be reduced. For out-of-distribution (OOD) inference, our objective is to enable the model to assign higher uncertainty to the OOD regions compared with their ID counterparts. To achieve this, we have devised a reward function as the ratio between the uncertainty of OOD regions and that of ID regions. By maximizing this designed reward with the RL algorithm, the model learns to mitigate the OOD over confidence.\n\n > * In the experiments, did the authors consider a model where only evidential learning was used as another baseline to compare against vanilla MLE? It would be nice to understand how strong of a baseline that is and whether this could be used in isolation.\n\nReply: Thanks for the comment. Following your suggestion, we have included a model named “vanilla EDL”, in which only evidential learning is used as a comparison with vanilla MLE. The results in the tables below show that vanilla EDL achieves superior performance compared to vanilla MLE, demonstrating the benefits of evidential learning in uncertainty estimation. Moreover, our FGRM model further outperforms vanilla EDL, attributing to our RL framework with the designed fine-grained parameter update mechanism. \n\n**LC segmentation**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method  | ECE ↓ |      MI ↑      | Dice ↑ |      PR ↑     | BR ↑ |\n| Vanilla MLE | 19.46 |      2.88      |  71.22 |      0.88     | 0.08 |\n|    Vanilla EDL   |  12.98 |      4.20      |  73.66 |      1.45     | 0.21 |\n|    FGRM   |  9.63 |      5.87      |  74.88 |      1.85     | 0.47 |\n\n**ESD segmentation**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method | ECE ↓ |      MI ↑      | Dice ↑ |      PR ↑     | BR ↑ |\n| Vanilla MLE | 17.76 |      2.08      |  84.12 |      0.82     | 0.10 |\n|    Vanilla EDL   |  12.15 |      3.68      |  86.20 |      1.40     | 0.18 |\n|    FGRM   |  10.42 |      4.72      |  87.23 |      1.78     | 0.54 |"}, {"review_id": "tOK0zT83hA", "reviewer": "Reviewer_jxwz", "summary": "This paper introduces a novel Fine-Grained Reward Maximization (FGRM) framework to improve uncertainty estimation in deep segmentation models for safety-critical applications. The approach uses a reinforcement learning-based model tuning paradigm to optimize and calibrate the model. The FGRM framework is the first to leverage reinforcement learning for uncertainty estimation in safety-critical vision tasks, demonstrating improved performance on two surgical scene segmentation datasets.", "questions": "See the weaknesses.", "limitations": "While the paper is largely comprehensive, there are some areas where it falls short:\n\n1. Comparison with Other Methods: The paper claims superiority over state-of-the-art methods, however, it lacks a direct comparison or in-depth discussion with recent popular general uncertainty estimation methods, such as \"Confidence estimation via auxiliary models\" or methods specifically tailored for medical image segmentation like [18]. Including these comparisons would validate their claims more convincingly.\n\n2. Metric Citation Issue: There seems to be a mistake in the metric citation, which raises concerns about the overall quality of the experimental setup. \n\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, logically organized, and effectively explains the novel aspects of the proposed framework. \n2. The method has been rigorously tested on two large safety-critical surgical scene segmentation datasets, demonstrating superior performance.", "weaknesses": "1. The evaluation metrics Uncertainty error mutual information (MI),  Pixel Ratio (PR), and Box Ratio (BR) can not be found in  [18].\n2. In the related works and experiments, lack of discussion and comparison with the auxiliary network-based method, e.g. [18] and Corbiere, Charles, et al. \"Confidence estimation via auxiliary models.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable comments, providing us with opportunities for improvement and clarification. We would like to address each of your comments in detail as follows.\n\n> * Comparison with other methods: In the related works and experiments, lack of discussion and comparison with the auxiliary network-based method, e.g. [18] and Corbiere, Charles, et al. \"Confidence estimation via auxiliary models.\"\n\nReply: Following your suggestion, we have added the raised auxiliary network-based method ConfidNet (i.e., Corbiere Charles et al [R1]) into our experiments for comparison. The tables below show the performance of our FGRM method and ConfidNet on two surgical video datasets as well as a urban scene dataset of Cityscape (an additional general dataset). The results show that our method achieves better uncertainty estimation performance compared to Corbiere Charles et al. We will include these additioanl experiments in final version.\n\n**LC segmentation dataset**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  | ECE ↓ |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MI ↑      | Dice ↑&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |      PR ↑     | BR ↑ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|\n| ConfidNet | 16.76 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.58      |  71.32 |      1.41     | 0.32 |\n|    FGRM (ours)   |  9.63 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5.87      |  74.88 |      1.85     | 0.47 |\n\n**ESD segmentation dataset**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  | ECE ↓ |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MI ↑      | Dice ↑&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |      PR ↑     | BR ↑ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|\n| ConfidNet | 15.52 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3.53      |  84.41 |      1.19     | 0.26 |\n|    FGRM (ours)   |  10.42 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.72      |  87.23 |      1.78     | 0.54 |\n\n**Cityscapes dataset**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  | ECE ↓ |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MI ↑      | Dice ↑&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |      PR ↑     | BR ↑ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|\n| ConfidNet | 10.14 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4.69      |  78.15 |      1.18     | 0.19 |\n|    FGRM (ours)   |  8.52 |      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6.97      |  79.23 |      1.77     | 0.48 |\n\n> * The evaluation metrics Uncertainty error mutual information (MI), Pixel Ratio (PR), and Box Ratio (BR) can not be found in [18].\n\nReply: Thank you for the comment. In paper [18], a comprehensive benchmark of various uncertainty estimation methods are conducted. The metric of uncertainty-error overlap in [18] is a discrete version of the uncertainty error mutual information used in our experiment. We agree the importance of adding explicit references to the papers that thoroughly describe these metrics. Hence, we would like to add reference [R2] for the metric of uncertainty error mutual information and reference [R3] for the metrics of pixel ratio and box ratio. \n\nThank you very much for the careful review, it definitely makes our paper more rigorous. But we also would like to assure the reviewer regarding the quality and soundness of our experimental setup. There is no need to worry, and we will release all our code, data and models upon publication of the paper.\n\nReferences:\n\n[R1] Corbiere, C., Thome, N., Saporta, A., Vu, T.H., Cord, M. and Perez, P. Confidence estimation via auxiliary models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 2021, pp.6043-6055.\n\n[R2] Judge, T., Bernard, O., Porumb, M., Chartsias, A., Beqiri, A. and Jodoin, P.M. Crisp-reliable uncertainty estimation for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2022, pp. 492-502.\n\n[R3] Zepf, K., Wanna, S., Miani, M., Moore, J., Frellsen, J., Hauberg, S., Feragen, A. and Warburg, F. Laplacian Segmentation Networks: Improved Epistemic Uncertainty from Spatial Aleatoric Uncertainty. arXiv preprint arXiv:2303.13123. 2023.\n"}, {"review_id": "tL0pj3KigH", "reviewer": "Reviewer_iwgk", "summary": "As existing approaches for uncertainty estimation have been limited by the guidance for calibrating the prediction risk and model confidence, the paper proposes a novel fine-grained reward maximization (FGRM) framework, which addresses uncertainty estimation by reinforcement learning based model tuning with an uncertainty metric related reward function. It adopts the fisher information matrix for capturing parameter importance, acting as weights for fine-grained updates. Besides, evidential pre-training is incorporated to distinguish between aleatoric and epistemic uncertainty. The experimental results on two surgical datasets show FGRM improves uncertainty estimation for both ID and OOD data while not harming original segmentation performance.", "questions": "1. In line 306, why you say `without harming the accuracy` instead of `while promoting the accuracy` with Dice performance increases during reward tuning?\n2. As you use evidential learning to distinguish between aleatoric and epistemic uncertainty, what is the principle behind how it benefits your uncertainty estimation? You haven't explained this clearly in the ablation study.\n3. What does `efficient sampling` mean in the lower part of Figure 1? How does your sample algorithm differs from simple mini-batch sampling?\n4. Typo in line 324: safter => safer.", "limitations": "1. The paper lacks theoretical provements for FGRM like the uncertainty bounds, which is significant in the uncertainty estimation context.\n2. In Figure 3 and Figure 4(b), the paper doesn't record FGRM's performance on the OOD counterpart, which is an important contribution claimed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well motivated for the uncertainty estimation problem, with reasonable application of fisher information matrix and evidential learning. Though built upon previous works that uses RL for guided training, I think the fine-grained update mechanism is novel and the contributions are clear.\n2. The presentation is clear and easy to follow.\n3. The empirical performance is great comparing to baselines.", "weaknesses": "1. While enjoying superior empirical performance, it would be better to provide some theoretical insights for FGRM like the uncertainty bounds, which is significant in the uncertainty estimation context.\n2. The experiment section has some ambiguous points (see limitations below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback regarding the novelty of our proposed fine-grained parameter update scheme and the great empirical performance of our method. Our responses to your comments are as follows. \n\n> * While enjoying superior empirical performance, it would be better to provide some theoretical insights for FGRM like the uncertainty bounds, which is significant in the uncertainty estimation context. \n\nReply: Thank you for the comment. We would like to provide theoretical insights regarding our proposed fine-grained parameter update scheme. For uncertainty metric reward maximization, directly adopting the policy gradient method can be sub-optimal, which can be proofed with the closed-form solution of the policy network. In RL tuning process, the overall optimization objective can be expressed as, $  J(\\phi) =  E_{({x}, {y})}[R({\\mu}, {\\hat{y}}, {y}) - \\beta \\text{log}(\\frac{\\pi_{\\phi}({x})}{\\pi_{\\hat{\\theta}}({x})})] $. According to the policy gradient theorem, the gradient with respect to the first term can be calculated as $R(\\mu, \\hat{y}, y) \\nabla_{\\phi} \\text{log} \\pi_{\\phi}(\\mu, \\hat{y}|x)$. Updating parameters  $\\phi$ based on the policy gradient is towards to $\\pi_{\\phi}^* \\in \\{ \\pi_{\\phi} | \\nabla_{\\pi_{\\phi}}\\mathcal{J(\\phi)} = 0 \\}$, which can be expressed as, $\\pi_{\\phi}^* = \\frac{\\pi_{\\theta}\\text{exp}(R(\\boldsymbol{\\mu}^i, \\boldsymbol{\\hat{y}}^i, \\boldsymbol{y}^i)/\\beta)}{\\int \\pi_{\\theta}\\text{exp}(R(\\boldsymbol{\\mu}^i, \\boldsymbol{\\hat{y}}^i, \\boldsymbol{y}^i)/\\beta) d(\\boldsymbol{x}^i, \\boldsymbol{y}^i)}$. The solution $\\pi_{\\phi}^*$ can be viewed as weighting the MLE model by the normalized exponential reward. The limited space for exploration constrains the optimization, resulting in sub-optimal solutions. Our proposed fine grained update mechanism tackles this problem by assigning rewards separately to each network parameter according to its influence on the uncertainty reward. \n\n> * In line 306, why you say without harming the accuracy instead of while promoting the accuracy with Dice performance increases during reward tuning?\n\nReply: Thank you for the comment. The choice of phrasing is intended to highlight that our method aims to improve uncertainty estimation without negatively impacting the accuracy of the segmentation predictions. This emphasizes that our primary focus is on enhancing uncertainty estimation. However, it is also valid to use “while promoting accuracy” as evidenced by the increasing Dice performance during reward tuning. \n\n> * As you use evidential learning to distinguish between aleatoric and epistemic uncertainty, what is the principle behind how it benefits your uncertainty estimation? You haven't explained this clearly in the ablation study.\n\nReply: The principle behind how evidential learning benefits our uncertainty estimation lies in its capability to provide a better understanding of uncertainty types. Specifically, aleatoric uncertainty emerges from the inherent, irreducible variability presented in the data itself, such as when dealing with unclear tissue boundaries in surgical images. differently, epistemic uncertainty arises due to limited knowledge about unseen data. By being able to differentiate these two types of uncertainty, our model gains the capacity to provide more informative uncertainty estimates. In the ablation study, we further added a model named “Vanilla EDL” that exclusively uses evidential learning without the incorporation of the fine-grained RL algorithm. This ablation experiment aims to investigate the contribution of evidential learning by directly comparing it with the “Vanilla MLE” model. The results, presented in the table below, clearly indicate that evidential learning enhances uncertainty estimation by enabling the model to capture different types of uncertainty more accurately. This helps the model to better understand the limits of its predictions and produce more meaningful uncertainty estimates.\n\n**LC segmentation**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method | ECE ↓ |      MI ↑      | Dice ↑ |      PR ↑     | BR ↑|\n| Vanilla MLE | 19.46 |     2.88      |  71.22 |      0.88     | 0.08 |\n|    Vanilla EDL   |  12.98 |      4.20      |  73.66 |      1.45     | 0.21 |\n|    FGRM   |  9.63 |      5.87      |  74.88 |      1.85     | 0.47 |\n\n**ESD segmentation**\n\n|           |       | ID calibration |        | OOD inference |      |\n|-----------|-------|----------------|--------|---------------|------|\n|   Method  | ECE ↓ |      MI ↑      | Dice ↑ |      PR ↑     | BR ↑ |\n| Vanilla MLE | 17.76 |     2.08      |  84.12 |      0.82     | 0.10 |\n|    Vanilla EDL   |  12.15 |      3.68      |  86.20 |      1.40     | 0.18 |\n|    FGRM   |  10.42 |      4.72      |  87.23 |      1.78     | 0.54 |\n\n> * What does efficient sampling mean in the lower part of Figure 1? How does your sample algorithm differs from simple mini-batch sampling?\n\nReply: The term “efficient sampling” in Fig. 1 refers to the inference process of uncertainty estimation. Our method only requires a single forward pass to obtain the uncertainty prediction, which is highly efficient compared to other ensemble or probabilistic-based methods that require multiple forward passes for the uncertainty estimation. \n\n> * In Figure 3 and Figure 4(b), the paper doesn't record FGRM's performance on the OOD counterpart, which is an important contribution claimed in the paper.\n\nReply:  For Fig. 3 and Fig. 4(b), we further added ablation studies on the OOD scenario using the endoscopic submucosal dissection dataset, including contribution of each key component, effect of the fine-grained parameter update, and the progression of uncertainty estimation and segmentation prediction. The results are presented in Fig. 3(b)-(d) of the uploaded PDF file. These results on the OOD scenario reaffirm the findings observed in the ID scenario. "}]

Author rebuttal:
We appreciate the reviewers for taking their time to review and provide constructive feedback. We are glad to see that most reviewers recognized the novelty of our method, the strength of our experimental evaluations, and the good presentation of our paper. 

We have made every effort to address all the concerns raised, including conducting additional experiments on the Cityscapes dataset to validate that our method can have broad usage, adding more comparison experiments and ablation studies, and providing clarifications where needed. Our detailed responses have been provided to each reviewer separately in below. We have also uploaded a PDF file containing updated figures for a clearer visualization of uncertainty estimation and more ablation results. 

We hope that our rebuttal can successfully address the reviewers' questions, and look forward to receiving the reviewers’ support for our work.

Meta-review: This paper introduces a new uncertainty estimation method for medical imaging segmentation tasks.
All but one reviewer recommended for acceptance. The key strengths are great paper writing and many novel contributions supported by thorough experimental results. Rw jxwz is the only one recommended borderline rejection. jxwz's main concerns are the lack of comparison with auxiliary network-based methods and SoTA medical image methods. The authors provided additional experimental results addressing these concerns. In the end, Rw jxwz didn't update the rating, but Rw mEtW followed the discussion and mentioned that the additional results further strengthened the paper. AC agrees with most of the reviewers and recommends acceptance.
Decision: Accept (poster)