AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images
Authors: Prithvijit Chattopadhyay
Bharat Goyal
Boglarka Ecsedi
Viraj Uday Prabhu
Judy Hoffman
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised Domain Adaptation
Sim2Real
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A method to reduce miscalibration for unsupervised Sim2Real adaptation by optimizing for calibrated predictions on augmented synthetic data.

Abstract: Synthetic data (Sim) drawn from simulators have emerged as a popular alternativefor training models where acquiring annotated real-world images is difficult. However, transferring models trained on synthetic images to real-world applicationscan be challenging due to appearance disparities. A commonly employed solution to counter this Sim2Real gap is unsupervised domain adaptation, where models are trained using labeled Sim data and unlabeled Real data. Mispredictions made by such Sim2Real adapted models are often associated with miscalibration – stemming from overconfident predictions on real data. In this paper, we introduce AUGCAL, a simple training-time patch for unsupervised adaptation that improves Sim2Real adapted models by – (1) reducing overall miscalibration, (2) reducing overconfidence in incorrect predictions and (3) improving confidence score reliability by better guiding misclassification detection – all while retaining or improving Sim2Real performance. Given a base Sim2Real adaptation algorithm, at training time, AUGCAL involves replacing vanilla Sim images with strongly augmented views (AUG intervention) and additionally optimizing for a training time calibration loss on augmented Sim predictions (CAL intervention). We motivate AUGCAL using a brief analytical justification of how to reduce miscalibration on unlabeled REAL data. Through our experiments, we empirically show the efficacy of AUGCAL across multiple adaptation methods, backbones, tasks and shifts.

Published as a conference paper at ICLR 2024
AUGCAL: IMPROVING SIM2REAL ADAPTATION BY UNCERTAINTY CALIBRATION ON AUGMENTED SYNTHETIC IMAGES
Prithvijit Chattopadhyay" Bharat Goyal Boglarka Ecsedi
Viraj Prabhu
Judy Hoffman
Georgia Tech
(prithvijit3, btaragoylll 3ecsedi, judy
ABSTRACT
Synthetic data (SIM) drawn from simulators have emerged popular alternative for training models where acquiring annotated real-world images is difficult. How- ever, transferring models trained synthetic images real-world applications can be challenging due to appearance disparities. A commonly employed solution to counter this SIM2REAL gap is unsupervised domain adaptation, where models are trained using labeled SIM data and unlabeled REAL data. Mispredictions made by such SIM2REAL adapted models often associated with miscalibration stemming from overconfident predictions on real data. In this paper, we introduce AUGCAL, simple training-time patch for unsupervised adaptation that improves SIM2REAL adapted models by - (1) reducing overall miscalibration, (2) reduc- ing overconfidence in incorrect predictions and (3) improving confidence score reliability by better guiding misclassification detection all while retaining or im- proving SIM2REAL performance. Given a base SIM2REAL adaptation algorithm, at training time, AUGCAL involves replacing vanilla SIM images with strongly aug- mented views (AUG intervention) and additionally optimizing for training time calibration loss augmented SIM predictions (CAL intervention). We motivate AUGCAL using a brief analytical justification of how to reduce miscalibration on unlabeled REAL data. Through our experiments, we empirically show the efficacy of AUGCAL across multiple adaptation methods, backbones, tasks and shifts.
INTRODUCTION
Most effective models for computer vision tasks (classification, segmentation, etc.) need learn from a large amount of exemplar data (Dosovitskiy et al., 2020; Radford et al., 2021; Kirillov et al., 2023; Pinto al., 2008) that captures real-world natural variations which may occur at deployment time. However, collecting and annotating such diverse real-world data can be prohibitively expensive for instance, densely annotating a frame of Cityscapes (Cordts et al., 2016) can take upto ~ 1.5 hours! Machine-labeled synthetic images generated from off-the-shelf simulators can substantially reduce this need for manual annotation and physical data collection (Sankaranarayanan et al., 2018; Ros et al., 2016; Blaga & Nedevschi, 2019; Savva et al., 2019; Deitke et al., 2020; Chattopadhyay REAL et al., 2021). Nonetheless, models trained on SIM data often exhibit subpar performance data, primarily due to appearance discrepancies, commonly referred to as the SIM2REAL gap. For instance, GTAV (SIM) + Cityscapes (REAL), an HRDA SIM-only model (Hoyer et al., 2022b) achieves an mloU of only 53.01, compared to 81 mloU attained by an equivalent model trained exclusively on REAL data.
While there significant effort in improving the realism of simulators (Savva et al., 2019; Richter et al.. 2022), there is an equally large effort seeking to narrow this SIM2REAL performance gap by designing algorithms that facilitate SIM2REAL transfer. These methods encompass both gener- alization (Chattopadhyay** et al., 2023; Huang et al., 2021: Zhao et al., 2022). aiming to ensure strong out-of-the box REAL performance of SIM trained models and adaptation (Hoyer al.. 2022b:c; Vu et al., 2019; Rangwani et al., 2022) attempting to adapt models using labeled SIM data and unlabeled REAL data. Such generalization and adaptation methods have demonstrated notable success in reducing the SIM2REAL performance gap. For instance, PASTA (Chattopadhyay* al., 2023) (a generalization method) improves SIM2REAL performance of a SIM-only model from
"Correspondence PC

---

Published as a conference paper at ICLR 2024
Sim2Rual:
Sim2Real: EniMiin
(Teddesir)) ky vewdemmmnttt Seoffied model
Reall Cityecapes imsge
Predictions
Predictions
Bbx7
53.0
LDA E:0.5
23 & S
Mindeenbee improdition
ov Lindlee
Cityscapes Ground Truth
Overconfident Mispredictions
Overconfident Mispredictions
100
88.7
62.8
73.6
incorrectly dentifee aas "raudd" with a0%
Inior rectly identiiie "wal" with 80%
E2
Bru Oni
corridence
comfideeccc
road
sidewalk
building
wall
fence
pole
traffic Igt
traffic sgm
vegetation
ignored
lerruin
sky
rider
truck
bus
train
motorcycle
bike
Figure 1: Overconfident SIM2REAL mispredictions. [Left] We show an example of what we mean by overconfident mispredictions. For SIM2REAL adaptation on GTAV - Cityscapes, we choose (DAFormer) HRDA + MIC (Hoyer al., 2022c) and EntMin MIC (Vu al., 2019) (highly performant SIM2RFAI. methods) and show erroneous predictions Cityscapes (bottom row). We see that the model identifies sidewalk pixels as road (2nd column) and fence pixels as wall (3rd column) with very high confidence. [Right] We show how pervasive this "overconfidence"" phenomena is. While better SIM2REAL adapted models from (DAFormer) Source- Onlee (Hoyer et al., 2022b) to (DAFormer) EntMin + MIC (Vu et al., 2019) to (DAFormer) HRDA + MIC (Hoyer et al., 2022c) exhibit improved transfer performance [Top, Right], they also exhibit increased overconfidence in mispredictions [Bottom, Right], affecting prediction reliability.
53.01 + 57.21 mloU. Furthermore, HRDA + MIC (Hoyer et al., 2022c) (an adaptation approach) pushes performance even higher to 75.56 mloU.
While SIM2REAL performance may increase both from generalization or adaptation methods, for safety-critical deployment scenarios, task performance is often not the sole factor of interest. is additionally important to ensure SIM2REAL adapted models make calibrated and reliable predictions on REAL data. Optimal calibration on real data ensures that the model's 's confidence in its predictions aligns with the true likelihood of correctness. Deploying poorly calibrated models can have severe consequences, especially high-stakes applications (such as autonomous driving), where users can place trust in (potentially) unreliable predictions (Tesla Crash, 2016; Michelmore et al., 2018). We find that mistakes made by SIM2REAL adaptation methods often associated with miscalibration caused by overconfidence - highly confident incorrect predictions (see Fig. Left). More interestingly, find that adaptation methods improve in terms SIM2REAL performance, the propensity make overconfident mispredictions also increases (see Fig. Right). Our focus in this paper is to devise training time solutions mitigate this issue.
Calibrating deep neural networks (for such SIM2REAL adaptation methods) crucial, as they routinely make overconfident predictions (Guo et al., 2017: Gawlikowski et al., 2021; Minderer al., 2021). While various techniques address miscalibration on "labeled data splits" for in- distribution scenarios, maintaining calibration in the face of dataset shifts, like SIM2REAL, proves challenging due to lack of labeled examples in the target (REAl) domain. To address this, we propose AUGCAL, a training- time patch to ensure existing SIM2REAL adaptation methods make accurate, calibrated and reliable predictions on real data. When applied to a SIM2REAL adaptation framework, AUGCAL aims to satisfy three key criteria: (1) retain performance of the base REM2REAL method, (2) reduce miscalibration and overconfidence and (3) ensure calibrated confidence scores translate to improved reliability. Additionally, to ensure broad applicability, AUGCAL aims to do so by making two minimally invasive changes a SIM2REAL adaptation training pipeline. First, by AUGmenting (Cubuk et al., 2020; Chattopadhyay* et al., 2023) input STM images during training using AUG transform that reduces distributional distance between SIM and REAL images. Second, by additionally optimizing for CAL.bbation loss (Hebbalaguppe et al., 2022; Liang et al., 2020a; Liu et al., 2022) at training time AU Amented SIM predictions. We devise AUGCAL based on an analytical rationale (see Sec. 3.2.1 and 3.2.2) illustrating how it helps reduce an upper bound on desired target (REAL) calibration error. Through experiments on PrAVV+Cityscapp. and VisDA SIM2REAL, we demonstrate how AUGCAL helps reduce miscalibration on REAL data. To summarize, we make the following contributions:
We propose AUGCAL, a training time patch, compatible with existing SIM2REAL adaptation methods that ensures SIM2REAL adapted models make accurate (measured via adaptation perfor- mance), calibrated (measured via calibration error) and reliable (measured via confidence guided misclassification detection) predictions.
2

---

Published as a conference paper at ICLR 2024
We conduct SIM2REAL adaptation experiments for object recognition (VisDA (Peng al., 2017))
and semantic segmentation (GTAV (Sankaranarayanan et al., 2018) B (Cordis et al., 2016)) with three representative UDA methods (pseudo-label based self-training, entropy min- imization and domaín adversarial training) and show that applying AUGCAI- (1) improves or preserves adaptation performance, (2) reduces miscalibration and overconfidence and (3) improves the reliability of confidence scores.
We show how AUGCAL improvements are effective across multiple backbones, AUG and CAL options and highlight choices that are more consistently effective across experimental settings.
2 RELATED WORK
Unsupervised Domain Adaptation (UDA). We focus on UDA algorithms to address covariate shifts in the SIM2REAL context (Chattopadhyay* et al., 2023; Choi et al., 2021; Zhao et al., 2022; Huang et al., 2021; Rangwani et al., 2022; Hoyer et al., 2022c; Sankaranarayanan al., 2018; Ros et al., 2016). This involves adapting a model to an unseen target (REAL) domain using labeled samples from a source (SIM) domain and unlabeled samples from the target domain. Here, the source and target datasets share the same label space and labeling functions, but differences exist in the distribution of inputs (Farahani et al., 2021; Zhang et al., 2019). SIM2REAL UDA methods (Ganin & Lempitsky, 2014; Hoffman et al., 2018; Saenko et al., 2010; Tzeng et al., 2014) range from feature distribution matching (Ganin & Lempitsky, 2014; Long et al., 2018; Saito et al., 2018; Tzeng ct al., 2017: Zhang et al., 2019), explicitly addressing domain discrepancy (Kang et al., 2019; Long et al., 2015; Tzeng et al., 2014; Rangwani et al., 2022), entropy minimization (Vu ct al., 2019) or pseudo-label guided self-training (Hoyer et al., 2022b;a;c We observe that existing SIM2REAL UDA methods usually improve performance at the expense of increasingly overconfident mispredictions on (REAL) target data (Wang et al., 2020b). Our proposed method, AUGCAL, is designed to retain SIM2REAL adaptation performance while reducing miscalibration on real data for existing methods. We conduct experiments on three representative UDA methods Entropy Minimization (Vu et al., 2019), Self-training (Hoyer et al., 2022b) and Domain Adversarial Training (Rangwani et al., 2022).
Confidence Calibration for Deep Networks. For discriminative models, confidence calibration indicates the degree to which confidence scores associated with predictions align with the true likelihood of conrectness (usually measured via ECE (Naeini al., 2015)). Deep networks tend to be be very poor at providing calibrated confidence estimates (are overconfident) for their predictions (Guo et al., 2017; Gawlikowski al., 2021; Minderer et al., 2021), which in turn leads to less reliable predictions for decision-making in safety-critical settings. Recent work (Guo et al., 2017) has also shown that calibration worsens for larger models and can decrease with increasing performance. Several works (Guo et al., 2017; Lakshminarayanan et al., 2017; Malinin & Gales, 2018) have explored this problem for modern architectures, and several solutions have also been proposed -including temperature scaling (prediction logits being divided by a scalar learned on a held-out set (Platt et al., 1999; Kull et al., 2017; Bohdal et al., 2021; Islam et al., 2021)) and trainable calibration objectives (training time loss functions that factor in calibration (Liang et al., 2020a; Karandikar et al., 2021)). Improving network calibration is even more challenging in out-of-distribution settings due to the simultaneous lack of ground truth labels and overconfidence on unseen samples (Wang et al., 2020b). Specifically, instead of methods that rely on temperature-scaling (Wang et al., 2020a; 2022) or maybe require an additional calibration split, AUGCAL explores the use of training time calibration objectives (Munir et al., 2022) to reduce micalibration for SIM2REAL shifts.
3 METHOD
3.1 BACKGROUND
Notations. Let x denote input images and y denote corresponding labels (from the label space y = 2,,, K}) drawn from a joint distribution P(x, y). We focus on the classification case, where the goal is to learn discriminative model Ms (with parameters 0) that maps input images to the desired K output labels, Mg X' + Y, using a softmax layer on top. The predictive probabilities for the given input can be expressed as po()) = (ootmax(MM((2)) We use ý = arg yaxyey Po Po(y to denote the predicted label for x and e to denote the confidence in prediction.
Unsupervised SIM2REAL Adaptation. In unsupervised domain adaptation (UDA) for SIM2REAL settings, we assume access to labeled (SIM) source dataset Ds = ((=). v | and unlabeled (REAL) target dataset Dr = { [T]
1==1 We assume Ds and Dr splits are drawn from source and target distributions ps (I, v) and pf (x. y) respectively. At training, we have access to D = DsUDr. We operate in the setting where source and target share the same label space, and discrepancies exist.
3

---

Published as a conference paper at ICLR 2024
only in imput images. The model Ma is trained on labeled source images using entropy,
[s
|s|
Σ Coe(E, (c:(:..01:0))
E yi log po lx,)) where o arg maxxappp( (x;)
(I)
yEy
UDA methods additionally optimize for adaptation objective on labeled source and unlabeled target data DAu. The overall learning objective can be expressed as,
12'l
[s]
min
L ve: +
AUDALUDA(Dd 0f
(2)
=1
Source Loss
Source Target Adapiation Loss
Different adaptation methods usually differ in terms of specific instantiations of this objective. While AUGCAL is applicable to any SIM2REAL adaptation method in principle, conduct ex- periments with three popular methods Entropy Minimization (Vu et al., 2019), Pscudo-Label driven Self-training (Hoyer et al., 2022b) (for semantic segmentation) and Domain Adversarial Training (Rangwani et al., 2022) (for object recognition). We provide more details on these methods in Sec. A.9 of appendix.
Uncertainty Calibration. For perfectly calibrated classifier, the confidence predictions should match the empirical frequency of correctness. Empirically, calibration can be measured using Expected Calibration Error (ECE) (Naeini et al., 2015). To measure ECE on test set D) [D]
{1r,, y:) |=) we first partition the test data into B bins, Do = {(x, r--1 S c < rb). using the confidence values such that {1,..., B) and ro ri S r2 measures the absolute differences between accuracy and confidence across instances in every bin,
rn 1. Then, ECE
B ECE Σ
B
Σ 1 (m. VED,
ΣG iED;
We are interested in models that exhibit high-performance and low calibration error (ECE). Note that Eqn alone does not indicate if a model is overconfident. We define overconfidence (OC) as the expected confidence on mispredictions. Prior work on improving calibration in -fist--diittibutiinn (OOD) settings (Wang al., 2022) and domain adaptation scenarios (Wang al., 2020b) typically rely on techniques like temperature scaling. These methods often necessitate additional steps, such as employing separate calibration split domains (Gong al., 2021) or training extra models (e.g.. logistic discriminators for source and target features (Wang et al., 2020b)). In contrast, we consider using training time calibration objectives (Liang al., 2020b; Hebbalaguppe et al., 2022; Liu et al., 2022) that can be optimized addition to task task-specific objectives for improved calibration.
3.2 AUGCAL
3.2.1
REDUCING MISCAL ON (REAL.) TARGET
Recall that (3 y) and pf (x,, v) denote the source (SIM) and target (REAL) data distributions. We assume P(x, 3) factorizes as P(x, v) P(x) P(ylz). We assume covariate shift conditions between (y)z) pT
ps and PT, i.e., pT(xx) # ps (I) while pT exist only in input images. When training model, we can only draw "labeled samples" (z, y) from ps (z, y). We do not have access to labels from pT (Ix y). Our goal is to reduce miscalibration on (unlabeled) target data using training time calibration losses. Let LCA. (z. y) denote such calibration loss we can minimize (on labeled data). Using importance sampling (Cortes et al., 2010), we can get an estimate of the desired calibration loss on target data as,
ps (y/lz) discrepancies across distributions
[./, CCAL y)TT (x. y) dx dy
LEca (I,, y)
a.y= (7.0))
PT(x) (y/x) [ Lcne (x, ps (S(x) (y/xr
ps pxx y) do dy
ws(x)
Lcau (x.H)
P8 (x..)
Importance Weight Source
4

---

Published as a conference paper at ICLR 2024
where ws(()) pf (a))
(x) denotes the importance weight. Assuming (C,,((r;)) v) 2 0', we can obtain
an upper bound on step 4 (Pampari Ermon, 2020; Wang et al., 2020b) as
[Ccal (z, y)]
E
(ws CcAL v)]
(5)
s,yn P
P (2.2)
3.y- (a..)
(s)())2
IE (S(x)))
(Cculas,
(6)
(x)
us (x)) (3)))
(eau w)2
pe (2)21)
Shift Dependent
Source Dependent
where steps 6 and 7 use the Cauchy-Schwarz and AM-GM inequalities respectively. For given model, the second RHS term in inequation is computed purely on labeled samples from the source distribution and can therefore be optimized to convergence over the course of training. The gap in LCAI across source and target dominated by the importance weight (first term). Following (Cortes et al., 2010) O), the first term can also be expressed as,
[ = d2( (s))IP((()))
E
where (PIQ) = Σx
with a > O is the exponential in base 2 of the Renyi- divergence (Rényi, 1960) between distributions /' and Q. The calibration error gap between source and target distributions therefore, dominated by the divergence between source and target distribu- tions. Consequently, inequation 7 can be expressed as,
E 7. PT (m.
(CcAL (x, v) S dz d2(PP" (x) (x))
CCAL w)
T Upper Bound
2 ps (m..)
Sourze and Target Divergence
Target Calibration Loss
Source Calibration Loss
where USS, T)) denotes the upper bound on target calibration loss. Therefore, to effectively reduce miscalibration on target data, one needs to reduce the upper bound, U(S T), which translates to (1) reducing miscalibration source data (second red term in 9) and (2) reducing the distributional distance between input distributions across source and target (first blue term in 9).
3.2.2 WHY AUGCAL?
Based on the previous discussion, improve calibration on target data, one can always invoke training time calibration intervention (CAL) on labeled source data to reduce Eps (x.a) [Cea (I, v)]. In practice, after training, we can safely assume that (xs [Ces (I) e + 0 (for some very small e). We note that while this is useful and necessary, it not sufficient. This is precisely where we make our contribution. To reduce both (red and blue) terms in 9, we introduce AUGČAL. To do this, in addition to training time calibration loss, LEAL AUGCAL assumes that access to additional AUG transformation that satisfies the following properties:
1. d2 (PT (PS) s (TTT()))) (x))
(I, w) so (p)().)) (Cu =
2. After training, (p)(),))
Property 1 states that the chosen AUG transformation brings transformed source data closer to target (or reduces SIM2REAL distributional distance). Property states that over the course of training, irrespective of the data (xsL(2,) y) is optimized on (AUG transformed or clean source),
(ps(xa) (. can achieve a sufficiently small value close to O. Given an AUG transformation that satisfies the above stated properties, we can claim,
AUG (S,T) < (S,TT)
(10)
where
AuG
A (S,T) = da (x)][PS (AUG((()
1 ps (s.2))
E
[Cca(AUG(())
(11)
We make the reasonable assumption that the calibration loss function is always non-negative.
5

---

Published as a conference paper at ICLR 2024
Supervised Loss Coe
AUG
CAL
Calibration Loss Cyv
Source GT
Source Image
Augmented Image
Model Ma
Adaptation Loss 4ppa
N/A
Target Image
AUGCAL Base UDA Method
Figure 2: AUGCAL pipeline. AUGCAL consists of two key interventions existing SIM2REAL adaptation method. First source SIM images are augmented via AUG transform. Supervised losses for SIM images are computed on augmented image predictions. Additionally, AUGCAL optimizes calibration on AUGmented SIM predictions.
That is, an appropriate AUG transform, when coupled with LCA., helps reduce tighter upper bound on the target calibration error than CAL. We call this intervention coupling AUG and CAL- AUGCAL. Naturally, the effectiveness of AUGCAL directly dependent on the choice of AUG that satisfies the aforementioned properties. While most augmentations can satisfy property
Table 1: Eligible AUG choices. transformations that reduce SIM2REAL distance and satisfy Property-1.
SIM2REAL
100x (RBF) MMD
REAL.)
(PASTA-SIMA REAL)
(R.Ang-SrM,
GTAV -Cily
3.795
5.214
5.213
check if an augmentation is valid according to property 1, we compute RBF Kernel based MMD distances for (SIM, REAL) and (AUGmented SIM, ReAL) feature pairs using trained model.? In Table. 1, show how PASTA (Chattopadhyay" et al., 2023) and Rand/ugment (Cubuk et al., 2020), two augmentations effective for SIM2REAL transfer, satisfy these considerations multiple shifts (read Table. left to right). PASTA and RandAug are additionally (1) inexpensive when combined with SIM2REAL UDA methods, and (2) generally beneficial for SIM2REAL shifts (PASTA via SIM2REAL specific design and RandAug via chained photometric operations).
3.2.3 AUGCAL INSTANTIATION
Given SIM2REAL adaptation method, AUGCAL additionally optimizes for improved calibration on augmented SIM source images. Since AUGCAL is applicable to any existing SIM2REAL adaptation method, we abstract away the adaptation component associated with the pipeline and denote as LUDA (see Eqn. 15). The steps involved in AUGCAL are illustrated in Fig. Given mini-batch, we first generate augmented views, (UG(x²), for SIM images x². Then, during training, we optimize Lcz those augmented SIM views and LUDA for adaptation. To improve calibration under augmentations, we optimize an additional LCAI. loss on augmented SIM images. The overall AUGCAL optimization problem be expressed as,
[T| [s|
min
(c
Aca Lcu. Cww. (AUG(s)))
8
Source Task Loss
Target Adaptation
Source Calibration Loss
(12)
where AUDA and AcAI denote the respective loss coefficients and the changes to a vanilla SIM2REAL adaptation framework are denoted in teal.
Choice of AAUG". Strongly augmenting SIM images during training has proven useful for SIM2REAL
transfer (Chattopadhyay* et al., 2023; Huang et al., 2021; Zhao et al., 2022; Cubuk al., 2020). For AUGCAL, we are interested in augmentations that satisfy the properties outlined in Sec, 3.2.2. As stated earlier, we find that RandAugment (Cubuk al., 2020) and PASTA (Chattopadhyay" al., 2023) empirically satisfy this criteria. We use both of them for our experiments and provide details on the operations for these AUG transforms in Sec. A.4 of appendix.
Choice of "CAL" (Lea.). AUGCAL relies on using training time calibration losses to reduce miscalibration. Prior work in uncertainty calibration has considered several auxiliary objectives to
Under bounded importance weight assumptions, MMD can be interpreted upper bound KL
divergence (Wang & Tay, 2022).

---

Published as a conference paper at ICLR 2024
calibrate a model being trained to reduce negative log-likelihood (NLL) (Hebbalaguppe et al., 2022; Kumar et al., 2018; Karandikar et al., 2021; Liang et al., 2020a). For "CAL" in AUGCAL, while we consider multiple calibration losses - DCA (Liang et al., 2020b), MbLS (Liu et al., 2022) and MDCA (Hebbalaguppe et al., 2022) - and find that DCA, simple "difference between confidence and accuracy (DCA)" loss proposed in (Liang et al., 2020a) is more consistently effective across experimental settings. DCA can be expressed as,
1s|
18
1 [8]
CAAL
Σ
Σ
Pa (9) (x;))
(13)
where (o 9 and (o( (x{} denote the correctness and confidence scores associated with predic- tions. The DCA loss forces the mean predicted confidence over training samples to match accuracy. In the following sections, we empirically validate AUGCAL across adaptation methods.
4 EXPERIMENTAL DETAILS
We conduct SIM2REAL adaptation experiments across two tasks Semantic Segmentation (SemSeg) and Object Recognition (ObjRec). For our experiments, we train models using labeled SIM images and unlabeled REAL images. We test trained models REAL images.
SIM2REAL Shifts. For SemSeg. we conduct experiments on the (TAV->Cityscappe shift. GTAV (Sankaranarayanan et al., 2018) consists of ~ 25k densely annotated SIM ground-view images and Cityscapes (Cordts et al., 2016) consists of ~ 5k REAL ground view images. We report all metrics on the Cityscapes validation split. For ObjRec, we conduct experiments on the VisDA SIM2REAL. benchmark. VisDA (Peng et al., 2017) consists of ~ 152k SIM images and ~ 55k REAL images across 12 classes. We report all metrics on the validation split of (REAL.) target images.
Models. We check AUGCAL compatibility with both CNN and Transformer based architectures. For SemSeg, we consider DeepLabv2 (Chen et al., 2017) (with ResNet- 101 (He et al., 2016) backbone) and DAFormer (Hoyer al., 2022a) (with an MiT-B5 (Xie et al., 2021) backbone) architectures. For ObjRec, we consider ResNet-101 and ViT-B/16 (Dosovitskiy et al., 2020) backbones with bottleneck layers classifiers. We start with backbones pre-trained on ImageNet (Deng al., 2009).
Adaptation Methods. We consider three representative SIM2REAL adaptation methods for our experiments. For SemSeg, we consider entropy minimization (EntMin) (Vu et al., 2019) and high- resolution domain adaptive semantic segmentation (HRDA) (Hoyer et al., 2022b). For ObjRee, we consider smooth domain adversarial training (SDAT) (Rangwani et al., 2022). For both tasks, further improve performance with masked image consisteney (MIC) (Hoyer et al., 2022c) on target images during training. We use MIC (Hoyer et al., 2022c)'s 's implementations of the adaptation algorithms and provide more training details in Sec. A.5 of appendix.
Calibration Metries. We use ECE to report overall confidence calibration on REAL images. Since we are interested in reducing overconfident mispredictions, we also report calibration error on incorrect samples (IC-ECE) (Wang et al., 2022) and mean overconfidence for mispredictions (OC).
Reliability Metries. While reducing overconfidence and improving calibration on real data is desirable, this is a proxy for the true goal of improving model reliability. To assess reliability, following prior work (de Jorge et al., 2023; Malinin et al., 2019), measure whether calibrated confidence scores can better guide misclassificction detection. To measure this, we use Prediction Rejection Ratio (PRR) (Malinin al., 2019), which if high (positive and close to 100) indicates that confidence scores can be used as reliable indicators of performance (details in Sec. A.6 of appendix).
Unless specified otherwise, we use PASTA as the choice of AUG and DCA (Liang et al., 2020b) as the choice of CAL in AUGCAL. We use ACAL = for DCA.
5 FINDINGS
5.1
IMPROVING SIM2REAL ADAPTATION
Recall that when applied to SIM2REAL adaptation method, we expect AUGCAL to to-(1 (1) retain SIM2RFAL transfer performance, (2) reduce miscalibration and overconfidence and (3) ensure calibrated confidence scores translate to improved model reliability. We first verify these criteria.
> AUGCAL improves or retains SIM2REAL adaptation performance. Since AUGCAL intervenes on an existing SIM2REAL adaptation algorithm, we first verify that encouraging better calibration does not adversely impact SIM2REAL adapation performance. We find that performance is either retained or improved (e.g., for EntMin + MIC in Table. 2 (a)) as miscalibration is reduced (Tables. 2(a) and (b), Perf. columns).
7

---

Published as a conference paper at ICLR 2024
Table 2: AUGCAL ensures SIM2REAL adapted models make accurate, calibrated and reliable predictions. We find that applying AUGCAL to multiple SIM2REAI. adaptation methods across tasks leads better calibration (ECE, IC-ECE), reduced overconfidence (OC) and improved reliability (PRR) - all while retaining or improving transfer performance. Highlighted rows are AUGCAL variants of the hase methods. For AUGCAL, we use PASTA
as AUG and DCA as CAL. 1 indicates standard error.
Method
Perf. (t)
Calibration Error (L)
Reliability (?)
EmiMin MIC
mloU
65.71
5.34 27
ECE IC-ECE
OC
45.97
PRR
Method
Perf. (f) mloU
BCE
Calibration Error ()
IC-BCE OC
Reliability (?) PRR
2 + ABGCAL
70.31
3.4 72,97 uss 82.80 as
62.66
SDAT + MIC 92.53 7.67 91.45 sou 89. 13
63.78
HRDA MIC
75.56
2.86 881.22: -9.72-a
68.91
AUGCAL 92.77a0 6.84gm 89.25 85.74301
67.80.01s
AUGGAL 75,90
2.45 79.09 a 88.26
20.35
(b) VisDA SIM2REAL. (ViT-B).
(a) GTAV +Cityscapes. (DAFormer).
Table 3: AUGCAL is better than applying AUG or CAL alone. On GTAV Cityscpes and VisDA, we show
that AUGCAL improves over just augmented SIM training (AUG) just optimizing for calibration on SIM data (CAL), For AUGCAL, use PASTA as AUG and DCA CAL. I indicates standard error.
Perf. (1)
Calibration Error ()) Reliability (1)
Pert. (1) Calibration Error (1) Reliability (f) PRR
Method
Method
mloU
ECE
IC-ECE
PRR
mAcc
ECE
IC-RCE
1
EntMin+ MIC
65.71
45.93
SDAT + MIC
92.53
7.67
91,45
63.78
2
AUG
67.58
4,30_av
77. 29
48.05
2
+ AUG
92.69
9.48
90.48
65.66 25.68gnd
CAL
68.70
4.04-12
75. 25.86s
52.52+a34
CAL
91.63-mn 7.30smm
91.19
66.62 66.62=1m
4 AUGCAL
70.31
3,43
72.97-0a
62.66
+ AUGCAL
92.87 -ass 6.84
89.25 ta.3
67.80 -mm
(a) GTAV -ittsccppsss (DAFormer).
(b) VisDA SIM2REAL (ViT-B).
> AUGCAL reduces miscalibration post SIM2REAL adaptation. On both GTAV -iitsscapes and VisDA, we find that AUGCAL consistently reduces miscalibration of the base method by reducing overconfidence on incorrect predictions. This is evident in how AUGCAL variants of the base adaptation methods have lower ECE, IC-ECE and OC values (AUGCAL rows, Calibration columns in Tables. 2 (a) and (b)). As an example, to illustrate the effect of improved calibration on real data, in Fig. 3, we show how applying AUGCAL improve the proportion of per-pixel SemSeg predictions that are accurate and have high-confidence (> 0.95).
> AUGCAL improvements in calibration improve reliability. As noted carlier, additionally in- vestigate the extent to which calibration improvements for SIM2REAL adaptation translate to reliable confidence scores via misclassification detection on REAL target data (see Sec. 4), as measured by PRR. We find that AUGCAL consistently improves PRR of the base SIM2REAL adaptation method (PRR columns for AUGCAL rows in Tables. 2(a) and (b)) ensuring that predictions made AUGCAL variants of base model are more trustworthy.
5.2 ANALYZING AUGCAL
We now analyze different aspects of AUGCAL.
> Applying AUGCAL is better than applying just AUG or CAL. Sec. 3.2.1 and 3.2.2, we discuss how AUGCAL can be more effective in reducing target miscalibration than just optimizing for improved calibration on labeled SIM images. We verify this empirically in Tables. 3(a) and (b) for SemSeg and ObjRec. We show that while AUG and CAL, when applied individually, improve over a base SIM2REAL method, they fall short of improvements offered by AUGCAL.
> AUG CAL is applicable across multiple AUG choices. In Sec. 3.2.2 and Table. 1, we show how both PASTA (Chattopadhyay** al., 2023) and RandAugment (Cubuk et al., 2020) are eligible for AUGCAL. In Table. 4(a), we fix DCA as CAL and find that both PASTA and RandAugment are effective in retaining or improving performance, reducing miscalibration and improving reliability.
> Ablating CAL choices for AUGCAL. For completeness, we also conduct experiments by fixing PASTA as AUG and ablating the choice of CAL in AUGCAL. We consider recently proposed training time calibration objectives - Difference of Confidence and Accuracy (DCA) (Liang et al., 2020b), Multi-class Difference in Confidence and Accuracy (MDCA) (Hebbalaguppe et al., 2022) and Margin- Label Smoothing (MbLS) (Liu et al., 2022) - as potential CAL choices (results for SemSeg outlined in Table. 4(b).. We find that while MDCA and MbLS can be helpful, DCA is more consistently helpful across tasks and settings.
objecti
> AUGCAL is applicable across multiple task backbones. Different architectures CNNs and Transformers - are known to exhibit bias towards different properties in images (shape. texture, etc.) (Naseer et al., 20.1). Since the choice of AUG transform (which can alter such properties) is central the efficacy of AUGCAL, we verify if AUGCAL is effective across both CNN and Trans- former backbones. To do this, we conduct our SemSeg, ObjRec experiments with both transformer (DAFormer, ViT-B) and CNN (DeepLabw2-R100,, ResNet- 101) architectures. We find that AUGCAL
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablating AUG and CAL choices in AUGCAL. For a DAFormer model on GTAV Citysccpes. AUGCAL successfully reduces miscalibration and produces reliable confidence scores for SIM2REAL adaptation using both PASTA (P) and RandAug (R) as AUG choices. We also ablate the choice of CAL in AUGCAL across
DCA, MDCA and M and find that DCA is more consistently effective in reducing miscalibration across tasks and settings, ACa. for MDCA and Aca = 0.1, m 10 for MbLS. 1 indicates standard error.
Calibration Error ())
Reliability (1)
Method
AUG
Porf. mloU
Calibration Error
Reliability
Method
CAL
mloU
ECE
IC-ECE
PRR
ECE
IC-ECE
PRR
ErtMin + MIC
65.71
5,34
72.73 noas
45.93
3 EniMin
65.71
5.34
77,73a00
45.93
AUGCAL
DCA
70.31
3/13
72.97
62.66
-AUGCAL
70.31
3443
72.97
AUGCAL
MDCA
69.50
3.22,3:s
72.65
99.96:35
4
AUGCAL
R
70.65
2.34
73.77
66.65 444
AUGCAL
Mbl.
68.77
2,90gm
72.53gm
61,37g3
7
HRDA
75.56
2.86
81.92ae
68 (8.91=m
HRDA + MIC
75.36
2.86gn
81.92
68.91
-AUGCAL
75.90
2.45
79.09
70.35-a51
AUGCAL DCA
75.90
2,45:1
79.09
20.35
AUOCAL
R
74.10
2.77.am
77.94
69.46-a44
AUGCAL MDCA
75.50
2,92-m
80.76
68.45.24
AUGCAL
MbLS
71.15
2.68
70.21
69.33...
(a) Ablating AUG in AUGCAL. (CAL= = DCA).
(b) Ablating CAL in AUGCAL. (AUG = PASTA)
(Rertt Ciyorapas Imagge
HANA MC
HAOM AUG
HADSA MD GAL
HADAMIC AUGGAL
81.97%
80.68%
80.09%
84.03%
E
EnIVI-MC- AUG
EriMir-MC CAL
EMM AUGCAL
69.39%
28%
69.72%
72.72%
'Aocurate" and 'Certain' (Confidaacee 0.96) predictions made by GitysCCyysapess adlapted model
road
sidewalk
building
wall
fence
pole
traffic Igt
traffic sga
vegetation
ignrrdd
lerruin
sky
person
rider
cur
truck
bus
train
motorcycle
bike
Figure 3: AUGCAL increases the proportion of "accurate" and "certain" predictions. For (DAFormer)
HRDA + MIC (row 1) and EntMin + MIC (row 2) GTAV Cityssappss, we show how different interventions affect the proportion of "accurate" and "certain" (confidence >> 0.95) predictions (indicated in gray per column). Regions in black do not satisfy the "accurate" and "certain" filtering criteria. We see that compared to a base adaptation method, AUGCAL increases the proportion highly-confident correct predictions (green boxes). AUG and CAL applied alone can potentially reduce that proportion (yellow boxes). AUG is PASTA, CAL is DCA. (with PASTA as AUG and DCA as CAL) is effective in reducing SIM2REAL miscalibration across all settings. We discuss these results in Sec. A.7 of appendix.
b How does AUGCAL compare with temperature scaling? While we focus on "training-time" also conduct an experiment to compare AUGCAL with
patches to improve SIM2REAL calibration. (TS) on VisDA. Specifically, we use 80% of VisDA SIM images for training models and rest (20%) for validation and temperature tuning. To ensure a fair comparison, we consider temperature tuning on both "clean" (C) and "PASTA augmented" (P) val splits. We find that irrespective of tuning C or P. unlike AUGCAL, TS is ineffective and increases overconfidence and miscalibration. We present these results in Sec. A.7 of appendix.
"post-hoc" temperature scaling
> AUGCAL increases the proportion of accurate and certain predictions. In Fig. 3, we show qualitatively for GTAV -ityssapps SemSeg how AUGÇAL increases the proportion of highly- confident correct predictions. In practice, we find that this improvement is much more subtle for stronger SIM2REAL adaptation methods, such as HRDA + MIC, compared weaker ones, such as EntMin + MIC, which have considerable room for improvement.
CONCLUSION
We propose AUGCAL, method to reduce the miscalibration of SIM2REAL adapted models, often caused due to highly-confident incorrect predictions. AUGCAL modifies a SIM2REAL adaptation framework by making two minimally invasive changes (1) augmenting SIM images via AUG transformations that reduce SIM2REAL distance and (2) optimizing for an additional calibration loss on AUGmented SIM predictions. Applying AUGCAL to existing adaptation methods for semantic segmentation and object recognition reduces miscalibration, overconfidence and improves reliability of confidence scores, all while retaining or improving performance on REAL data. AUGCAL is meant to be task-agnostic, general purpose framework to reduce miscalibration for SIM2REAL adaptation methods and we hope such simple methods are taken into consideration for experimental settings beyond the ones considered in this paper.
9

---

Published as a conference paper at ICLR 2024
Acknowledgements. This work has been partially sponsored by NASA University Leadership Initiative (ULI) #80NSSC20M0161, ARL and NSF #2144194.
REPRODUCIBILITY STATEMENT
We provide training, implementation and optimization details associated with our experiments in Sec. A.5 of the appendix. Most of our experiments follow the implementations of SIM2REAL adaptation methods from (Hoyer et al., 2022c), with our additional modifications on top. Additionally, in Sec. A. of appendix, we provide details surrounding the assets (and corresponding licenses) used for our experiments. Assumptions surrounding the analytical justification behind AUGCAL (Sec. 3.2.1 and 3.2.2) have been presented in the same sections.
ETHICS STATEMENT
Our proposed patch, AUGCAL, is meant to improve the reliability of SIM2REAL adapted models. We assess reliability in terms confidence calibration (prediction scores aligning with true likelihood of correctness) and the extent to which calibrated confidence scores are useful for assessing prediction quality (measured via classffasiiia detection). AUGCAL adapted models have promising consequences for downstream applications. A well-calibrated and reliable SIM2REAL adapted model can increase transparency in REAL predictions and facilitate robust decision making in safety-criticas scenarios. That said, we would like to note that while AUGCAL is helpful for our specific measures of reliability, exploration along other domain specific notions of reliability remain.
REFERENCES
Slanca-Cerasela-Zele Blaga and Sergiu Nedevschi. Semantic segmentation learning for autonomous uavs using simulators and real data. In 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 303-310, 2019. doi: 10.1 1099ICC4444
3019.8959563.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.0961 2021.
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnaw: Towards benchmarking robustness embodied navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15691-15700, 2021.
Prithvijit Chattopadhyay** Kartik Sarangmath* Vivek Vijaykumar, and Judy Hoffman. Pasta: Proportional amplitude spectrum training augmentation for syn-to-real domain generalization. In Proceedings of the IEEF/VV International Conference on Computer Vision (ICCV), pp. 19288- 19300, October 2023.
Liang- Cang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 2017.
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selec- tive whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11580-11590, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In NIPS, 2010.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
10

---

Published as a conference paper at ICLR 2024
Pau de Jorge, Riccardo Volpi. Philip Torr, and Gregory Rogez. Reliability in semantic segmentation: Are we on the right track? arXiv preprint arXiv:2303.1 1298, 2023.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-reat embodied ai platform. In CVPR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 2009. doi: 10.1 109/CVVR.2009.5506888.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19929, 1929, 2020.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in Data Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020, pp. 877-894, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv
preprint arXiv: 1409.. 7495, 2014.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.055 107,03342, 2021.
Yunye Gong, Xiao Lin, Yi Yao, Thomas G Dietterich, Ajay Divakaran, and Melinda Gervasio. Confidence calibration for domain generalization under covariate shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9958-8967, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Ramya Hebhalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16081-16090, 2022.
Judy Hoffman, Eric Tzeng. Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexci Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pp. 1989-1998, 2018.
Lukas Hoyer. Dengxin Dai, and Lue Van Gool. Daformer:
Improving network architectures and training strategies for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9924-9935, 2022a.
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Hrda: Context-aware high-resolution domain- adaptive semantic segmentation. In Computer Fision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pp. 372-391. 372-331..pringerr 2022b.
Lukas Hoyer, Dengxin Dai, Haoran Wang, and Luc Van Gool. Mic: Masked image consistency for context-enhanced domain adaptation. arXiv preprint arX11:22/2.01322. 2022c.
11

---

Published as a conference paper at ICLR 2024
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomiza- tion for domain generalization. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.
Mobarakol Islam, Lalithkumar Seenivasan, Hongliang Ren, and Ben Glocker. Class-distribution- aware calibration for long-tailed visual recognition. arXiv preprint arXiv:2109.05263 2021.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 464-480. Springer, 2020.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann, Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4893-4902 2019.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 01129768-777777 2021.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything, arXiv preprint arXiv:2304. 02643,26333 024404334.0 2023.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics, pp. 623-631. PMLR, 2017.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805-288 14. PMLR, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Gongbo Liang, Yu Zhang, and Nathan Jacobs. Neural network calibration for medical imaging classification using dca regularization. In International Conference Machine Learning (ICML), 2020a.
Gongho Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved trainable calibration method for neural networks medical imaging classification. arXiv preprint arXiv:2009. 2000:000000057 2020b.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin- based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80-88, 2022.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In NeurIPS, 2018
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. arXiv preprint arXiv: /9055 000.1990500000 2019.
12

---

Published as a conference paper at ICLR 2024
Rhiannon Michelmore, Marta Kwiatkowska, and Yarin Gal. Evaluating uncertainty quantification in end-to-end autonomous driving control. arXiv preprini arXiv: 181106177, 2018.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:1 14:11562-11 15694,, 2021.
Muhammad Akhtar Munir, Muhammad Haris Khan, M Sarfraz, and Mohsen Ali. Towards improving calibration in object detection under domain shift. Advances in Neural Information Processing Systems, 35:38706-388 18, 2022.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuad Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-23308, 2021.
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pp. 80-111. Springer, 1981
Anusri Pampari and Stefano Ermon. Unsupervised calibration under covariate shift. In 1004452006.16405, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: /ttt:::/pppprs..nete wewripppps....irsppppppppppp.p.p www.ine.riiipom http:/ssereiiip..ccpppppppesspes cc/ppprr// wwwpppytooocc---iimerre - -eresseteeee-e--e-frorreeeeeeeee
pdf.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda:
The visual domain adaptation challenge. arXiv preprint arXiv:1710.0692: 2017.
Nicolas Pinto, David D Cox, and James J DiCarlo. Why real-world visual object recognition hard?
PLoS computational biology, 4( 4(1))e27, 2008.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakrish- nan. A closer look smoothness in domain adversarial training. In International Conference on Machine Learning, pp. 18378-18399, PMLR, 2022.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-1 182.118. Springer, 2016.
Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun. Enhancing photorealism enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5(2):1700-171) 2022.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234-3243, 2016.
Alfréd Rényi. On measures of information and entropy. In Proceedings of the 4th Berkeley Symposium Mathematies, Statistics and Probabilit, 1960.
13

---

Published as a conference paper at ICLR 2024
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regulariza- tion. In International Conference on Learning Representations, 2018.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Juliam Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019.
Guardian Tesla Crash. Tesla driver dies in first fatal crash while using autopilot mode - theguardian com. https://www.. theguardian wwt.cooo//cc/ceee honm:technology/21666600 //n////////////////
musk, 2016. [Accessed 16-May-2023].
Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1379-1389, 2021.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7167-7176, 2017.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2517-2526, 2019.
Bailin Wang, Mirella Lapata, and Ivan Titov. Meta-learning for domain generalization in semantic parsing. arXiv preprint arXiv:2010.11998 2020a.
Chong Xiao Wang and Wee Peng Tay. Practical bounds of Kullback-leiblee. divergence using maximum mean discrepancy. arXiv preprint arXiv:2204. 02031, 2022.
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks:
Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34: 11809-11820, 2021.
Dongdong Wang, Boqing Gong, and Liqiang Wang. On calibrating semantic segmentation models: Analysis and an algorithm. arXiv preprini arXiv: 12053. 2022.
Ximei Wang, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable calibration with lower bias and variance in domain adaptation. Advances in Neural Information Processing Systems, 33: 13:19212-19223, 2020b.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 120777727-2099 2021.
Yuchen Zhang. Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Leurning, pp. 7404-7413, 2019.
Yuyang Zhao, Zhun Zhong, Na Zhao, Nieu Sebe, and Gim Hee Lee. Style-hallucinated dual consis- tency learning for domain generalized semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via on computer vision (ECCV), pp. 289-305, 2018.
self- eeff-training. In Proceedings of the European conference
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 OVERVIEW
This appendix is organized as follows. In Sec. A.4. we provide more details on the AUG choices for AUGCAT- PASTA and RandAugment. Sec. A.5 outlines training, implementation details and choice of hyperparameters. Then, in Sec. A.6, we discuss Prediction Rejection Ratio (PRR), the metric used to assess reliability in our experiments. In Sec. A.7, we provide AUGCAL results for CNN backbones, comparisons with temperature scaling and discuss sensitivity ACAl (coefficient of LCAL). Then, we provide more supporting qualitative examples in Sec. A.8. In Sec. A.9, we describe the SIM2REAL adaptation methods we in detail. Finally, Sec. A.10 summarizes the assets used for our experiments and their associated licenses.
A.2 AUG CHOICES
A.3 PAsTA
We use Proportional Amplitude Spectrum Training Augmentation (PASTA) (Chattopadhyay" et al., 2023) as one of the AUG choices AUGCAL. PASTA applies structured perturbations (controlled by hyper-parametent a, 3, k) to the amplitude spectra of synthetic images to generate augmented views. For single-channel image, R H (for illustration purposes), the augmentation process in PASTA is outlined below. We refer the reader to (Chattopadhyay* et al., 2023) for more details.
1. Set a = 3.0, 8 = 0.25, k = 2 for PASTA
2. Use FFT (Nussbaumer, 1981) to obtain the Fourier spectrum of synthetic image r, as F(x) = FFF()) E C#*
3. Obtain the corresponding amplitude and phase spectra as A(x) = Abs(F(x)) and P(x) =
Ang(F((x:) respectively.
4. Zero-center the amplitude spectrum, as A(x) FFTShift( to get the lowest frequency components the center.
5. Define perturbation strength, a E Hx as o(m, n] = (2ay)
m²+
+ 3, where m, n
H²-W3
denote the spatial frequencies.
6. Sample perturbations using the perturbation strength as, Nf(1,02)
7. Perturb the amplitude spectrum A(x) = A(z)
8. Reset the low-frequency components, A(x) FFTShift(A(z))
9. Obtain the augmented synthetic image via inverse FFT as, iFFT(((())) P(x))
We set PASTA hyper-parameters as a 3.0, 0.25, k 2 it seems to work well across multiple SIM2REAL shifts in practice. We use Pytorch (Paszke et al., 2019) to vectorize all operations in the steps above and apply PASTA on minibatches, instead of individual synthetic images.
A.4 RANDAUGMENT
RandAugment (Cubuk et al., 2020) generates augmented views by applying a series of transforms sampled from predefined vocabulary. While the vocabulary includes geometric as well as photomet- ric transforms, we only use photometric transforms for our experiments. For SemSeg, the models we train already use geometric transforms that are optimal for SIM2REAL SemSeg performance. For ObjRec, we find it empirically beneficial to fix geometric transforms to a CenterCrop and use photometric transforms from RandAugment. These choices follow (Hoyer al., 2022c). These oper- ations are AutoCont rast, squaliie, Contrast, Br Brightness, Sharpness, Poster ize, Solarize and SolarizeAdd. We (m 30, n 8) while sampling operations from this vocabulary for RandAugment. We refer the reader to (Cubuk et al., 2020) for more details.
A.5 TRAINING AND IMPLEMENTATION DETAILS
We outline training and implementation details associated with our experiments.
Semantic Segmentation. For SemSeg, we use GTAV (Sankaranarayanan et al., 2018) as the source SIM dataset and Cityscapes (Cordts et al., 2016) the target real dataset. We consider two
15

---

Published as a conference paper at ICLR 2024
Table 5: Reproducing SIM2REAL Adaptation Baselines, We summarize results from our reproduction of the SIM2REAI. adaptation baselines (from (Hoyer et al., 2022c)) used in our experiments. We find slight difference between reported and reproduced adaptation results across models on both (a) GTAV -TAV-+Cityscappe and (b) VisDA syn-treal. + indicates standard crror.
Method
Model
Status
mloU (t)
Method
Model
Status
mAce (1)
1 HRDA + MIC DecpLabv2 (Hoyer et al., 2022c)
64,20
1 SDAT MIC ResNet-100 101 (Hoyer et al., 2022c)
86,90
2 HRDA + MIC
DeepLabvv2
Reproduced
64.05
2 SDAT + MIC ResNet-101
Reproduced
81.03
3 HIRDA + MIC DAForee (Hoyer et al., 2022c)
75.90
3 SDAT MIC
ViT-B
(loyer el al., 2022c)
92.80
HRDA + MIC DAFormer Reproduced
75.56
4 SDAT MIC
VIT-B
Reproduced
92.62+a0s
(a) GTAV + Cityscapes.
(b) VisDA SIM2REAL.
segmentation architectures for our experiments - DeepLabv22 (Chen et al., 2017) (with a ResNet- 101 (He et al., 2016) backbone) and DAFormer (Hoyer et al., 2022a) (with an MiT-B5 (Xie et al., 2021) backbone). Both models are initialized from backbones pretrained on ImageNet (Deng et al., 2009). For EntMin (Vu et al., 2019) (with DAFormer/ / DeepLabv2), we use SGD as optimizer with a learning rate of 2.5 x 10-4 and use AUDA 0.001 as the coefficient of the "unconstrained" entropy loss. For HRDA, we use the multi-resolution self-training strategy from (Hoyer et al., 2022b) -AdamW (Loshchilov & Hutter, 2017) as the optimizer with a learning rate of 6 x 10-3 for the encoder and 6 x 1 for the decoder, with a linear learning rate warmup (warmup iterations 1500; warmup ratio 10 10-9), followed by polynomial decay (to an eventual learning rate of 0). For the teacher-student self- training setup in HRDA, we additionally couple cross-domain mixing with augmentations (DACS (Tranheden al., 2021)) to improve self-training, use the ImageNet feature distance loss (Hoyer et al., 2022a), set AU = 1 and use a = 0.999 as the teacher EMA factor. For all SemSeg settings, we use batch size of 2 (2 source images, 2 target images) and additionally use rare class sampling (based on source; following (Hoyer et al., 2022a)) to ensure consistent adaptation improvements across all classes. For all our SemSeg experiments, as input to the models, irrespective of the adaptation method. PASTA for AUGÇAL is applied to the same crops. We train all segmentation models for 40k iterations and use the last checkpoint to report results (the SIM2REAL adaptation setting does not assume access to labels on real data which prevents selecting "best-on-target" checkpoint). For semantic segmentation, following prior work (Wang et al., 2022), for ECE (and other associated metrics), instead of pooling all pixels of different images into set, we compute per-image numbers and then average across images. We use the same to compute standard error. We report all metrics (performance, calibration, etc.) as percentages. We use 15 bins to compute ECE.
feed crops of size 1024 x 1024
Object Recognition. For ObjRec, we conduct experiments on the VisDA (Peng et al., 2017) SIM2REAL benchmark, using standard source-target splits (Hoyer et al., 2022c) for training. We con- sider ImageNet (Deng et al., 2009) pretrained ResNet-101 (He al., 2016) and VIT-B/I6 (Dosovitskiy et al., 2020) backbones with standard bottleneck (with Linear, BatchNorm and ReLU) and classifier layers. As noted earlier, we use SDAT (Rangwani et al., 2022) as the adaptation method, which relies on Conditional Domain Adversarial Adaptation (CDAN) (Long et al., 2018) and Minimum Class Confusion (MCC) (Jin al., 2020) with SAM (Foret et al., 2020) (smoothness 0.2) all adaptation losses are combined and AA is set to 1. We use SGD as the base optimizer with a learning rate of 2 x 10 with batch size of 32 (for both source and target). PASTA for AUGCAL is applied to the raw input SIM images. We train ResNet-101 101 backbones for 30 epochs and ViT-B/16 backbones for 15 epochs and use the last checkpoint to report results (the SIM2REAL adaptation setting does not assume access to labels on real data which prevents selecting -n-ttee"t checkpoint). We report all metrics (performance, calibration, etc.) as percentages. We use 15 bins to compute ECE. For our key results in the main paper (Tables. 2(b) & 3(b)), we run experiments across 3 random seeds.
MIC Hyperparameters. We also use MIC (Hoyer et al., 2022c) with all the discussed SIM2REAL adaptation methods since demonstrably improves performance. For MIC, following prior work (Hoyer et al., 2022c), we use a masking patch size of 64, a masking ratio of 0.7, a loss weight of 1 and an EMA factor of 0.999 for the pseudo-label generating teacher.
Compute. We conduct all object recognition experiments on RTX 6000 GPUs - every experiment requiring single GPU. For semantic segmentation, we use one A40 GPU per experiment.
16

---

Published as a conference paper at ICLR 2024
Table 6: AUGCAL ensures SIM2REAL adapted models make accurate, calibrated and reliable predictions. We find that applying AUGCAA to multiple SIM2REAI. adaptation methods across tasks leads better calibration
(ECE, IC-ECE), reduced overconfidence (OC) and improved reliability (PRR) - all while retaining or improving transfer performance. Highlighted rows are AUGCAL variants of the hase methods. For AUGCAL, we use PASTA
as AUG and DCA as CAL. 1 indicates standard error.
Method
Perf. (t)
Calibration Error (L)
Reliabälty (?)
EmiMin MIC
mlaU
47.77
493arm 20.09 a
ECE IC-ECE
OC
PRR
Method
Perf. (1) mioU
Calibration Error ())
IC-ECE
OC
Reliability (?)
PRR
2 + ABGCAL
49.96
3.2 67.9 74.33 am
52.07 84m
SDAT + MIC 81.03a2 14.32ar in 85 15.06 (0.0018...72000 82.77an
46.92 46.92
HRDA MIC
64.05
3.52am 78.94.s _6.48_a
63.20sea
AUGCAL 84 11.9 82.98 81.10000
53,26
+ AUGCAL 63,95
2.58 74.71 84.13
65.37
(b) VisDA SIM2REAL. (iesNet-10))
(a) GTAV +Cityscapes. (DeepLabv2 R-101).
Table 7: Comparing AUGCAL with Temperature Scaling for VisDA SIM2REAL. We do controlled
experiment (with an 80-20 split of the VisDA SIM2REAL split) compare AUGCAL with Temperature Scaling (TS). B SDAT + MIC (ViT-B). (clean) and (PASTA augmented) indicate the synthetic "labeled" validation splits the temperature was tuned on. For AUGCAL, we use PASTA as AUG and DCA as CAL. + indicates
standard error.
Method
Pert. mlaU
Calibration Error ())
IC-ECE OC
Reliability (?)
PRR
Method
Perf. (*) Calibration Error (i)
ECE
IC-ECE
Reliubility PRR
1 SDAT + MIC (8) 92 24 8.13 91. 24 89 482
63 63.92a14
SDAT MIC (BB 92.24.as 8. 1.13ss 91 54gaie 89 48
63.92
1 TS (C)
92.24..0
8.19 95.41 93.97
66.58
AUGCAL
92.68
7.24.on 90.40 we 87.57 iasi
66.49
1B+
92.24
8.54 99.5522.0 92.00.uz
64.05.1.
+TS
92.68.Au 8.01 sas 94,4 .2.7
10
2 AUGCAL
92.68 7.24 90.40 2M
66.49..4
+ TS (P)
92.68_an 8 8.022nn 94,46am 92, 80
66.05
(b) AUGCAL vs Temperature Scaling.
(b) AUGCAL + Temperature Scaling.
Reproduced Results. We use open-sourced code for (Hoyer et al., 2022c)² and re-run the base adaptation methods for HRDA + MIC and SDAT + MIC at our end to obtain baseline results. In Table 5, we summarize reported and reproduced results for the same methods.
A.6 PREDICTION REJECTION RATIO (PRR)
As noted in Sec. 4 of the main paper, in addition to measuring reduced miscalibration, we also assess the extent to which such improvements in calibration are useful and lead to a more reliable model. To measure this, following prior work (de Jorge et al., 2023; Malinin et al., 2019), we measure whether confidence scores can reliably guide misclassification detection - measured via the PRR metric.
Ideally, in a real-world scenario, given a model, we would like to retrieve all (potentially) samples misclassified by the model based on confidence scores. These samples can then be skipped when the model is used for decision decision-making (since the model is likely to make incorrect predictions). Since confidence scores, practice, are imperfect, measuring misclassification detection helps us assess this specific capability of the SIM2REAL adapted models. For a model that is less overconfident on incorrect samples (meaning has reduced miscalibration), this specific ability should naturally be enhanced.
This can be measured using Egjection-Accuralt curves (de Jorge et al., 2023; Malinin et al., 2019) where we reject samples below a threshold and keep track of accuracy and the fraction of rejected samples. Since such eurves are naturally biased towards models that have improved performance, the AUC for such a rejection curve can be normalized by that of an oracle. Additionally, we can subtract score associated with random bascline (sorting predictions for filtering in random order) (Wang et al., 1). Finally, we can compare this value (PRR; ranging from -100 to 100; higher is better) for multiple models to assess how reliable underlying confidence scores are.
A.7 AUGCAL RESULTS
> AUGCAL results with CNN architectures. Key results presented in Table. 2 of the main paper use transformer architectures (DAFormer for SemSeg. ViT-B for ObjRec). In Tables. (a) and (b), we verify that AUGCAL improvements translate to CNN based architectures as well (also discussed in Sec. 5.2 of the paper). We use DeepLabv2 (ResNet-101) for SemSeg and a ResNet- 101 based classifier for ObjRec and find that applying AUGCAL improves or retains performance, reduces miscalibration and improves reliability.
& Comparing AUGCAL with Temperature Scaling (TS). While we focus on "training-time" patches to improve SIM2REAI. calibration, we also conduct an experiment to compare AUGCAL with "post- hoc" temperature scaling (TS) on VisDA. Specifically, we use 80% of VisDA SIM images for training
Phups//ggihuh.co//hher//tt
17

---

Published as a conference paper at ICLR 2024
Table 8: Sensitivity to Aca for AUGCAL on GTAV -itys-eis.s...es..
vary the value of ACAL {0.1, 0.5, 1. 1.0, 5.0, 10.0, 20.0, 100.0) and report the effect on adaptation performance, reduced miscalibra- tion and improved reliability. We find that our choice of ACai. = leads to balanced performance across desired metrics. Rows in red correspond to the baseline SIM2REAL adaptation method without the application of
AUGCAL.
Perf. (1)
Calibration Error (.) Reliability (?)
Perf. (?)
Calibration Error ()) Reliability (1)
Value of Acas.
Value Acu.
mloU
ECE
IC-ECE
PRR
mloU
ECE
IC-ECE
PRR
AUCCAL
65.71
77.73
45.93
AUGCAL
75.56
2.86
81.92
68.91
Aca. 0.1
69.72
2.88
74.55
57.06
Apa. 0.1
75.25
2.94
80.68
69.96
3 Aca. 0.5
69.27
2.71
73.54
60.32
3 Apa. 0.5
75.05
2.77
80.23
70.62
4 Aca
72.97
62.66
ACAL
2.45
5 Aca. 5.0
66.24
3.55
70.22
62.66
5 Aça. 5.0
73.80
2.25
76.61
70,54
8 ACa. 10.0
64.27
3.28
67.63
64.55
6 ACAL 10.0
71.28
2.50
75.85
69.17
7 ACaL 20.0
55.37
4.17
65.55
63.20
7 ACAL
20.0
62.31
2,43
73.32
68.27
8 ACac 100.0
27.73
6,06
52.24
54.47
8 ACAL
100.0
31.01
7.68
73.56
56.52
(a) EntMin + MIC (DAFormer).
(a) HRDA + MIC (DAFormer).
models and rest (20%) for validation and temperature tuning. To ensure fair comparison, consider temperature tuning on both "clean" (C) and "PASTA augmented" (P) val splits. We present these results in Table. 7 (a). We find that irrespective of tuning on C or P, unlike AUGCAL, TS is ineffective and increases overeonfidence and miscalibration. In Table. 7, we additionally consider temperature scaling on the logits of an AUGCAL improved SIM2R model and find that it worsens miscalibration and overconfidence. Note that this is not entirely surprising since TS depends heavily on the "data-split" the temperature is tuned on, which in our case is SIM and not REAL.
> Sensitivity to ACAL. For an existing SIM2REAL adaptation pipeline, AUGCAL involves optimizing for an additional calibration loss LCAL on augmented synthetic images. In Table. 8, we vary the coefficient ACAL (values in the set {0.1, 0.5, 1.0, 5.0, 10.0, 20.0, 100.0)) for LCAL and note the effect on adaptation performance, calibration on real data (ECE, IC-ECE) and reliability (PRR). In Table 2 of the main paper, we already note how applying AUGCAL with ACAL 1 improves over baseline adaptation method (red and blue rows in Table. 8). We further note that compared to other values, our choice of ACAL 1.0 achieves balance between adaptation performance, reduced miscalibration and improved reliability. We find that overly high values of ACAL 2 5 can potentially lead to reduced adaptation performance ACAL 5 significantly raises the scale of LCAL compared to LCE and LUDAA which leads to models optimizing for improved calibration at the expense of task performance. Based on our experiments across multiple models, shifts and tasks, we recommend restricting ACAL < for SIM2REAL adaptation.
A.8 QUALITATIVE PREDICTIONS
In Figures 4 and 5, we provide more examples to demonstrate how AUGCAL improves the proportion of "accurate" and "certain" predictions. In Fig. 4, where we compare predictions for a (DAFormer) HRDA + MIC model w/o AUGCAL 75.56 mloU and w AUGCAL 75.90 mloU. We find that applying AUGCAL improves the proportion of "accurate" and "certain" predictions (confidence 0.95) see Fig. 4, columns and 5, guided by yellow arrows. For instance, we find that the base model (w/o AUGCAL) has trouble assigning high-confidence to correct "sidewalk" and "vegetation" predictions. For EntMin + MIC (w/o AUGCAL 65.71 mloU and w AUGCAL. 70.31 mloU) in Fig. 5, find that AUGCAL is considerably more effective in ensuring high-confidence correct predictions. Notably, we find these improvements to be more subtle as the SIM2REAL adaptation method itself improves (HRDA + MIC > EntMin MIC).
A.9 SIM2REAL ADAPTATION METHODS
We conduct experiments with three SIM2REAL adaptation methods across two tasks. We wanted to assess the compatibility of AUGCAL with SIM2REAL adaptation methods that are competitive and are representative of the broader class of approaches used for SIM2REAL transfer. We first cover some background and describe these methods in detail.
NNote that temperature tuning (by definition) only affects calibration and has impact on performance.
18

---

Published as a conference paper at ICLR 2024
"Accurate" and "Certain" Predictions
"Accurate" and "Certain" Predictions
Target Image
Predictions
Predictions
(DAFormer) HRDA MIC
(DAFormer) HRDA + MIC AUGCAL
Figure 4: AUGCAL increases the proportion of "accurate" and "certain" predictions. For base DAFormer SemSeg model trained with HRDA MIC (State-of-the-art) on GTAV--Ciysscapes, we show how applying AUGCAL training time (right) can improve proportion "accurate" and "certain" (confidence 0.95) predictions over the vamilla adaptation method (left). Regions in black do not satisfy the "accurate" and "certain" filtering criteria.
In unsupervised domain adaptation (UDA) for SIM2REAL settings, we assume access to labeled (SIM) source dataset Dg ((= ) and unlabeled (REAl. target dataset Dr {- [T] We assume Ds and D, splits are drawn from source and target distributions p8 (x: y) and pd (xx y) respectively. At training, we have access to = Ds U Dr. We operate in the setting where source and target share the same label space, and discrepancies exist only in input images. The model Mg is trained on labeled source images using cross entropy,
Is Σ v = Σ of log po (0 |2 where of arg max (o xx{) 1=1
[8|
(14)
yEY
For object recognition, where ) (1, 2, ..., K}. For semantic segmentation, since we make predictions across a vocabulary of classes for every pixel. the corresponding label can be expressed as y E "
represent the labels corresponding to images E H E y
UDA methods additionally optimize for an adaptation objective on labeled source and unlabeled target data (A). The overall learning objective be expressed as,
min Con(u, N
AunACuDAleT
(15)
Source Loss
Source Target Adaptation Loss
Different adaptation methods usually differ in terms of specific instantiations of this objective. As adaptation methods, we use HRDA (Hoyer et al., 2022b) and EntMin (Vu et al., 2019) for segmentation and SDAT (Rangwani al., 2022) for recognition.
Entropy Minimization (EntMin). We consider the unconstrained direct entropy minimization approach from (Vu et al., 2019) one of the adaptation methods. EntMin builds top of the assumption the models trained only on source data tend to be under-confident (make high-entropy predictions) target images. EntMin enforces high prediction certainty target images by ensuring
19

---

Published as a conference paper at ICLR 2024
Target Image
Predictions
"Accurate" and "Certain" Predictions
Predictions
"Accurate" and "Certain" Predictions
(DAFormer) EntMin + MIC
(DAFormer) EntMin + MIC AUGCAL
Figure 5: AUGCAL increases the proportion of "accurate" and "certain" predictions. For base DAFormer SemSeg model trained with EntMin MIC on GiyyssCiysscaess, time (right) can improve the proportion of "accurate" and "certain" (confidence > 0.95) predictions vanilla adaptation method (left). Regions in black do not satisfy the "accurate" and "certain" filtering criteria.
show how applying AUGCAL training
that the model makes high-confidence predictions on the same. This is realized by minimizing the normalized entropy of target predictions. Specifically, given target image x²' if the predictive probabilities per-pin per-pixel are expressed as P (hh,0,8)) E [0, 1] xx x the adaptation loss on target can
be expressed
LUDA(
Σ 1=1
P (k) log P (l.w.))) xT
(16)
HW
log
K
h,w
EntMin can also be viewed ssoft-assignment version of self- self-training (Lee al., 2013; Zou et al., 2018). We refer the reader (Vu al., 2019) for more details.
Adapive Semantic Segmentation (HRDA). The base adapation method in HRDA (Hoyer al., 2022b) self-training (Zou et al., 2018; Lee al., 2013). HRDA additionally introduces components that are beneficial for domain-adaptin segmentation. Specifically, HRDA proposes multi-resolution framework by relying on (1) low- resolution context crоp to learn long-range contextual dependencies and (2) a high-resolution detail crop make detailed and accurate predictions. During adaptation, HRDA fuses predictions from both crops using input dependent attention. For a target image xt pseudo-labelt are obtained from teacher network Ms (a moving average of Ma) as P (h),
Context-Aware High-Rec Sigh-Resollutin
semantic
arg max, Mo(xT). HRDA adapts to
target by minimizing cross entropy w.r.t. P (h.w)) as,
LUDA(æ"; ;0) = = -- PerPPl
(hh.w) aT
(h.w)
log Mo(x") with far being a quality estimate for P (h,w)
(17)
h,w
Instead of "trff-ttaaniig on target images, HRDA uses cross domain mixing (DACS (Tranheden al., 2021)) to obtain pseudo-labels on augmented images. Additionally, components from DAFormer (Hoyer et al., 2022a) rare class sampling and an ImageNet feature distance loss incorporated in the pipeline facilitate better adaptation. The quality estimate g,r is computed as the proportion of pixels that have confidence above a specified threshold. This naturally ensures a warmup stage, where model is first trained only synthetic images for few iterations, followed
20

---

Published as a conference paper at ICLR 2024
by training on both synthetic and real images. We refer the reader to (Hoyer al., 2022b) for more low-level implementation details.
Smooth Domain Adversarial Training (SDAT). SDAT (Rangwani et al., 2022) is an adaptation method for object recognition. The underlying adaptation method in SDAT domain adversarial training (DAT), which involves reducing the discrepancy between source and target image distribu- tions. This is realized by confusing an additional discriminator that is designed distinguish between source and target samples. While multiple versions of DAT exist, SDAT uses CDAN (Long et al., 2018) as the default DAT method. SDAT investigates the lass-landssspes of DAT style methods and notes that smoother loss-landscapes on source data result in improved transfer to target. Consequently, SDAT proposes optimizing for smoother loss landscapes on labeled source data by modifying the supervised LCE loss as;
[8]
|s| Σ ==1
CC :) Σ
y max log po+e (0 lx;))
(18)
|ells
Perturbing the weights of the network @ by c in some neighborhood ensures lower loss values in that neighborhood, thereby encouraging smoother loss landscape. In practice, this is realized by using sharpness aware minimization (SAM) (Foret et al., 2020) to update model parameters. We refer the reader to (Rangwani et al., 2022; Long et al., 2018) for more details on the UDA losses used.
general technique applicable to any existing adaptation method to improve context utilization while making predictions. MIC involves a teacher-studens self-training setup where pseudo-labels are obtained from a weakly augmented target sample (via the teacher). Student predictions for masked image are forced to match the obtained pscudo-labels. MIC relies on the recent success of masking an auxiliary task (He et al., 2022) but instead of reconstruction, MIC sets up a prediction consistency task. This simple addition an existing adaptation pipeline leads considerable improvements across multiple tasks and shifts. A "masked" image for MIC is obtained by dividing the original image into patches and randomly masking a subset of the same. Similar to HRDA (Hoyer et al., 2022b), the MIC (consistency) loss is multiplied by a quality estimate of the pseudo-labels. We refer the reader to (Hoyer et al., 2022c) for more details on MIC.
Masking Image Consistency (MIC). MIC (Hoyer et al., 2022c)
A0 ASSETS AND LICENCES
The assets used in this work can be grouped into three categories Datasets, Code Repositories and Dependencies. We discuss source and licences for each of these below.
Datasets. For semantic segmentation, we use the GTAV (Richter et al., 2016) and Cityscapes (Cordts et al., 2016) datasets. Code used to extract densely annotated images from the GTAV game is distributed under the MIT license.5 5 The Cityscapes' license agreement dictates that the dataset is made freely available academic and non-academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation and that permission to use the data is granted under certain conditions. For object recognition, we use the VisDA syn-reeal (Peng et al., 2017) benchmark. The VisDA-C development kit on github does not have a license associated with it, but it does include Terms of Use, which primarily states that the dataset must be used for non-commercial and educational purposes only.
Code Repositories. For our experiments, apart from code that we wrote ourselves, we build on top of the open-sourced codebase for MIC (Hoyer et al., 2022c) repository. MIC distributed under the MIT License.
Dependencies. We use Pytorch (Paszke et al., 2019) as the deep-learning framework for all our experiments. Pytorch, released by Facebook, is distributed under a Facebook-specific license." 9
com http://ghhhccoo//iiiiooaaaaaaiie
"https://githubl https://gihuh..oo/llyyyytt hwtp:h//..oooooo/////..C.C
o "htps://gttuuh.coo// http:///gti..ccmmooooooooeeeoaen
21


Reviews:
[{"review_id": "pAMxYNS1Xz", "reviewer": "Reviewer_iaFo", "summary": "This paper proposes AUGCAL to improve the confidence calibration and model reliability of existing unsupervised domain adaptation approaches. Specifically, the authors propose to replace the original images with strongly augmented ones and additionally optimize for calibration loss on the augmented images. Detailed analytical justification has been derived to show how AUGCAL can reduce miscalibration on real data. Extensive experiments on different datasets and different UDA backbones have been conducted to show the effectiveness of AUGCAL.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Extensive experiments on different tasks, datasets and UDA backbones are provided in the paper to evaluate the proposed method. \n\n2. Theoretical derivations have been provided to motivate the design of AUGCAL.\n\n3. The proposed AUGCAL is simple, which only introduces two small changes to the existing UDA pipelines.", "weaknesses": "1. The technical contribution of this paper is a bit limited. The key components of AUGCAL are the data augmentation and DCA based calibration loss. However the data augmentation technique is commonly used in domain adaptation and the DCA proposed by previous work for model calibration.\n\n2. The proposed method is only compared with diffident backbone UDA models. A more convincing comparison would be to compare with existing confidence calibration methods, such as [Wang et al. 2022], [Gong et al. 2021].\n\n3. The paper exceeds the page limit of 9 pages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DV2cwv9n6b", "reviewer": "Reviewer_dFNn", "summary": "This paper starts with a nice theory demonstrating that to achieve better calibration loss in target domain, one should minimize the miscalibration in source domain and reduce the distributional distance between source and target domains. Then, to address this, the paper proposes AUGCAL, which augments the source data and apply a calibration loss on it. Experiments demonstrate the effectiveness of the proposed method.", "questions": "please see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theory is nice and understandable.\n2. The paper is well-written and experiments are extensive.", "weaknesses": "1. Usually adding calibration loss will have lower ECE but also lower accuracy (in your case mIoU). Can you give some intuition in why adding calibration loss on source give better mIoU as shown in Table 3 (a)?\n2. This paper proposes two properties and empirically verifies PASTA and RandAugment satisfy the criterion. It would be good to give some intuition in the main text.\n3. [1] uses StyleNet which learns a style transfer from source to target, it would be interesting to analyze whether this inherently satisfy the property 1.\n[1] Donghyun Kim, Kaihong Wang, Kate Saenko, Margrit Betke, and Stan Sclaroff. A unified framework for domain adaptive pose estimation. In ECCV. Springer, 2022.\n4. Do the two AUG choice meet the property 2? Is the \\epsilon as small as before AUG?\n55. Since equation 9 is the upper bound of calibration loss, how is this related to mIoU?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xszzRr0U9d", "reviewer": "Reviewer_WuQd", "summary": "This paper proposes a general Sim2Real adaptation framework, AUGCAL, for semantic segmentation and object recognition.\nAUGCAL introduces synthetic data augmentation and model calibration on synthetic data to reduce miscalibration during synthetic training. Combining with standard unsupervised domain adaptation methods on unlabeled real-world data, those techniques further improve SIM2REAL performance on real-world data.\nEspecially, this paper provides theoretical analysis to show the target calibration loss can be bound by the source calibration loss. This is the reason that calibration on synthetic data is beneficial for the performance of real-world data. Also, it shows the necessity of introducing the augmentaiton.\nExperiments are conducted on GTAV to Cityscapes and VisDA SIM2REAL, and show that the proposed framework can improve the baselines easily.", "questions": "[Augmentation] The choice of augmentation should satisfy the property 1 in Sec. 3.2.2. Even Tab.1 shows that PASTA/R.Aug-SIM are closer to Real than SIM based on MMD, it is not intuitive. It would be better to clarify it more.\n- For PASTA, it is designed to bridge the syn-to-real gap and maybe reduce the artifacts of synthetic data. But I am not sure why R.Aug even closer to Real. \n- Also, if we have large-scale unlabeled real-world data, I am wondering if style transfer-based augmentation will have better performance because it directly uses unlabeled real-world data.\n- Last, as property 1 is a distribution distance, I am not sure how those appearance augmentations affected distribution.\n\n[OC] After Eq.3 and in Sec.4, the paper introduces overconfidence (OC) as a metric. But I am not sure of the exact definition.\n\n[Eq.9] Eq.9 shows a useful upper bound. It would be more interesting to provide more insights regarding this bound. For example, what issues (e.g., a large gap between syn and real) and why will those issues lead to the loose of the upper bound?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "[Clarity] The paper is well structured, and I find myself easy to follow.\n\n[Baseline] Baselines with AUGCAL outperform those without AUGCAL in all metrics.\n\n[Implementation] The presented method looks straightforward yet effective and easily re-implementable.\n\n[Novelty]\n- Augmentation and calibration are complementary and can be integrated easily to achieve the goal.\n- Miscalibration is an overlooked factor for unsupervised domain adaptation, and this paper proposes an easy way to handle it.\n- The paper provides insights on how the calibration loss on synthetic data can reduce miscalibration on real data via a theoretical analysis. This is beneficial for other related research fields.", "weaknesses": "I listed some suggestions in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)