ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation
Authors: Zhitong Gao
Shipeng Yan
Xuming He
Venue: NeurIPS 2023 poster (2023)
Keywords: ood detection
semantic segmentation
anomaly segmentation
test-time adaptation

Abstract: Recent advancements in dense out-of-distribution (OOD) detection have primarily focused on scenarios where the training and testing datasets share a similar domain, with the assumption that no domain shift exists between them. However, in real-world situations, domain shift often exits and significantly affects the accuracy of existing out-of-distribution (OOD) detection models. In this work, we propose a dual-level OOD detection framework to handle domain shift and semantic shift jointly. The first level distinguishes whether domain shift exists in the image by leveraging global low-level features, while the second level identifies pixels with semantic shift by utilizing dense high-level feature maps. In this way, we can selectively adapt the model to unseen domains as well as enhance model's capacity in detecting novel classes. We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, observing consistent performance improvements across various baseline models. Code is available at https://github.com/gaozhitong/ATTA.

ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation
Zhitong Gao', Shipeng Yan', Xuming He!
'School of Information Science and Technology, ShanghaiTech University Shhnghai Engineering Research Center of Intelligent Vision and Imaging agaozht,yanshp yanshp, hexxm)@shanghhatech edu. en
Abstract
Recent advancements in dense out-of iit-o--distributionn (OOD) detection have primarily focused on scenarios where the training and testing datasets share a similar domain, with the assumption that domain shift exists between them. However, in real- world situations, domain shift often exits and significantly affects the accuracy of existing out-of-distribution (OOD) detection models. In this work, propose a dual-level OOD detection framework to handle domain shift and semantic shift jointly. The first level distinguishes whether domain shift exists in the image by leveraging global low-level features, while the second level identifies pixels with semantic shift by utilizing dense high-level feature maps. In this way, we can selectively adapt the model to unseen domains as well enhance model's capacity in detecting novel classes. We validate the efficacy of our method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, observing consistent performance improvements across various baseline models. Code is available at hhttps: //github. com/ htte//gozhiitee tong/ ATTA
Introduction
Semantic segmentation, fundamental computer vision task, has witnessed remarkable progress thanks to the expressive representations learned by deep neural networks [33]. Despite the advances, most deep models are trained under a close-word assumption, and hence do not possess knowledge of what they do not know, leading to over-confident and inaccurate predictions for the unknown objects [18]. To address this, the task of dense diitr-ddittibbution (OOD) detection [5, which aims to generate pixel-wise identification of the unknown objects, has attracted much attention plays vital role in variety of safety-critical applications such autonomous driving.
Recent efforts dense OOD detection have primarily focused on the scenarios where training and testing data share a similar domain, assuming no domain shift (or covariant shift) between them 50
15 38 However, domain shift widely exists in real-world situations 39 and can also be observed in common dense OOD detection benchmarks [29]. In view of this, we investigate the performance of existing dense OOD detection methods under the test setting with Nomain--hiff and observe significant performance degradation in comparison with the setting without domain-shift (cf. Figure []. In particular, the state-of-ther tth-artt detection models typically fail to distinguish the distribution shift in domain and the distribution shift in semantics, and thus tend to predict high uncertainty scores for inlier-class pixels.
A promising strategy to tackle such domain shift is to adapt model during test (known as test-time adaptation (TTA) [46), which utilizes unlabeled test data to finetune the model without requiring prior information of test domain. However, applying the existing test-time domain adaption (TTA) techniques 446] 7 31 47 the task of general dense OOD detection faces two critical challenges. First, traditional TTA methods often assume the scenarios where all test data are under domain shift
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input image
OOD score
Histogram
Evaluate the OOD detection performance under domain shift.
92
94
87
87
82
81
AUPRC
3 Domain shift
57
PEBAL
Ours
+TBN
TTnt
FS Static
FS Static with Domain Shift
[b)
Figure 1: (a) We visualize OOD score maps and the corresponding histograms generated by the SOTA method PEBAL [44] on both original and a domain-shifter Asmog-coruptee image. (b) We quantify the drop in PEBAL's performance with the added domain shift and compare it to the performance when combined with method or existing test-time adaptation methods such as TBN 36 and Tent [46] Please refer to Section 4.3 for additional results.
while our dense OOD detection task addresses a more realistic setting where test data can come from seen or unseen domains without prior knowledge. In such a scenario, TTA techniques like the transductive batch normalization (TBN) [36] 40 2]. which substitutes training batch statistics with those of the test batch, could inadvertently impair OOD detection performance on images from seen domains due to inaccurate normalization parameter estimation (cf. bi) On the other hand, the existence of novel classes in test images further complicates the problem. Unsupervised TTA losses like entropy minimization 46 32 45 often indiscriminately reduce the uncertainty or OOD scores of these novel classes, leading to poor OOD detection accuracy (ef. Figure ())). Consequently, how design effective test-time adaptation strategy for the general dense OOD detection in wild remains an open problem.
In this work, we aim to address the aforementioned limitations and tackle the problem of dense OOD detection wild with both domain- and semantic-level distribution shift. To this end, we propose a novel dual-level test-time adaptation framework that simultaneously detects two types of distribution shift and performs online model adaptation in a selective manner. Our core idea is to leverage low-level feature statistics of input image to detect whether domain-level shift exists while utilizing dense semantic representations to identify pixels with semantic-lever shift. Based on this dual-level distribution-shif estimation, design anomaly-aware self-training procedure to compensate for the potential image-level domain shift and to enhance its novel-class detection capacity based on re- balanced uncertainty minimization of model predictions. Such selective test-time adaptation strategy allows us to adapt an open-set semantic segmentation model to a new environment with complex distribution shifts.
Specifically, we develop a cascaded modular TTA framework for any pretrained segmentation model with OOD detection head. Our framework consists of two main stages, namely selective Batch Normalization (BN) stage and an anomaly-aware self- soff-training stage. Given a test image (or batch), first estimate the probability of domain-shin based on the statistics of the model's 's BN activations and update the normalization parameters accordingly to incorporate new domain information. Subse- quently, second stage performs online self self-training for the entire segmentation model based on an anomaly-awan entropy loss, which jointly minimizes a re- re-balanced uncertainty of inlier-class prediction and outlier detection. As the outlier-class labels unknown, design mixture model in the OOD score space to generate the pseudo-labels of pixels for the entropy loss estimation.
We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, based on FS Statie [T, FS Lost&Found [ Roamany (29] and SMIYC [3]. The results show that our method consistently improves the performance dense OOD detection across various baseline models especially on the severe domain shift settings, and achieves new fate-of--hhe-ate performance the benchmarks.
To summarize, our main contribution is three-folds: We propose the problem of dense OOD detection under domain shift (or covariance shift), revealing the limitations of existing dense OOD detection methods in wild. (ii) We introduce an anomaly-aware test-time adaptation method that jointly tackles domain and semantic shifts. (iii) Our extensive experiments validate our approach, demonstrating significant performance gains on various OOD segmentation benchmarks, especially those with notable domain shifts.

---

2
Related Work
Dense fut-of-distribution Detection The task of Out-of-distributioo (OOD) detection aims to identify samples that are not from the same distribution the training data 18] While most work focuses on image-level have begun to study dense OOD detection [T] [5] 3 (or anomaly segmentation), which is a more challenging task due to its requirement for granular, pixel-level detection and the complex spatial relationships in images. Existing work dealing with dense OOD detection often resorts to either designing specialized OOD detection functions [15 29
-f-tiiibibtttinn detection [18 27
some researchers
48 or incorporating additional training Different from these approaches, our work aims to enhance the model's 's ability to detect OOD objects at test time by utilizing online test data and design model-agnostic method applicable most differentiable OOD functions and training strategies. Furthermore, we target for a more general dense OOD detection problem, where domain-shin potentially exists, which brings new challenges unaddressed in the prior literature.
objectives or data to boost performance in OOD detection 10
Teme-Time Domain Adaptation The task of test-time domain adaptation (TTA) 46 aims study the ability of a machine learning model trained on a source domain to generalize to a different, but related, target domain, using online unlabeled test data only. Existing work on TTA mostly tackles the problem from two aspects: adapting standardization statistics in normalization layers and adapting model parameters as self-training. The first line includes utilizing test-time statistics in each batch 36 or moving averages [35 20], or combining source and target batch statistic
40,
[5 22
28 52 The self- self-training technique including entropy minimization [12 32 45 and
self-supervised losses [3] 31]. In our work, we also approach test-time adaptation from these two perspectives. However, our primary distinction lies in addressing more general open-world scenario, where test data can originate from seen or unseen domains, and novel objects may exist. Consequently, we design our method by explicitly considering these factors and aiming to jointly tackle domain shift and semantic shift challenges.
Novel Class in Unseen Domain While the two primary forms of distribution shift, domain shift and semantic shift, are typically studied independently in literature, there are instances where both are explored in a more complex setting. One such example is Open Set Domain Adaptation [37]. which investigates the unsupervised domain adaptation problem in scenarios where the target data may include new classes. These novel classes must be rejected during the training process. Another line of research focuses on zero-shot learning in the presence of domain shift 449. In this case, novelty rejection must be performed during the testing phase. A domain generalization method is often employed, which requires data from other domains during training. Some work also discuss the impact of covariant-shift in OOD detection [34]. However, our study diverges from these in several key aspects. First, we study the problem of semantic segmentation where the impacts of domain and semantic shifts are more complex and nuanced compared to the general classification problems. Second, we focus on the situation when no additional data or prior knowledge of the test domain can be obtained during the training.
3 Method
3.1 Problem Formulation
We first introduce our general dense OOD detection problem setting with potential domain and semantic distribution shift. Formally, we denote an instance of training data as (x*,y") ~ Pxy e X x )dd where z", y denotes the input image and corresponding segmentation label, X' R3xd represents the input space (an image with d pixels). y = [1, C) is the semantic label space at each pixel, and Pxy is the training data distribution. A test data instance is represented as (x, v) QXY where QXY is the test data distribution. In the OOD detection problem, we have QXY * PXY and our goal is to identify all pixels where their labels do not belong to the training label space, ie.. yi & y. In contrast to previous works, we here consider general problem setting, dense OOD detection with potential domain shift, where in addition the input distributions Px + Qx but with overlaps in their support. Such domain and semantic-class changes leads to a complex data distribution shift, which poses new challenges for the conventional dense OOD detection approaches that are unable to distinguish different shift types.
3

---

Selective Batch Normalization (S8N)
Anomally-waree Self-Training (AST)
Frezen icornable
SIN
Get Domain Prub.
Online Unnabeled Tesl Data
(x)
of!
Xg
8+
KG.
Norm
OUT
Outtier G
P(2 11k)
Entropy
Ly(x)
(x())
16',,
GMM Fining
Output 9
(b) Updata BN Stat.
Image
Intier
Backpropagation
Figure 2:
The overview of the two-stage Anomaly-awe aware Test-Time Adaptation (ATTA) framework. For each test image (or batch), our first stage determines the existence of domain shift and develop a selective Batch-Normalization module to compensate for the input distribution deviation, In the second stage, we devise anomaly-awan self-training procedure via minimizing re-balanced uncertainty of model predictions to enhance the OOD detection capacity.
3.2 Model Overview
In this work, we aim to address the general dense OOD detection problem by leveraging unlabeled test data and adapting segmentation model during test time. Specifically, we begin with a segmentation network that has been trained using data from Pxy. Here we consider a common differentiable OOD network design [5] 30 15 [
that consists of two main components: seen-class classifier f:: X' + yd and an unseen class detector ge : X + Rd. Typically, both network components share the same set of parameters 0. For instance, 9o can be the negative energy score in [13 30 23] or
the maximized logit score as in [15]. During testing, an image (or a batch) sequentially arrives from Qxy, and our goal is to update the model parameters for each batch and produce anomaly- semantic segmentation outputs.
In order to cope with potential domain and semantic shift in test data, we introduce novel dual- level test-time adaptation framework, which allows us to simultaneously identify two types of distribution shifts and performs online self-supervised learning in a selective fashion. Specifically, we instantiate our TTA strategy as a two-stage cascaded learning process. Our first stage determines the existence of domain shift in the image by cxploiting the consistent changes in image-level feature statistics. Based on the inferred shift probability, we develop selective Maich-Normaaizationn module to compensate for the input distribution deviation. In the second stage, devise anomaly- self-training procedure to enhance the model's 's capability in detecting novel classes. The procedure iteratively refines the outlier estimation based mixture model of OOD scores and minimizes re-balanced uncertainty of pixel-wise model predictions for effective adaptation to the test data. An overview of our framework demonstrated in Figure 2
We update the model parameters in an episodic manner, where for each (batch of) image(s) we use same initial network parameter. This makes it more resilience if there is apparent connection between two different (batches of) image(s). In the following, we first explain our design of the selective BN module in Sec. 3.3 followed by our anomaly- self-training stage in Sec. 3.4
3.3 Selective Test- Time Batch Normalization
Our first-stage module aims to identify whether an input image from the seen or unseen domain and perform model adaptation to compensate any potential domain shift. To achieve this, we take inspiration from the transductive Batch-Normalization (TBN) (36), which is a common strategy of exploiting test-time data for model adaptation. The vanilla TBN, however, often suffers from unstable estimation due to the small size of test data, and hence can lead to performance degradation for our general setting where test data may have varying levels of domain shift.
To tackle this, we introduce a selective BN module that first estimates the probability of input image being generated from unseen domain, and then performs mixture of Conventional batch normalization (CBN) and TBN according to the inferred domain-shiff probability. Specifically, we denote P(= = 1|r)) as the probability of an image æ from an unknown domain, and we estimate the probability by considering the distribution distance between the deep features of test and training data in the Normalization layers of the segmentation network. Formally, 1et μÍ of" be the running mean

---

and standard deviation at the 1-th BN layer calculated in the end of the model training, pr(z), oz
be the mean and standard deviation the 1-th BN layer calculated for each test input z, we compute the domain-shift probability as follows:
P(= = 1|x) = ha,s Σ (KLLN( ott))
(I)
where N denotes the normal distribution, KL denotes the Kullback-Leible divergence, and the ha,5(x) sigmoid((2: + a)/b) is sigmoid function with linear transform, which normalizes the distance into probability value. The parameters a, are estimated based the variance of training data statisties such that a data point that is far away from the training statistics has higher probability of image-level domain shift. We then update the BN statistics of the network according to the above probability as follows:
in = P(= = 11>==>) (1(x)) + P(=* = P(=4=0 * MT"
P
0 (z) - (oi
where E [1, L] and L is the max depth of the BN layers. Such an adaptive BN enables us to balance between a stable BN for the data without domain-shift and test-time BN tailored the data with domain shift.
3.4 Anomaly-aware Self-training Procedure
After compensating for the potential domain gap, introduce second stage of test-time model adaptation, aiming for enhancing the model capacity in OOD detection and closed-set prediction on the test data. To this end, we propose an online self-training procedure for the entire segmentation model based on an anomaly-aware prediction entropy loss. By minimizing weighted uncertainty of inlier-class prediction and outlier detection, we are able promote the confidence of model predictions, which leads to a more discriminative pixel-wise classifier on the test data.
Formally, we construct (C' + cllsss pixelwise probabilistic classifier based on the two network modules fo and go and denote its output distribution Y [0, 10, 1](C-))xd with each element Yo, being the probability of Po(ys = c|x). E [1, C' + 1]. For each pixel i, the first C channels represent the probabilities of being each closed-set class while the last channel is the probability of being an outlier class. Given the output distribution, we define our learning objective function as,
Lo(x) -- -
Σ- c=1
weYe, log(Ye.)),
where w. is the weight for class We introduce the class weights to cope with the class imbalance problem between the inlier and outlier classes as the latter typically have much lower proportion in images. More specifically, we set we for 1 < c S C' and we = A > for c = C + 1. In the following, we first describe how we estimate the output probabilities Y based on the seen-class classifier fo and the OOD detector go. followed by the loss optimization procedure.
Anomaly-aware output representation To derive the output probability Y, we introduce an auxiliary variable Z E {0. 1]dd where 2 = 1 indicates the i-th pixel is an outlier, and Po 1|x) denotes the corresponding probability. We also assume that the seen classifier fa outputs a pixelwise probability map F E [0, 1| c[ for the known classes c y and the OOD detector go generates pixelwise outlier score map G E Rdd. The anomaly-aware output distribution V, for the i-th pixel can be written as,
YO,- Po(ú; = c|a, 2 0) 0/.) Po = c|., - 111)
(5)
= Fc,(1 - Pz! = 1 3/ + Po( =
(6)
where [.I is the indicator function, and we use the fact that Pa = c/2 Z: -0) = for c E y and 0 for C + 1, and Po( c| - = 1) 0 for < and 1 for 1
Given the marginal probability in Eqn (6. our problem reduced to estimating the pixelwise outlier probability (z) = 1/z). However, as we only have arbitrary outlier score map G output by ge.
5

---

it is non-trive non-trivial to convert this unnormalized score map into valid probability distribution. A naive nonlinear transform using the sigmoid function or normalizing the scores with sample mean could lead to incorrect estimation and hurt the performance. To tackle this problem, we develop data- driven strategy that exploits the empirical distribution of the pixelwise OOD scores to automatically calibrate the OOD scores into the outlier probability (Z)
More specifically, we observe that the empirical distributions of pixelwise OOD scores {G,) appear to be bimodal and its two peaks usually indicate the modes for inlier and outlier pixels (cf. Figure I We therefore fit two-component Mixture of Gaussian distribution in which the components with lower mean indicates the inliers and the one with higher mean corresponds to the outliers. Given the parameters of the Gaussian components, we now sample-dependent Platt scaling function to estimate the outlier probability as follows,
(7) where we set the calibration parameter a(x) as the value achieving equal probability under two Gaussian distributions, i.e. a(x)
(2 = -1/x) = sigmoid( - a(z))/b<x)),
maxfa NzN((alu1.0)) = Ta 02)} where I TT. Ml M2,, 01,02 22 the parameters of the GMM model, and b(z) as the standard derivation of the sample OOD scores. We note that while it is possible to analytically compute the outlier probability based the estimated GMM, we find the above approximation works well in practice and tends to be less sensitive to the noisy estimation.
Entropy minimization After plugging in the estimated Y, we can re-write our learning objective in Eqn the following form:
Co(x) = Σ
Ya, log(Ye,)) + logg,,oo((fC+..))
(8)
= =-Σ Σ Σ Go) G.)) + Go
where G. - P2 = 1|z) denotes the calibrated outlier probabilities. Direct optimizing this loss function, however, turns out to be challenging due to the potential noisy estimation of the outlier probabilities. To mitigate the impact of pixels with unreliable estimates, we select a subset of pixels with high confidence in outlier estimation and adopt loss function in an iterative manner. Concretely, in each iteration, we first compute the calibration parameters in Eqn 7 and choose subregion D based on thresholding the outlier probabilities. Given a(x), b(x) and D, we then use the binarized outlier probabilities as the pseudo labels for the loss minimization follows:
labeling strategy optimize the
x
Σ RED
Σ
-t) log(Fo, -1 + X t, log G.
(10)
D= (i: G. <T Tr or G. & 4- < + + [G: > r2..
(11)
where TI, T2 are the threshold parameters. In addition, we set = E.14 [[/1;=]]/// 01/4,44 1] in an image-dependent manner to automatically determine the class weight. Following the common practice in test-time adaptation [46. 7 31 47], we only update subset of network parameters to avoid over-fitting, In this work, we choose update the final classification block, which brings the benefit of faster inference after each update iteration.
4 Experiments
We evaluate our method on several OOD segmentation benchmarks, including those with significant
domain shifts and those without, based on FS Static (1, FS Fost&Found (1). RoadAnomaly [29)
and SMIYC (3] Below, we first introduce dataset information in Sec. 4.1 and experiment setup in
Sec.
Then we present our experimental results in Sec.
4.5
4.1 Datasets
Following the literature 44
10
we use the Cityscapes dataset 9 for training and perform OOD
detection tests several different test sets, all of which include novel classes beyond the original

---

Table I: We benchmark OOD detection methods on the corrupted FS Static dataset (gray rows) and compare with the results on the original dataset (white rows). Our ATTA method improves the model's s robustness against corruption when combined with PEBAL.
MSP [P]
Entropy 25
Max logit
[6]
Energy
3D
Mcta-OOD A
PEBAL 54
Ours
TBN 36
Tont
3i
1
92.36
93.14
95.66
95.90
97.56
99.61
99.66
99.25
99.04
70.85
71.23
74.13
74.02
78.34
67.63
99.21
98.96
98.93
A
19.09
26.77
38.64
41.68
72.91
92.08
93.61
86.51
82.38
10.52
14.32
23.60
22.36
$2.31
57.02
87.14
81.97
81.42
FPRss
23.99
23.31
18.26
17.78
13.57
1.52
1.15
2.33
4.09
100.0
100.00
89,94
89.94
100.0
91.17
2.94
4,26
4.43
Cityscapes labels. We note that these datasets may exhibit varying degrees of domain shift due to their source of construction. In the following. we provide a detailed overview of each dataset.
The Road Anomaly dataset [29] comprises real-world road anomalies observed from vehicles. Sourced from the Internet, the dataset consists of 60 images exhibiting unexpected elements such as animals, rocks, cones, and obstacles on the road. Given the wide range of driving circumstances encapsulates, including diverse scales of anomalous objects and adverse road conditions, this dataset presents considerable challenge and potential for domain shift.
The Fishyscapes benchmark [1] encompasses two datasets: Fishyscapes Lost & Found (FS L&F) and Fishyscapes Static (FS Static). FS L&F comprises urban images featuring 37 types of unexpected road obstacles and shares the same setup Cityscapes 38.. thereby rendering the domain shift in this dataset relatively minimal. On the other hand, FS Static is constructed based on the Cityscapes validation [9] with anomalous objects extracted from PASCAL VOC [II] integrated using blending techniques. Consequently, this dataset exhibits no domain shift. For both datasets, we first employ their public validation sets, which consist of 100 images for FS L&F and 30 images for FS Static. Then, we test our method on the online test dataset.
The FS Static -C dataset is employed to investigate the impact of domain shift existing OOD detection methods. We modify the original public Fishyscapes Static dataset [I by introducing random smog, color shifting, and Gaussian blur [6]. mirroring the domain shift conditions [8]
The SMIYC benchmark 3 consists of two datasets, both encompassing variety of domain shifts. The RoadAnomaly21 dataset contains 100 web images and serves as an extension to the original RoadAnomaly dataset [29], representing a broad array of environments. On the other hand, the RoadObstacle21 dataset specifically focuses obstacles in the road and comprises 372 images, incorporating variations in road surfaces, lighting, and weather conditions.
4.2 Experiment Setup
Baselines:
We compare our method with several dense OOD detection algorithms 15
17
24
26
30 4, [0 14 44 and test-time adaptation algorithms (36] 46). To evaluate its generalize ability, method across different OOD detection backbones, including Max Logit [[5]. Energy 33 and PEBAL [4). This allows us to examine its performance with varying capacities of
implement
00I) detection baselines and different OOD function forms.
Performance Measure: Following 10 44 44,, we employ three metrics for evaluation: Arca Under the Receiver Operating Characteristics curve (AUROC), Average Precision (AP), and the Falsc Positive Rate at a True Positive Rate 95% (FPR95).
to use DeepLabv3+ [6] with Wet3ResNet38 trained by Nvidia as the backbone of our segmentation models. In our method, the confidence thresholds 71 and T2 are set to 0.3 and 0.6 respectively. Considering the standard practice in segmentation problem inferences, we anticipate the arrival of one image at a time (batch size = 1). We employ the Adam optimizer with a learning rate of le-4. For efficiency, we only conduct one iteration update for each image. The hyperparameters are selected via the FS Static -C dataset and are held constant across all other datasets. See Appendix B for other details.
Implementation Details For fair comparison, we follow previous work
7

---

Table 2: We compare method the OOD detection benchmarks: Road Anomaly dataset, Fishyscapes Lost Found dataset, and Fishyscapes Static dataset. Our method consistently improve upon several OOD detection methods, with particularly significant improvements observed on the Road Anomaly dataset where domain shift is prominent.
OoD Data
Road Anomaly
FS
LostAndFound
FS Static
Methods
AUC
AP
FPRss
AUC
AP1
FPRg5
AUC
AP t
FPRas
MSP 17
67.53
15.72
71.38
89.29
4.59
40.59
92.36
19.09
23.99
Entropy
68.80
16.97
71.10
90.82
10.36
40.34
93.14
26.77
23.31
Mahalanobis
62.85
14.37
81.09
96.75
11.24
27.37
11.7
Meta-OoD
93.06
41.31
37.69
97.56
72.91
13.57
Synboost
81.91
38.21
64.75
96.21
60.58
31.02
95.87
66.44
25.59
DenseHybrid
99.01
5.09
99.07
76.23
4.17
Max Logit 15
X
72.78
18.98
70.48
93.41
14.59
42,21
95.66
38.64
18.26
+ ATTA (Ours)
76.60
23.96
63.49
93.53
17.39
40.69
95.48
41.23
20.89
Energy 30
73.35
19.54
70.17
93.72
16.05
41.78
95.90
41.68
17.78
ATTA (Ours)
77.41
25.27
62.57
93.30
17.47
43.32
96.0
41.84
17.63
PEBAL
87.63
45 10
44.58
98.96
58.81
4.76
99.61
92.08
1.52
ATTA (Ours)
92.11
59.05
33.59
99.05
65.58
4.48
99.66
93.61
1.15
Road
iihyscappe Found
iishsscapes ttatic
PEBAL Insg
PEBAL hrove
PEBAL
Oum
Ous
OUT
Figure We present qualitative results on the Road Anomaly, FS Lost & Found, and FS Static datasets, where our method improves the previous state-of-the-an model, PEBAL 44, by effectively reducing domain shift and enhancing outso--distribution detection. This improvement is particularly pronounced in the Road Anomaly dataset, which typically presents higher domain shift compared to the Cityscapes training set.
4.3 Results simulated FS Static -C Dataset
To investigate the performance of existing OOD detection method in wild, we benchmark several
existing dense OOD detection methods [17 24 15
30 4 44). Then we integrate our method with the previous saate-of-the-are model the FS Static dataset, and conduct comparison test-time adaptation techniques [36] 46).
As shown in Table I the introduction of corruption notably affects the performance of OOD detection methods, with an average decrease 30% AUC and an average increase of 70% in FPR95. This suggests that current OOD detection methods are highly susceptible to domain shifts. Notably, when combined with our test-time adaptation method, the performance the state-of-the-art OOD detection method, PEBAL [44). remains more stable in the face of domain shifts, demonstrating marginal 0.4% drop in AUC. addition, we note that traditional TTA methods (TBN (36), Tent 46) can result in performance degradation in the original FS Static dataset where no domain shift occurs (cf. rows in white). By contrast, our method consistently enhances the performance of OOD detection, demonstrating it greater adaptability in real-world scenarios where domain prior information is uncertain.
We present additional results on this dataset in Appendix including the results of combining our method with other OOD detection approaches, seen class prediction performance, and experiments under isolated domain shifts.

---

4.4 Results on existing dense OOD detection benchmarks
We then evaluate the performance our method in existing dense OOD detection benchmarks and integrate it with several OOD detection methods. These include saute-of-the-arn techniques that do not require retraining or additional OOD data, such as Max Logit [15] and Energy [0), as well as methods that do, such as PEBAL 44 As shown in Table 2 our ATTA method consistently improve upon previous state-of-the-ard models, with partieularly notable enhancement observed within the Road Anomaly dataset where the domain shift is more pronounced. Specifically, enhance PEBAL's performance from 87% to 92% in AUC, 45% to 59% in AP, and reduce the FPR95 error from 44% to 33%. When compared other methods in the benchmarks, our method in conjunction with PEBAL attains new state-of-the-ar performance across three datasets.
Figure 3 provides qualitative results of our method across these datasets. Our ATTA approach improve the performance of OOD detection backbones by mitigating the impact of domain- and encourage the confidence of the model predictions for both inliers and outliers.
We also submit method, in combination with PEBAL 44 include the SegmenMelf YouCa 3 and the online Fishyscapes ①1 benchmarks, where our method consistently outperforms PEBAL For further details, please refer to Appendix D.3 and D.4
various online benchmarks. These
4.5 Ablation Study
We further evaluate the efficacy of our model components on the Road Anomaly dataset using PEBAL [44] as the baseline model. We begin by analyzing the effectiveness of our proposed modules, Selective Batch Normalization (SBN) and Anomaly-aware Self- celf-nraining (AST), and subsequently delve into more detailed examination of each module's design.
As illustrated in Table 3, the individual application of either SBN or AST contributes incremental enhancements to the baseline model shown in the first row. When these two modules combined, the performance enhancement is further amplified.
Table 3: Ablation study of our two main modules: SBN and AST.
SBN
AST
AUC
AP1
FPRg5
X
X
87.63
45.10
44.58
/
88.72
48.11
43.66
X
90.84
55.81
37.48
V
92.11
59.05
33.59
We then serutinized the internal design of each module in Table We first replace SBN with versions that only employ batch-wist statistics (row #1) or training statisties (row #2), demonstrating that our SBN, which incorporates both types of statistics, outperforms these variants. Subsequently, we evaluated the design of our self-training module, initially replacing it with closed-world entropy [46) calculated only on the seen classes (row #3), and then altered our normalization a naive z-score normalization (row #4). We observed that ablating each component resulted in performance degradation, which underscores the effectiveness of our original design.
sample-specific
Table 4: Ablation study of our internal design of each module.
Train
Batch
Entropy
Norm
AUC f
AP 1
FPRgs 4
/
anomaly-aware
GMM
86.29
48.65
57.03
x
anomaly-aware
GMM
88.72
48.11
43.66
V
V
seen-class only
90.46
54.64
39.28
V
/
anomaly-aware
7-score
91.25
56.65
36.33
/
anomaly-aware
GMM
92.11
59.05
33.59
present the time and memory overhead of the proposed methods, as well as some other ablation results in Appendix

---

Conclusion
In this work, we propose the problem of dense OOD detection under domain shift, revealing the limitations of existing dense OOD detection methods in wild. To address the problem, introduce dual-level OOD detection framework to handle domain shift and semantic shift jointly. Based on selectively adapt the model to unseen domains well enhance model's capacity in detecting novel classes. We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with signifieant domain shifts and those without, demonstrating significant performance gains on various OOD segmentation benchmarks, especially those with notable domain shifts.
framework,
Despite these promising results, there is still room for further refinement in our ATTA method. Similar to other self-training methods, ATTA relies well-initialized model. Our experiments show that our adaptation method yields strong results on sfte-of-thee-r art models, but the improvement is less noticeable with weaker backbones. Another practical consideration is the additional time needed for test-tine cest-time adaptation methods. This is an area where future research could make substantial strides, potentially developing more efficient algorithms that could streamline the process and reduce inference time. All all, our study introduces a more realistic setting of dense OOD detection under domain shift, and we hope it may offer some guidance insights for future work in this field.
Acknowledgments
This work was supported by Shanghai Science and Technology Program 21010502700, Shanghai Frontiers Science Center of Human-centered Artificial Intelligence, and MoE Key Lab of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University).
References
[1] Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Mcasuring blind spots in semantic segmentation. arXiv preprint arXiv:1904.03215 2019.
[2] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tasknorm: Rethinking batch normalization for leta-learning. In Proc. of the International Conference on Machine Learning (ICML). PMLR, 2020.
[3] Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann. Segmentmeifyoucann: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104. 14812, 2021.
[4] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of nuttof-distribution detection in semantic segmentation. arXiv preprint arXiv:2012.00555, 2020.
[5] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5128-5137, 2021.
[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proc. of the European Conference on Computer Vision (ECCV), pages 801-818, 2018
[7] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. Proc. of the European Conference Computer Vision (ECCV), 2022.
[8] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. Proc. of the European Conference Computer Vision (ECCV), 2022.
10

---

[9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of IEEE conference computer vision and pattern recognition (CVPR), pages 3213-3223, 2016.
[10] Giancarlo Di Biase, Hermann Blum, Roland Siegwart, and Cesar Cadena. Pixel-wise anomaly detection in complex driving scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16918-16922 2021.
[11] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision (ICV), 111(1):98-136 2015.
[12] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. 2004.
[13] Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacohsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like onc. arXiv preprint arxiv: 1II.::9112.03663. 2019.
[14] Matej Greić, Petra Bevandić, and Siniša Segvić. Densehybrid: Hybrid anomaly detection for dense open-set recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV. pages 500-517. Springer, 2022.
[15] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling -iit--iiittiittionn detection for real-worke settings. arXiv preprint 1r1iv: 1332, 2020.
[16] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1203. 2019.
[17] Dan Hendryeks and Kevin Gimpel. baseline for detecting misclassified and cut-of-distribution examples in neural networks. arXiv preprint arXiv:
2016.
[18] Dan Hendryeks and Kevin Gimpel. baseline for detecting misclassified and out-of distribution examples in neural networks. In Proc. of the International Conference on Learning Representa- tions (ICLR), 2017.
[19] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In Proc. of the International Conference on Learning Representations (ICLR), 2018.
[20] Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser- Nam Lim, Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint arXiv: 2021.
[21] Sanghun Jung, Jungsoo Lee, Daehoon Gwak, Sungha Choi, and Jaegul Choo. Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scems segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[22] Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprint arXiv:21 2021.
[23] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[24] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In Proc. of the International Conference on Learning Representations (ICI.R) 2018.
[25] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting -itto--distrilution samples and adversarial attacks. In Proc. of the Advances Neural Information Processing Systems (NeurIPS), pages 7167-7177, 2018.
11

---

[26] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-o--distriuution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[27] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In Proc. of the International Conference on Learning Representations (ICLR), 2018.
[28] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. arXiv preprini arXiv:2302.05155, 2023.
[29] Krzysztof Lis, Krishna Nakka, Pascal Fua, and Mathieu Salzmann. Detecting the unexpected via image resynthesis. In Proc. of IEEE international conference on computer vision (ICCV), pages 2151-2161, 2019.
[30] Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li. Energy-basee out-of-distribution detection. arXiv preprint arXiv:2010.03759 2020.
[31] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tu+++: When does self-supervised test-time training fail or thrive? 2021.
[32] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. 2016.
[33] S Minaee, YY Boykov, F Porikli, AJ Plaza, N Kehtarnavaz, and D Terzopoulos. Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[34] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of- distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10051-10059, 2022.
[35] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proc. of IEEE conference on computer vision and pattern recognition (CVPR), 2022.
[36] Zachary Nado, Shreyas Padhy, Sculley, Alexander D' Amour, Balaji Lakshminarayanaa, and Jasper Snoek. Evaluating prediction-tine -ime batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 1000../096.. 10963, 2020.
[37] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE international conference computer vision, pages 754-763, 2017.
[38] Peter Pinggera, Sebastian Ramos, Stefan Gehrig, Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In Proc. of IEEF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1099-1106. 2016.
[39] Joaquin Quinonero- Candella, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008
[40] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. 2020.
[41] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVE international conference on computer vision, 2021.
[42] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 2021.
[43] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test- time training with sepersupe for generalization under distribution shifts. In Proc. of the International Conference on Machine Learning (ICML), 2020.
12

---

[44] Yu Tian, Yuyuan Liu, Guansong Pang, Fengbei Liu, Yuanhong Chen, and Gustavo Cameiro. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. In Computer Vision-FCCV 2022: 17th European Conference, Tel Avix, Israel, October 23 23-27, 2022, Proceedings, Purt XXXIX, pages 443-663. Springer, 2022.
[45] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proc. of IEEE conference on computer vision and pattern recognition (CVPR), 2019.
[46] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by minimization. In Proc. of the International Conference Learning Representations (ICLR), 2020.
[47] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proc. of IEEE conference on computer vision and pattern recognition (CVPR), 2022.
[48] Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, and Alan Yuille. Synthesize then compare: De- tecting failures and anomalies for semantic segmentation. In Proc. of the European Conference on Computer Vision (ECCV), pages 145-161, 2020.
[49] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection, In Proceedings the IEEE/CVF International Conference on Computer Vision, 2021.
[50] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110. 10..1334, 2021.
[51] Fuming You, Jingjing Li, and Zhou Zhao. Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:2 10.04065, 2021.
[52] Yuliang Zou, Zizhao Zhang, Chun-Liang Li, Han Zhang, Tomas Pfister, and Jia-Bin Huang. Learning mustance-specifie adaptation for cross-domain segmentation. Proc, of the European Conference Computer Vision (ECCV), 2022.
13

---

Appendix
We provide the pseudo code in Sec. A more implementation details in Sec. additional experimental results in Sec. [and Sec. [ additional ablation results in Sec. visualization results in Sec. F and some additional discussion in Sec. G
A Pseudo Code of ATTA
The pseudo code for our proposed method, Anomaly-aware Test-Time Adaptation (ATTA), is presented in Algorithm [ parameter for initialization. The adaptation process consists of two stages: ddomiin-aware selective Batch Normalization (BN) updating stage and an online self-training stage based on an anomaly- aware entropy loss. The final output of the model includes the inlier probability map and outlier score map for each test image.
In this algorithm, each test image utilizes same pre-trained network
Algorithm 1: Anomaly-aware Time Adaptation (ATTA)
Input Test samples Duest {zz Y seen-class classifier fe and unseen-class detector with pretrained 6, number of iterations N, learning rate 17, confidence thresholds T1, T2. Output: Inlier probability maps (Fr) outlier score maps (Gt)
for T2 E Diest do
Initialize parameters 0g <- 0.
Compute the domain-shift probability based on (q.(1).
Update the BN statistics of the network using the above probability as in E2,(,,3)
for i L,..., N do
Get the inlier probability map Fi + (xx) and outlier score map G. + go, (1²). Calculate the anomaly aware entropy loss C(Fe, G. TI, 71,72) as in (10,11. Update the model parameters: 0g < Adam(VLL 0g, 1))
end
Output the inlier probability map F: So, (Ix) and outlier score map G. +- go, (1x).
end
B More Implementation Details
Experimental Results of Previous OOD Methods: All experimental results for the previous dense OOD detection methods in our paper are obtained by running the pretrained model weights provided by the official code repository or officially reported in their original papers.
Implementation Details for Tent: For the Tent method [46. we follow the instructions provided in their original paper regarding their experiments the segmentation task. This involves considering the parameters of the affine transformation in the Batch Normalization layers as the trainable param- eters, using transductive batch normalization (36), and adopting the episodic training manner. To ensure fair comparison with our method, we update the model only once for each test image, use Adam optimizer and tune the learning rate the FS Static-C dataset. These steps allow to align our experimental setup with that of the Tent method and facilitate a meaningful comparison.
During the process of fitting the Gaussian Mixture Model (GMM) in our method, we employ a peak-finding algorithm to identify the right-most peak in the distribution of OOD scores. By considering this peak as the initial mean value for the right GMM component before performing the Expectation-Maxinn Maximization (EM) optimization, we aim mitigate potential issues arising from the presence of multiple peaks in the inlier distribution. This approach leverages the prior knowledge that the right-most peak is more likely to represent the outliers, enhancing the robustness and accuracy of our GMM modeling process.
Implementation Details for GMM Modeling in Our Method:
Details in Constructing FS Static-C Dataset: As described in Section 4.1 of our paper, we simulate the domain shift in the FS Static-C dataset by introducing random smog, color shifting, and Gaussian blur. Specifically, we incorporate the fog function from Hendrycks and Dietterich [16] and utilize the 'ColorJitter' and "GaussianBlur" transformations provided by the torchvision library. Each transformation is independently applied with a 50% probability. By randomly combining these transformations, generate the final transformed images for the FS Static-C dataset. This process
14

---

enables us to create diverse and challenging dataset that encompasses various forms and levels of domain shift (ef. Sec. Consequently, we able comprehensively evaluate the performance of our method under different conditions and demonstrate its robustness in handling domain shift.
C Additional Results on Simulated FS Static -C Dataset
C.1 ATTA Combined with Other OOD Detection Methods
We extended our evaluation to include not only PEBAL 44 but also the remaining five OOD detection methods. The results in Table 5 below demonstrate that our method consistently enhances the robustness of these OOD detection techniques in scenarios with potential domain shifts. Specifically, we achieved significant performance gains on the Static-C dataset for all OOD detection methods, with average increase of AUC of around 20% (cf. row #2), AP of 10% (cf. row #4), and decrease of FPR95 nearly 80% (cf. last row). Some results on the FS Static-C dataset are even better than the original performance of the corresponding methods the non-domain-shile FS- FS-Static dataset, such as MSP, Entropy, and Meta-OOO, as shown in the table. By combining our method with various OOD detection methods, demonstrate its efficacy and general applicability across different scenarios.
Table We display additional results of our method combined with various previously established O0D) detection methods on both the FS Static (white rows) and FS Static-C datasets(gray rows).
MSP 15
Entropy
Max logit
Ours
Energy
Meta-OOD
1
92.36
93.91
93.14
95.18
95.66
95.48
95.90
96.00
97.56
98.19
70.85
92.97
71.23
94.33
74.13
94.80
74.02
95.41
78.34
98.06
19.09
26.57
26.77
39.57
41.23
41.84
72.91
83.11
AP1
10.52
20.81
14.32
30.78
23.60
31.13
22.36
32.13
$2.31
75.75
23.99
20.80
23.31
18.98
18.26
20.89
17.78
17.63
13.57
11.63
FP
100.0
22.58
100.00
20.21
89.94
23.59
89.94
18.63
100.0
11.17
C.2
Segmentation Performance Seen Classes
In Sec. 4.3 of the main text, demonstrated the enhancement of OOD detection performance by our method on both the original FS Static dataset and its corrupted version. Here, in Table 6 we further show the improvement of our ATTA method on the segmentation performance of seen classes. We utilize the commonly adopted metrics, mloU (mean Intersection over Union) and mAcc (mean accuracy), and calculate them exclusively the inlier pixels.
Notably, our method significantly improves the mloU metric from 58.83% 85.64% on the corrupted dataset, approaching the performance level of 87.52% achieved on the original FS Static dataset. This demonstrates the effectiveness our method not only in OOD detection but also seen class classification. Besides, we observe that the TBN (36] and Tent (46] methods, although providing satisfactory performance on the corrupted dataset, could harm the segmentation performance on the original dataset. In contrast, the performance gain of our method remains relatively stable across datasets with different levels of domain shift.
Table Segmentation performance evaluation on seen classes the original FS Static dataset and its corrupted version. Comparison is made with previous OOD detection method 44 and two test-time adaptation methods [36]]]]]]]] 46 using mloU(%) and mAcc(%) as metrics.
mIoU
mAcef
FS Static
FS Static +C
FS Static
FS Static +
PEBAL
87.52
58,83
97.56
72.32
ATTA (Ours)
87.44
85.64
97.54
97.09
+ TBN
36
84.88
84.02
97.13
96.70
Tent
46
81.66
84.07
96.63
96.72
15

---

C.3 Performance on Isolated Domain Shifts
We conduct additional experiments by introducing smog. color shifting, and Gaussian blur individually to the original FS Static dataset. Results, as shown in Table 7 reveal consistent improvements by our method across isolated domain shifts.
Table 7: We modify the original FS Static dataset by introducing fog, color shifting, and Gaussian blur separately, analyze model performance isolated domain shifts. We compare method with the previous OOD detection method, PEBAL.
Fog
Color
Blur
AUC 1
AP 1
FPRu5
AUC 1
AP
FPR
AUC 1
AP 1
FPRus +
PEBAL 44
48.37
1.58
91.82
98.58
81.93
6.26
99.46
89.46
2.07
+ ATTA (Ours)
98.92
79.98
3.48
99.15
87.43
2.91
99.55
90.73
1.71
D Additional Results on Dense OOD Detection Benchmarks
D.1 Results of Other TTA Methods the Benchmarks
We compare our method with two other TTA methods, TBN 36 and Tent [46, on three OOD detection benchmarks: the Road Anomaly dataset [29]. the Fishyscapes Lost & Found dataset (Л) and the Fishyscapes Static dataset M. Our method consistently improve upon the previous state- of-the-art OOD detection method, PEBAL [44), while other TTA methods tend degrade the performance of the baseline model, especially in terms of the FPR95 metric. This indicates that traditional TTA methods often indiseriminately reduce the uncertainty or OOD scores of novel classes, resulting in poor OOD detection accuracy. In contrast, our method demonstrates superior performance by effectively handling the domain shift and semantic shift jointly.
Table 8: Comparison of our method with other TTA methods the OOD detection benchmarks:
Road Anomaly dataset, Fishyscapes Lost & Found dataset, and Fishyscapes Static dataset.
Road Anomaly
FS LostAndFound
FS Static
Methods
AUC f
AP
FPRg5
AUC f
AP 1
FPRgs
AUC f
AP 1
FPR
PEBAL. 44
87.63
45.10
44.58
98.96
58.81
4.76
99.61
92.08
1.52
ATTA (Ours)
92.11
59.05
33,59
99.05
65.58
4.48
99.66
93.61
1.15
TBN
85.56
47.81
59.43
95.40
35.01
29.37
99.05
82.45
4.05
Tent
85.35
47.51
60.0
95.39
35.29
29.54
99.02
81,99
4.17
D.2
Results with ResNet 101 backbone
To assess the generalization capability of method, conduct experiments using different segmentation backbone: DeepLabv3+ with ResNet101. Since some previous work, such as [44), does not release their pretrained models this backbone, we compare method with simpler OOD detection basclines, namely Max Logit (15) and Energy Score (30), which do not require additional training. The results in Table 9 demonstrate that our method consistently improves upon these bascline methods across three evaluation metrics.
Table 9: Evaluation of our method with the ResNet101 backbone on the Road Anomaly dataset, Our method consistently improves upon the Max Logit and Energy Score haselines.
AUC +
AP
FPRg5 1
Max Logit 15
76.59
22.97
66.40
+ ATTA (Ours)
84.39
37.84
49.72
Energy 30
77.24
24.05
66.20
+ ATTA (Ours)
84.02
38.95
51.27
16

---

D.3 Results on the SMIYC benchmark
We evaluate our method on the SegmentMelrYouCan (SMIYC) [3 benchmark, which particularly includes significant domain shifts such as variations in illumination and weather. We submit our outputs to the benchmark test set and present the results in Table TO Our experiments on the RoadAnomaly21 and RoadObstacle21 datasets, both part of the SMIYC benchmark, demonstrate significant improvements over the previous SOTA method PEBAL [44]. Notably, in the RoadOb- stacle21 dataset, PEBAL's performance is hampered by lack of robustness, whereas method has increased the AUPRC score from 5.0% to 76.5%. This validates our method's s adaptability to substantial domain shifts.
Table 10: Results on the SMTYC official test benehmark. The results for our model were obtained by submitting the model outputs to the benchmark organizer, as required. The results for PEBAL were taken from the benchmark's official website.
RoadAnomaly21
AP
FPRg3 4
sloUf
PPV1
F1
PEBAL 44
49.1
40.8
38.9
27.2
14.5
+ ATTA (Our))
67.0
31.6
44.6
29.6
20.6
RoadObstacle21
APM
FPRg3 +
sloUf
PPV1
F1
PEBAL 44
12.7
29.9
7.6
5.5
+ ATTA (Ours)
76.5
2.8
43.9
37.7
36.6
D.4 Results on the Fishyscapes Online Testing Set
We evaluate our method on the Fishyscapes Online Testing set [ As presented in Table 11 our method outperforms the previous state-of-the-art, PEBAL 44
Table II: Results on the Fishyscapes online test benchmark. The results for our model were obtained by submitting it to the benchmark organizer. The results for PEBAL were taken from their published paper.
Online FS Lost & Found
APT
FPRgs
PEBAL 44
44.17
7.58
+ ATTA (Ours)
55.94
4.66
Online Static
AP1
FPR
PEBAL 4
92.38
1.73
ATTA (Ours)
94.68
0.68
E Additional Ablation Results
In this section, first evaluate method's inference overhead, then analyze the efficacy of our model components including trainable parameters, optimization techniques, and loss weight.
Analysis of Inference Time We evaluate the average inference time each image. As shown in Table [2] our method is only 2.25 times slower than direct inference, and faster than another test-time adaptation method, Tent [46]. and some ood detection methods with posthoe operations: ODIN (7).
Mahalanobis Distance 225 and Synboost [10]. This efficiency is attributed to our design, which
Table 12:
Comparison of inference time (seconds per image). We calculate the complete time from input to the final prediction and/or ood score. Experiments are conducted on one NVIDIA TITAN Xp device, and results are averaged over all images in the FS Lost & Found validation set, with image size (1024 x 2048).
Methods
Direct Inference
ATTA (Ours)
ATTA (Ours) w/o SBN
Tent
ODIN
SynBoost
Mahalanobis
Time (s)
1.5
5.1
9.2
224.2
17

---

updates only once per image and confines learnable parameters to the classifier block. The latter design enables us to perform backward and the subsequent forward pass only on the classifier block, and is the main reason we achieve much faster inference than Tent. Furthermore, in scenarios where data from the same domain are known, we can reduce the computation by performing domain-shift detection only once, maintaining the variable for subsequent images. To illustrate this, we show the inference time without domain-shin detection, which further reduces it to 1.25 times the direct inference speed. Practically, our episodic training model allows for parallel inference across multiple processors. With ongoing hardware advancements, we anticipate a further reduction in the time gap.
Analysis of Memory Overhead For our method, the memory consumption does increase tem- porarily for storing activation memory during loss computation but aligns with direct inference after backpropagation. Table 13 below illustrates that our memory requirement is higher than direct inference but significantly more efficient than Tent.
Table 13: Comparison of maximum GPU memory consumption during test time (in MB), using DeepLab v3+ with WideResNet38 on an NVIDIA TITAN Xp. input image size (1024 x 2048).
Methods
Max Memory (MB)
Direct Inference ATTA (Ours)
1170.2
3388.6
Tent
12796.7
Analysis of Trainable Parameters In Table 14 we show the results on the Road Anomaly dataset with different trainable parameters. The configurations include: 'All' (all parameters trainable), 'Body' (only the feature extraction part of the network is trainable while the final classification layers are fixed), 'Head" (only the final classification layers trainable), and 'BN affine' (only the parameters of the affine transformation of BN layers are learnable, similar to [46). We observe that all configurations are plausible in our setting, with the performance difference being less than 1% for most metrics. Notably, by training only the parameters the head, we enable faster inference for each iteration since we do not need recalculate the features extracted by the body part.
Table 14: Analysis of the effect of different trainable parameters on the Road Anomaly dataset.
Trainable parameters
AUC
AP1
FPRg5 4
All
92.20
59.60
31.99
Body
92.19
59.58
32.00
BN affine
91.73
58.68
34.30
Head (Ours)
92.11
59.05
33.59
Analysis of Episodic Optimization As stated in the main text, we adapt the model parameters on each image independently an episodic manner. To evaluate alternative test-time optimization strategies, we conduct experiments on the Road Anomaly dataset with two variants. In the first variant, we continue updating the model parameters with the incoming test data (denoted as 'Continue') In the second variant, we processed two images together as a batch (batch size 2) instead of one image at a time. The results are shown in Table IS] We observed that both variants led performance decrease compared to our episodic updating method. This can be attributed to the lower correlation between images in the Road Anomaly dataset. The information learned from previous data potentially negatively affected the model performance on subsequent data, and processing different images together as batch introduced inconsistencies and inaccuracies in domain-shift identification, impacting the final performance. Consequently, episodic updating method tends to be more robust, enabling better adaptation and OOD detection in scenarios with less correlated or diverse data.
Analysis of the outlier class weight We also analyze the effect of the outlier class weight in our loss (represented by in (10)) by removing this term. The results are shown in Table We note that this weight is designed to address the class imbalance problem between the inlier and outlier classes. our analysis, we observe that for the Road Anomaly dataset, where the class imbalance not severe, the performance without the weight is comparable to the performance with the weight.
18

---

Table 15: Analysis of optimization manner on the Road Anomaly dataset. For the "Continue" updating, the data order may affect the output. Hence, we examine with two different data orders.
Training manner
Batch Size
AUC
AP
FP
Continue order
1
87.19
57.3
67.88
Continue order 2
1
89.09
58.14
50.77
Episodic
2
89.57
50.66
39.99
Episodic
1
92.11
59.05
33.59
However, for the Fishyscapes Lost & Found and Fishyscapes Static datasets, where the proportion of outlier objects in images significantly lower, there is a notable performance gap. Therefore, our design of outlier class weight enables more robust performance across different datasets with varying proportions of outlier objects.
Table 16: Analysis of the class weight on three datasets with varying proportions of outlier objects.
Road Anomaly
FS LostAndFound
FS Static
Methods
AUC 1
AP
FPRgs +
AUC
AP
FPRas
AUC t
AP
FPRgs
No Weight
92.05
58.39
33.47
98.78
57.60
6.28
99.57
91.62
1.76
Weight (Ours)
92.11
59.05
33.59
99.05
65.58
4.48
99.66
93.61
1.15
Additional Qualitative Results
Visualization the FS Static Dataset Figure shows additional qualitative results the FS
Static Dataset and its corrupted version. Our method effectively mitigates domain shift effects, as evidenced by clearer separation between inliers (blue) and outliers (orange) in the histograms.
Image
Original
(ATTA)
Original Histogram
(ATTA)
Figure 4: Visualization of the original images in the FS Static dataset and their corrupted versions. We present the OOD score outputs by PEBAL [44] (denoted as OOiginal') and our method, along with their corresponding histograms. The ground truth OOD objects are highlighted with black lines.
19

---

Visualization on the Road Anomaly Dataset As shown in Figure S our method effectively reduces the high uncertainty region associated with the inlier classes, and the separation between the inliers (depicted in blue) and outliers (depicted in orange) is more distinct in the histogram.
Image
Original OOD Score
Ours (ATTA)
Original Histogram
Ours (ATTA)
known class
Figure 5: Visualization results on Road Anomaly. Ground truth OOD objects are marked in black.
Visualization Results on SMIYC validation sets. Figure O presents qualitative results on the SMIYC validation set. Our method clearly reduces the high uncertainty region of inlier classes.
Image
PEBAL
Ours
Image
PEBAL
Ours
(a) Road Anomaly 21 (Validation)
(b) Road Obstacle 21 (Validation)
Sooll
Figure 6: Visualization results on the SMIYC validation set, including (a) the Road Anomaly 21 dataset, and (b) the Road Obstacle 21 dataset. Ground truth OOD objects are marked in green.
20

---

G
Discussion
G.1
Specific Domain Shift in the Road Anomaly Dataset
The specific 'domain shift' in the Road Anomaly dataset, compared to the 'Cityscapes' dataset, encompasses adverse road conditions, diverse weather and lighting conditions, and various camera perspectives and conditions. Here, we detail each of these aspects:
Adverse Road Conditions The Cityscapes dataset mainly focuses on urban scenes with well- paved roads and uniform coloration, whereas the Road Anomaly dataset extends to include more diverse locations like villages and mountains. These rural pathways exhibit varied textures and colors due to different materials, wear, soil types, and vegetation. Examples in Fig. illustrate this contrast, with the visualization of Fishyscapes Lost & Found / Static serving as a reference for typical Cityscapes road conditions. Besides, we see that previous OOD detection methods often mistake these variations for anomalies, while our algorithm reduces such errors, showing better adaptation to the domain shift in road conditions. We refer to Appendix Fig. 5 for more visualization cases on the Road Anomaly dataset, where other examples with various road conditions can be seen.
Weather and Lighting Variations The Road Anomaly dataset encompasses diverse weather conditions not presented in Cityscapes, including snowy, rainy, foggy weather, and nighttime scenes. These variations affect not only the road but also the surrounding areas and the sky, leading to more false positive errors in existing OOD detection models. Examples of these weather-related differences and their effects on OOD detection can be found in Appendix Fig. / Our method strives to account for these variations, enhancing the model's 's adaptability to changes in weather and lighting.
Various Camera Conditions Besides differences in content, the Cityscapes and Road Anomaly datasets also diverge in the conditions under which the images were captured. In the Road Anomaly dataset, images may be captured from various angles and locations, such as alongside the road, which differs from the typical road-centered perspective in Cityscapes. This variance can disrupt previously learned biases in road surface predictions. Additionally, some Road Anomaly images demonstrate a focus effect where the background is intentionally blurred, an effect not commonly seen in Cityscapes. This can lead to false positive errors by initial OOD detection models, as shown in Figure illustrating the sensitivity of models different camera conditions.
In summary, the Road Anomaly dataset's construction, with images gathered from various internet sources, reflects real-world scenario that involves complex domain-level distribution shifts. These shifts present an intriguing problem for existing OOD detection methods. Our research contributes to understanding and addressing these domain shifts within the dataset.
Snowy
Rainy
FogSY
Night
Dark Bac Backeround
Light Background
PEBAL
Figure 7: First Row: Examples of images from the Road Anomaly Dataset, showcasing various weather conditions including snowy, rainy, and smoggy scenes, along with differing lighting condi- tions. Second Row: Corresponding OOD score maps generated by the prior OOD detection method (PEBAL). Third Row: OOD scores after applying our ATTA algorithm for model adaptation. Green arrows indicate regions with prominent domain-shift, highlighting that the previous method in the second row assigned high scores these regions.
21

---

G.2 Architectures without BN
While we primarily target models with BN, it's worth noting that most modern networks utilize certain types of normalization techniques including Instance Normalization (IN) and Layer Normalization (LN). For the case of IN and LN, we note that they inherently offer more stability across domain shifts, thus saving special adjustments in our first cascaded stage. This is because, unlike BN, they normalize individual samples, making them less sensitive to variations between domains. We have also empirically tested our anomaly-aware self-training on the mentioned Segmenter backbone [4] which employs layer normalization. Results shown in Table 17 demonstrate the enhanced performance when incorporating our method.
Table 17: Results on the RoadAnomaly Dataset with Segmenter. We use the pre-trained model weights on the Cityscapes dataset provided by [41]
AUC +
AP
FPRg5
Energy
95.43
75.60
19.76
+ATTA (Ours)
96.49
84.77
18.56
Max logit
94.81
71.65
20.96
+ATTA (Ours)
96.08
81.88
19.19
Still, we note that, since our goal is to adapt at the test phase without retraining, and that BN prevalent in most network architectures, it's practical recognize BN's ubiquity and design accordingly, especially in our task, where many methods 44 are based on the Deep Lab V3+ structure, which utilizes BN.
In summary, our framework is broadly applicable across the vast majority of modern neural network architectures that employ some form of normalization. The test-time adaptation of models without normalization layers remains an open UDA problem, which is out the scope of this work.
22


Reviews:
[{"review_id": "K9HcummKQV", "reviewer": "Reviewer_Nknk", "summary": "The proposed method considers the OOD sample with domain shift and semantic shift.  They address the problem of current OOD approaches that often ignore to handling domain shifts. They introduce an anomaly-aware test-time adaptation method that\njointly tackles domain and semantic shifts. The experiments on different benchmarks, the proposed method demonstrated significant performance gains on various OOD segmentation benchmarks. They show significantly robust performance on benchmarks with notable domain shifts.", "questions": "The computation efficiency of the proposed method is unknow. ", "limitations": "According to the paper, the model's improvement is less noticeable with weaker backbones. The proposed method will add additional time. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper has well-organized writing and clear motivation for each part of the proposed method.\n\nThe proposed method is relatively novel and presented clearly.\n\nThe proposed method achieves SOTA results with large performance gains compared with other SOTA methods. \n\nThe proposed method first time focuses on handling domain shift OOD detection, which has been overlooked by previous OOD methods. \n\n", "weaknesses": "From Figure 2, readers cannot figure out any details regarding each of the proposed components and the overall framework. I would suggest the authors could include more details in the figure to have more detailed basic information for each of their contributions. \n\nAlthough the authors compare their approach to the Road Anomaly dataset for domain shift effectiveness evaluation. The domain shift is even more significant in SegmentIfYouCan benchmarks. Hence, I would suggest the authors to show more results on the SegmentIfYouCan benchmark and see if the proposed ATTA can improve the results. From what I observed, the performance of previous SOTA such as PEBAL, and Meta-OOD, perform relatively worse on the domain-shifted benchmarks like SegmentIfYouCan. \n\nIt would be better if the authors can compare their approach to the online Fishyscape testing set. The validation set only contains a few images, which may not be enough to effectively quantify the performances of the method. \n\nComputational efficiency is important to perform real-time detection for self-driving systems, could authors present the training and inference time of the proposed approach and compare it with other methods? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our contribution and for your thoughtful and detailed feedback.  We have presented our results of SMIYC and analyzed the inference time of our method in the general response. Please find our responses to your other concerns below:\n## W1: Detailed Figure 2 - Model Overview\n\nWe thank the reviewer for the suggestion and have provided a revised Figure 2 in the attached PDF, containing details of our model design. This updated figure provides a more detailed illustration of each of the proposed components and the overall framework.\n\n## W3: Results for online FS Test set\n\nWe thank the reviewer for the suggestion and have submitted our results to the online Fisyscapes testing server during rebuttal. Unfortunately, due to an unexpected shutdown of the testing server, we are unable to provide the results at this moment. We promise to provide the results when they are available and will include them in our revised paper.\n\nTo evaluate our method on a larger benchmark, we have tested our model on the Lost And Found dataset [1], which contains 1203 images. As presented in Table 1, our method outperforms the previous OOD detection methods, Max Logit and PEBAL across all metrics, validating our approach's efficacy. We note that since this dataset contains minimal domain shift, our improvement is relatively small compared to the performance gains observed on other datasets exhibiting more pronounced domain shifts.\n\nTable A1. Comparison of methods on the Lost And Found dataset. Results were obtained by running the publicly available pre-trained models on our device.\n\n|  Lost And Found  |           |           |           |\n|----------------|:---------:|:---------:|:---------:|\n| Methods          | AUROC$\\uparrow$    | AP$\\uparrow$       | FPR95$\\downarrow$    |\n| Max Logit        | 92.73     | 53.22     | 52.51     |\n| Max Logit + ATTA | **93.66** | **57.93** | **47.27** |\n| PEBAL            | 96.88     | 71.21     | 14.63     |\n| PEBAL + ATTA     | **96.95** | **72.39** | **14.55** |\n\n\n[1]  Peter Pinggera et al. Lost and found: detecting small road hazards for self-driving vehicles. IROS, 2016.  \n\n## W4 & Question: Computational Efficiency\n\nWe appreciate the reviewer's concern about computational efficiency. \n- **Training Time:** Our method is designed primarily for test time and can leverage a pretrained model, eliminating additional training. If a pretrained model is not available, we can follow the standard closed-world segmentation training procedure, utilizing the training set. The training time would then depend on the specific segmentation model used, but our method does not add to this time.\n- **Inference Time:** In General Response 2, we demonstrate the inference time of our method and compare it with other methods. We conclude that our method takes only 1.25-2.25 times the pure inference time and is faster than Tent and some OOD detection methods with posthoc operations, such as ODIN, Synboost, and Mahalanobis Distance.\n\nIn case the reviewer specifically means the training and inference time within our self-training procedure, we provide a detailed breakdown in Table A2. We note that the inference time to get the final OOD score is only 0.1s since we only need to do the re-calculation for the classification block.\n\nTable A2:  Detailed time (second per image) for our self-training procedure. The experimental setting is kept the same as in Table G2. The initial forward includes the forward pass of both the feature extractor and classifier.\n| Components | Initial Forward | Loss and Backward | Classifier Forward |\n|:----------:|:---------------:|:-----------------:|:------------------:|\n|  Time (s)  |       1.2       |        0.2        |         0.1        |\n"}, {"review_id": "aD8JYxnLZd", "reviewer": "Reviewer_TGMY", "summary": "The papers deal with two levels of domain shift in semantic segmentation, namely the domain shift on the semantic pixel level and the domain shift on the image level. The paper argues that the current dense out-of-distribution (OOD) detection methods are particularly vulnerable in the presence of image-level domain shifts and therefore likely to predict wrong OOD scores on the pixel level. Based on this observation, it presents a two-stage test-time adaptation approach. In the first step, it is performed selective test-time batch normalization, forcing more adaptation in the scenarios which are identified to be novel, while n the second step, it is proposed a self-training procedure using an anomaly-aware output map to enhance the dense OOD detection. The approach is evaluated on a realistic domain-shift dataset, as well as, two non-domain-shift datasets where it shows good performance.\n\n\nWhile the rebuttal addresses several points of the reviews, the changes and additional experiments discussed are several. The paper should go through another round of peer review with many changes. Therefore, I will not increase my score.", "questions": "- It is not straightforward to implement the paper. There is no discussion about releasing the code.", "limitations": "- Not applicable .", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The analysis of how current OOD detection methods perform on datasets with distribution shifts, as shown in Figure 1 and Table 1, is very valuable and shows the importance of the problem studied throughout the paper.\n\n- The idea is easy to follow. \n\n- The combination of two steps, first to reduce the domain gap at the image level and second to improve the dense OOD detection performance of the models, is not well explored in the literature.\n\n- The results are convincing when compared to the prior work.", "weaknesses": "- The clarity of the method has place for improvement. For instnace, the notation is one point, e.g. 120 line x^s, x^s not defined, Eq. 1 \\mathcal{N} not defined. The method can be understoood in general, it could be further polished to make it easier. There are also minor typos, e.g. line 38, line 245, line 194. Also, KL divergence is not clearly defined in Eq 1.\n\n- In Fig. 1: What is shown in (a) and (b)? A detailed description would be  helpful.\n\n- For comparison with their method, the authors add Tent to PEBAL. An explanation of how this is implemented would be helpful.\n\n- Table 1 compares 6 different OOD detection methods on the synthesised corrupted FS Static (FS Static -C) dataset. It shows a performance drop for all OOD detection methods on the FS Static -C dataset, but applying the proposed approach to PEBAL reduces this drop. It would be interesting to see if a similar effect can be achieved for the remaining 5 OOD detection methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for insightful feedback and recognition of the novel aspects of our work. We are committed to improving the clarity in our final version, and we intend to release the code after the double-blind stage. Below, we respond to the specific concerns raised:\n\n## W1: Clarity\n\n- *Notation:* We thank the reviewer for the suggestion and will ensure to make the notation more clearly explained in the revised manuscript. Specifically,  in line 120, $x^s$ and $y^s$  denote the input image and corresponding segmentation label in the training data, respectively. In Eq. 1, $\\mathcal{N}$ denotes the Normal distribution, and $KL$divergence is defined in its standard form as $KL(P \\| Q)=\\sum_{x \\in \\mathcal{X}} P(x) \\log \\left(\\frac{P(x)}{Q(x)}\\right)$. This equation serves to calculate the KL divergence between two normal distributions with different parameters, as defined in the manuscript. \n\n- *Typos:* We acknowledge the typos mentioned, specifically in lines 38, 245, and 194, and will ensure they are corrected in the revised manuscript.\n\n## W2: Detailed Description of Figure 1 Components (a) and (b)\n\nWe provide a more detailed description and results analysis of Figure 1 in the following, which will also be incorporated in the revised manuscript's caption.\n\n- In Figure 1 (a), we show the visualization results of the OOD score maps produced by the previous SOTA method (PEBAL) for an original image and its domain-shifted version (corrupted with smog).  The first column displays both images, while the second and third columns depict the corresponding OOD score maps produced by PEBAL and their histograms. By comparison, we observe a significant dampening in the OOD prediction score due to the domain shift, as the unknown object outlined in black lines becomes less distinguishable in the OOD score map of the second row, and the separation in the histogram between known and unknown classes diminishes. \n\n- In Figure 1(b), we show the qualitative results of different methods tested on the original FS Static dataset and its domain-shift version. We quantify the drop in PEBAL's performance with the added domain shift and the substantial mitigation of this deterioration when combined with our method. We also examine test-time adaptation methods like TBN and Tent, which gain improvement in high-domain-shift datasets at the cost of the performance in the original dataset. Unlike these, our method enhances performance across both non-domain-shift and domain-shift scenarios, reflecting greater robustness to real-world settings where test data can come from seen or unseen domains.\n\n## W3: Implementation details of Tent on PEBAL\n\nWe use the pretrained model and inference code of PEBAL provided by their official github repository and optimize the parameters of this pretrained model during test time by implementing Tent specifically.\n\nAs detailed in Appendix B of our manuscript, we closely followed the original Tent paper and its official code to implement this combination. This involves using transductive batch normalization, calculating the entropy loss on the inlier classifier, and updating the affine parameters of all Batch Normalization layers. We also adopt the episodic training manner, a batch size of 1, and Adam optimizer, a setting consistent with both the Tent paper's guidance regarding their segmentation experiment and our own method's configuration. To ensure a fair comparison, we update the model only once for each test image and tune the learning rate on the FS Static-C dataset. These steps allow us to align our experimental setup with that of the Tent method and facilitate a meaningful comparison. \n\n## W4: Our Method Combined with Various OOD Detection Methods on FS Static-C\n\nThank you for the suggestion. As requested, we extended our evaluation to include not only PEBAL but also the remaining five OOD detection methods. The results in Table B1 below demonstrate that our method consistently enhances the robustness of these OOD detection techniques in scenarios with potential domain shifts. Specifically, we achieved significant performance gains on the FS Static-C dataset for all OOD detection methods, with an average increase of AUC of around 20% (cf. row #2), AP of 10% (cf. row #4), and a decrease of FPR95 of nearly 80% (cf. last row). Some results on the FS Static-C dataset are even better than the original performance of the corresponding methods on the non-domain-shift FS-Static dataset, such as MSP, Entropy, and Meta-OOD, as shown in the table. By combining our method with various OOD detection methods, we demonstrate its efficacy and general applicability across different scenarios.\n\nTable B1: We display additional results of our method combined with various previously established OOD detection methods on both the FS Static and FS Static-C datasets.\n\n| Metric | Dataset     | MSP   | + Ours    | Entropy | + Ours    | Max logit | + Ours    | Energy | + Ours    | Meta-OOD | + Ours    |\n|--------|-------------|-------|-----------|---------|-----------|-----------|-----------|--------|-----------|----------|-----------|\n| AUC $\\uparrow$    | FS-Static   | 92.36 | **93.91** | 93.14   | **95.18** | 95.66     | 95.48     | 95.90  | **96.00** | 97.56    | **98.19** |\n|        | FS-Static-C | 70.85 | **92.97** | 71.23   | **94.33** | 74.13     | **94.80** | 74.02  | **95.41** | 78.34    | **98.06** |\n| AP $\\uparrow$     | FS-Static   | 19.09 | **26.57** | 26.77   | **39.57** | 38.64     | **41.23** | 41.68  | **41.84** | 72.91    | **83.11** |\n|        | FS-Static-C | 10.52 | **20.81** | 14.32   | **30.78** | 23.60     | **31.13** | 22.36  | **32.13** | 52.31    | **75.75** |\n| FPR95 $\\downarrow$  | FS-Static   | 23.99 | **20.80** | 23.31   | **18.98** | 18.26     | 20.89     | 17.78  | **17.63** | 13.57    | **11.63** |\n|        | FS-Static-C | 100.0 | **22.58** | 100.00  | **20.21** | 89.94     | **23.59** | 89.94  | **18.63** | 100.0    | **11.17** |\n"}, {"review_id": "xcOepAFjAK", "reviewer": "Reviewer_7M1E", "summary": "This paper focuses on the challenging task of open-set semantic segmentation (i.e., dense out-of-distribution (OOD) detection) with domain shift. It proposes a dual-level test-time adaptation framework to overcome domain shift and semantic shift simultaneously, which leverages low-level feature statistics to detect whether domain shift exists while identifying pixels with semantic shift by utilizing dense high-level feature maps. Specifically, it designs an anomaly-aware self-training component to address potential domain shift and improve its ability to detect novel classes through re-balanced uncertainty minimization. The proposed framework is demonstrated to obtain consistent performance improvements across various baseline models.", "questions": "Please refer to the weakness part.", "limitations": "Do not apply.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores the realistic and challenging task of open-set semantic segmentation in a real-world scenario with domain shift, considering the impact of both domain shift and semantic shift (novel class from unknown domain) comprehensively.\n2. The method in this paper seems reasonable and the experimental results prove the significant superiority of the proposed framework on several OOD segmentation benchmarks, regardless of with or without domain shifts. \n3. The method and math presentation in this paper is generally clear.\n", "weaknesses": "1. The visualization of the method in Fig.2 appears overly simplistic and fails to highlight the key components of the proposed framework effectively.\n2. How to choose the parameters of the GMM in L220? Is the performance sensitive to their variations? The hyperparameter experiment of these parameters would be desirable.\n3. Fig.1 intuitively demonstrates a \"domain shift\" (fog), but through the visual experiment in Fig.3, I can not understand what the specific \"domain shift\" that exists in the Road Anomaly dataset for the semantic segmentation is. This casts doubt on the practical application of this paper, despite the experimental evidence confirming the effectiveness of the proposed framework.\n4. All the formulas in this paper have not been correctly punctuated.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and recognition of our work. To address your concerns, we have revised Figure 2 to better highlight the key components of our proposed framework and the overview of our methodology. The updated visualization can be found in the attached PDF. Additionally, we acknowledge the issue with punctuation in the formulas and will ensure that it is corrected in the revised version of the paper. We respond to your other concerns in the following.\n\n## 1. Parameters of GMM:\n\n>How to choose the parameters of the GMM in L220? Is the performance sensitive to their variations? The hyperparameter experiment of these parameters would be desirable.\n\nWe apologize for any confusion. The parameters of our two-component GMM, specifically $\\pi_1, \\pi_2, \\mu_1,\\mu_2,\\sigma_1,\\sigma_2$ in Line 220, are not determined by manual selection, but rather estimated by fitting the OOD scores to the GMM (its two-component structure is fixed), as described in line 214. This fitting process is conducted using the Expectation-Maximization algorithm, facilitated by the scikit-learn package [1]. The means for the GMM components are initialized in a manner that avoids trivial issues with multiple peaks in the inlier distribution (detailed in Appendix B), and the algorithm's convergence parameters are kept at their default settings within the library. \n\n[1] https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html\n\n## 2. Specific \"Domain Shift\" in the Road Anomaly Dataset\n>Fig.1 intuitively demonstrates a \"domain shift\" (fog), but through the visual experiment in Fig.3, I can not understand what the specific \"domain shift\" that exists in the Road Anomaly dataset for the semantic segmentation is. This casts doubt on the practical application of this paper, despite the experimental evidence confirming the effectiveness of the proposed framework.\n\nThe specific 'domain shift' in the Road Anomaly dataset, as compared to the 'Cityscapes' dataset, encompasses adverse road conditions, diverse weather and lighting conditions, and various camera perspectives and conditions.  Here, we detail each of these aspects:\n\n- **Adverse Road Conditions**: The Cityscapes dataset mainly focuses on urban scenes with well-paved roads and uniform coloration, whereas the Road Anomaly dataset extends to include more diverse locations like villages and mountains. These rural pathways exhibit varied textures and colors due to different materials, wear, soil types, and vegetation. Examples in Fig. 3 illustrate this contrast, with the Visualization of Fishyscapes Lost & Found / Static serving as a reference for typical Cityscapes road conditions. Besides, we see that previous OOD detection methods often mistake these variations for anomalies, while our algorithm reduces such errors, showing better adaptation to the domain shift in road conditions. We refer to Appendix Fig. 2 for more visualization cases on the Road Anomaly dataset, where other examples with various road conditions can be seen.\n\n- **Weather and Lighting Variations**: The Road Anomaly dataset encompasses diverse weather conditions not presented in Cityscapes, including snowy, rainy, foggy weather, and nighttime scenes. These variations affect not only the road but also the surrounding areas and the sky, leading to more false positive errors in existing OOD detection models. Examples of these weather-related differences and their effects on OOD detection can be found in the attached PDF. Our method strives to account for these variations, enhancing the model's adaptability to changes in weather and lighting.\n\n- **Various Camera Conditions**: Besides differences in content, the Cityscapes and Road Anomaly datasets also diverge in the conditions under which the images were captured. In the Road Anomaly dataset, images may be captured from various angles and locations, such as alongside the road, which differs from the typical road-centered perspective in Cityscapes. This variance can disrupt previously learned biases in road surface predictions. Additionally, some Road Anomaly images demonstrate a focus effect where the background is intentionally blurred, an effect not commonly seen in Cityscapes. This can lead to false positive errors by initial OOD detection models, as shown in Figure 3, illustrating the sensitivity of models to different camera conditions.\n\nIn summary, the Road Anomaly dataset's construction, with images gathered from various internet sources, reflects a real-world scenario that involves complex domain-level distribution shifts. These shifts present an intriguing problem for existing OOD detection methods. Our research contributes to understanding and addressing these domain shifts within the dataset. Additional experiments in the SegmentMeIfYouCan benchmark, also characterized by domain-shift variations in road surfaces, lighting, and weather conditions, further substantiate the observations. For a detailed presentation of the results, please refer to General Response 1."}, {"review_id": "6mSNNLO6fW", "reviewer": "Reviewer_C29A", "summary": "This paper proposes ATTA (Anomaly-aware Test-Time Adaptation), which introduces test-time domain adaptation (TTA) for anomaly segmentation. As a result, anomaly segmentation can be performed well even in a harsher environment where domain shift and semantic shift occur simultaneously. To create an environment with domain shift, the authors create the FS Static -C (Corrupted FS Static) dataset, and develope a Selective Test-Time Batch Normalization (SBN) method to propose a new TTA. They also introduce self-supervised learning using GMM.", "questions": "1. Is the FS Static -C dataset is just a data augmentation of the original FS Static dataset? Or is there a significant difference enough to be called another 'dataset'?\n2. Is there a performance change when ATTA is trained on the original FS Static dataset rather than the FS Static -C dataset?\n3. Is there a performance change when K-means is used instead of GMM clustering to separate inlier and outlier clusters from the OOD score set G for self-supervised learning?\n", "limitations": "1. The authors have well summarized the common features of the OOD field that are affected by the performance of the backbone, and the weakness that the inference is slow because learning is performed during the test time when TTA is used.\n2. The inference time is expected to be slow, so it is necessary to compare the inference time for each method to check if the overhead is too large.\n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors show existing methods are vulnerable to domain shift by creating the FS Static -C dataset and experimenting with it.\n2. The effectiveness of TTA in the field of anomaly segmentation is demonstrated experimentally (Table 1).\n3. ATTA with SBN and self-supervised learning via GMM, shows better performance than TBN or Tent, which were used in the existing vision field.\n4. The authors provide a mathematically convincing motivation.\n", "weaknesses": "1.\tTable 1 shows that ATTA is exposed to the FS Static -C dataset at test time, and the FPR is re-measured for FS Static -C and the original dataset (FS Static). Therefore, the OOD that must be classified should be seen by the model, so the result should be good. As FS Static -C is a variation of FS Static, it can have a significant impact on the performance of the original dataset. In order to make a meaningful comparison, at least other methods such as Meta-OoD should also be exposed to the FS Static -C dataset and then compared.\n2.\tThe contribution and effect of ATTA is unclear. Using OOD data to improve detection performance has been proposed in the past (e.g., Outlier Exposure[1], Meta-OoD[2], etc.). ATTA also eventually exposes the model to additional OOD data to improve detection performance. However, it is necessary to add a description of the advantages that ATTA has over existing methods by exposing it to the test time. For example, it can be shown in a graph that the segmentation performance increases as the batch progresses during the test time. It is also necessary to show how much the performance is improved through TTA on other datasets other than FS Static -C.\n3.\tIt is unclear whether a fair comparison was made with the OOD data using methods (e.g., Meta-OoD, Synboost, DenseHybrid) in Table 2. Since TTA is able to obtain more information than MSP, Entropy, Mahalanobis and post hocs, ATTA should be superior to these methods that do not use additional OOD data. Therefore, the main competitors of ATTA are methods that utilize additional OOD data in learning (e.g., Meta-OoD, Synboost, DenseHybrid). The authors' method ATTA uses the FS Static -C dataset as an OOD data for training, but the OOD data used for training in the existing methods mentioned above is not specified. Therefore, it is necessary to perform an ablation study to determine whether the superior performance of ATTA is the effect of ATTA, or simply the effect of data augmentation caused by the introduction of the FS Static -C dataset. In addition, ATTA adopted PEBAL as a partner and compared the performance of PEBAL, which was trained on COCO as an additional OOD, with the existing report. PEBAL should also be compared after being exposed to the FS Static -C dataset (excluding ATTA).\n4.\tThe metrics used are AP, AUROC, and FPR95. However, except for AP (Average Precision), these metrics are mainly used in OOD classification rather than anomaly segmentation. It is also necessary to compare the sIoU, PPV, and F1 metrics proposed in the benchmark paper [3] mentioned by the authors in related works.\n5.\tAs the authors mentioned in the Conclusion, TTA is used, so the learning is performed simultaneously at the test time, so the inference time is slow. It is necessary to check if the overhead is too large by comparing the inference time for each method. In particular, GMM clustering must be performed for each image, and it is necessary to check if the overhead due to this is excessive.\n\n[1] Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. \"Deep anomaly detection with outlier exposure.\" arXiv preprint arXiv:1812.04606 (2018).\n\n[2] Chan, Robin, Matthias Rottmann, and Hanno Gottschalk. \"Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation.\" Proceedings of the ieee/cvf international conference on computer vision. 2021.\n\n[3] Chan, Robin, et al. \"Segmentmeifyoucan: A benchmark for anomaly segmentation.\" arXiv preprint arXiv:2104.14812 (2021).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and detailed feedback. Unfortunately, there appears to be some misunderstanding regarding our method, which we will clarify in the following responses.\n\n## W1: Clarification of the FS Static -C dataset\n\nWe would like to clarify that our comparison is indeed fair with other OoD methods, as our approach does not expose the model to any additional information than previous methods. Specifically, our algorithm only encounters online unlabeled image data during inference, a procedure that is also followed by other OOD methods, including Meta-OoD. FS Static-C is merely a testing dataset used to examine the performance of existing OOD methods when a domain shift is introduced. Our algorithm does not utilize any additional information beyond the testing images. Any stated impact of FS Static-C on FS Static appears to be a misunderstanding.\n\n## W2: Our Contribution compared to existing OOD methods\n\n**Our Contribution:** Our ATTA method leverages online unlabeled test images to enhance OOD detection capacity in potential domain-shift scenarios. This stands apart from previous work in two key aspects: \n\n- *We Do Not Rely on Additional Data:* We exclusively utilize online unlabeled test image, which is naturally available during inference, reflecting the characteristics of the test domain. This presents a unique advantage to our method, as methods like Outlier Exposure and Meta-OoD depend on utilizing labeled OOD data during training and are not applicable to our online and unlabeled scenarios. \n\n- *We Explicitly Address Domain-Shift:* This is an area often overlooked by existing OOD detection methods. Our experiments (see Table 1) underline the fragility of prior methods when faced with domain shifts and validate the effectiveness of our approach in these situations.\n\n**Segmentation performance as batch progresses:** We appreciate the suggestion to showcase how segmentation performance might evolve with more data. However, it's worth noting that our design handles different batches of images individually (cf. Sec 3.2), reflecting the reality that different domains may be represented in each batch.\n\n**Performance of TTA on other datasets:**  In addition to FS Static-C, we have evaluated both our method and other test time adaptation techniques (e.g., TBN, Tent) across several datasets, including the Road Anomaly dataset, Fishyscapes Static dataset, and Fishyscapes Lost And Found dataset (results detailed in Table 2 and Appendix Table 2). We have further included extra results on the SegmentMeIfYouCan (SMIYC) benchmark, outlined in the General Response. For all these datasets, we see a clear performance increase by combining our algorithm, which indicates its efficacy.\n\n## W3: Fair Comparison\n\n**Fair Comparison:** As detailed in our previous response to W1, our method doesn't utilize FS Static-C or any additional OOD data for training. Instead, we only employ the online unlabeled test image that is naturally available during inference. In Table 2, we've maintained a fair comparison by evaluating our approach alongside various OOD methods. This includes those that use additional OOD data during training (e.g., PEBAL), and those that do not (e.g., Energy score or Max Logit), providing a balanced and fair assessment.\n\n**General Applicability:** We would like to note that our method is not designed specifically for PEBAL, but is applicable to a wide array of differentiable OOD functions and training strategies (cf. Sec. 3.2). Empirically, we have tested on Max Logit, Energy, PEBAL (as shown in Table 2), as well as Meta-OOD, Entropy, MSP (as detailed in Table B1 in response to reviewer TGMY). These results collectively demonstrate the general applicability of our method.\n\n## W4: Metric\n\nWe acknowledge the reviewer's suggestion. It is worth noting that the metrics we have employed, namely AP, AUROC, and FPR95, are not only standard in anomaly segmentation literature [42, 21, 14, 10] but also aligned with the main metrics of the Fishyscapes benchmarks [1]. Additionally, we have included experiments on the SegmentMeIfYouCan (SMIYC) benchmark, where we assessed the model's performance using the sIoU, PPV, and F1 metrics. Details of these evaluation results can be found in General Response.\n\n## W5 & Limitation: Time Overhead\n\nIn our General Response 2, we demonstrate the inference time of our method, concluding that it is only 1.25-2.25 times the pure inference time. This makes our approach much faster than some other OOD detection methods with post hoc operations. Concerning the GMM clustering overhead, it's worth emphasizing that the impact is minimal in our algorithm. This is primarily due to the fact that the score to be fitted is one-dimensional, and we sample 1% of the total data in clustering. Due to the redundancy in pixel-wise output, this operation does not affect the fitting results. Hence, the overhead due to GMM clustering is not excessive for network inference.\n\n## Q1: Is FS-Static-C just a data augmentation?\n\nThe FS Static-C dataset is utilized to test model performance (as detailed in our previous response), not as 'data augmentation,' which is a technique applied during training. Therefore, the term is not appropriate in this context.\n\n## Q2: Trainning on original FS Static\n\nAs we explained earlier, we do not use any additional data for training, and FS Static -C serves solely as a test set. Therefore, the considered scenario does not apply.\n\n## Q3: Alternative for GMM clustering\n\nK-means clustering is not suitable for our case, as it assigns each point to the nearest centroid, leading to equally sized clusters. Given that outliers are typically much fewer than inliers, this would result in an inaccurate boundary between inliers and outliers, thereby adversely affecting the performance of the self-training model.\n\n\n"}, {"review_id": "rJwoJtcRZy", "reviewer": "Reviewer_XNCt", "summary": "The manuscript considers dense OOD detection under domain shift. The manuscript shows that the contemporary methods for dense OOD detection experience performance drop under domain shift and propose an adaptation framework to mitigate the issue. The proposed framework has two steps. The first step determines whether domain shift exists and attempts to reduce it by adapting the statistics of batch normalization layers. The second step iteratively adapts network parameters by optimizing outlier-aware self-supervised loss. Outlier identification during the test-time training is done by renormalizing arbitrary OOD scores. The resulting method can accommodate various anomaly detectors and achieves competitive results in considered benchmarks for dense OOD detection with and without domain shift.", "questions": "C1. More detailed Fig. 2 may improve clarity.\n\nC2. Since the manuscript introduces a new task it is very important to establish proper evaluation experiments. How does the model perform under different weather and illumination?  E.g. when applying transformations similar to [a] which imitate rain/fog.\n\nC3. The SMIYC benchmark [3] contains images with different illumination (night) and weather (snow, fog). How does the method perform on this benchmark?", "limitations": "The manuscript did not reflect on the limitations of the model. One possible limitation may be described in W3.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1. The road driving scenes indeed experience domain shift. E.g. changes in weather conditions and geolocation affect the captured scenes. Hence, the task of dense OOD detection under domain shift makes sense and appears to be novel.\n\nS2. The developed method can accommodate multiple methods for dense OOD detection, which advocates for general applicability.\n\nS3. The method achieves competitive results in OOD detection in road driving scenarios under considered domain shifts.", "weaknesses": "W1. Contemporary works for dense predictions in traffic scenes consider four main types of domain shift: geolocation, weather conditions [a], day-to-night [b], and synthetic to real [c]. Yet, the manuscript deals only with dense OOD detection under different geolocation (RoadAnomaly). Moreover, FS Static-C contains blurred and colour-jittered images which may not adequately reflect real-world domain shifts in traffic scenes. Authors should experimentally cover all possible domain shifts in traffic scenes. \n\nW2. The proposed framework requires gradient-based optimisation (and hence backpropagation) during the inference. The manuscript should report the time and memory overhead as in [14,21]. Adding considerable computational burden may make the proposed method inapplicable to the considered application.\n\nW3. The proposed framework may only work with models which have batchnorm layers (Sec. 3.3). The manuscript should reflect on models which do not use batch norm layers (eg. attention-based architectures [d]).\n\nW4. The manuscript misses relevant works [e,f,g].\n\n[a] Christos Sakaridis, Dengxin Dai, Luc Van Gool: Semantic Foggy Scene Understanding with Synthetic Data. Int. J. Comput. Vis. (2018)\n\n[b] Christos Sakaridis, Dengxin Dai, Luc Van Gool:\nGuided Curriculum Model Adaptation and Uncertainty-AwareEvaluation for Semantic Nighttime Image Segmentation. ICCV 2019.\n\n[c] Lukas Hoyer, Dengxin Dai, Luc Van Gool: DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation. CVPR 2022.\n\n[d] Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, Cordelia Schmid: Segmenter: Transformer for Semantic Segmentation. ICCV 2021.\n\n[e] Shu Kong, Deva Ramanan: OpenGAN: Open-Set Recognition via Open Data Generation. ICCV 2021\n\n[f] Matej Grcic, Petra Bevandic, Sinisa Segvic: Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. VISAPP 2021.\n\n[g] Chen Liang, Wenguan Wang, Jiaxu Miao, Yi Yang: GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models. NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty and applicability of our work. We have added results on the SMIYC benchmark, inference time analysis, and a refined Figure 2 in our general response and attached PDF. We appreciate the reviewer for pointing out missing relevant works and will include them in the related work section. We address your other concerns below.\n\n## W1: Different types of domain shift\n\nIn this work, we focus on the dense OOD detection problem and consider realistic domain shifts likely to exist in this task. We agree that geolocation, weather conditions, and day-to-night shifts are relevant. However, the synthetic-to-real shift is less applicable in our context. Besides, we emphasize that the adopted datasets, namely RoadAnomaly and FS Static-C datasets, do cover various domain adaptations. Specifically,\n\n- For the RoadAnomaly dataset, images vary significantly from specialized collections in Cityscapes. Differences include geographical location, weather, lighting (day/night), road conditions, camera settings, etc. Please refer to our reply for Reviewer 7M1E - 2 for the detailed analysis and the submitted pdf for image examples.\n\n- FS Static-C is a synthetic dataset comprised of random smog, color shifting, and Gaussian blur. The random smog simulates weather conditions, while the color shifting may represent various lighting scenarios such as dusk or sunset, and the Gaussian blur could signify different camera settings. Although these transformations might not perfectly align with real-world scenarios, they are commonly used in the domain adaptation literature [7,27,45] and reflect the robustness of different models against domain shifts, which is the primal focus of this dataset in our paper. \n\nWe also performed additional experiments on the SMIYC benchmarks, containing variations in geolocation, weather condition, and night information. Our results significantly improved OOD detection, further validating the effectiveness of our method (See General Response for details). \n\nWe note that our work is different from those specifically targeting domain adaptation, and we focus more on the pronounced issues within the OOD detection task, such as ensuring the model works both with and without domain shift, without affecting the prediction of OOD scores.\n\n## W2: Memory Overhead\n\nMemory consumption does increase temporarily for storing activation memory during loss computation but aligns with direct inference after backpropagation. Table E1 below illustrates that our memory requirement is higher than direct inference but significantly more efficient than Tent.\n\nTable E1: Comparison of maximum GPU memory consumption during test time (in MB), using DeepLab v3+ with WideResNet34 on an NVIDIA TITAN Xp, input image size (1024 x 2048). \t\n|                  | Max Memory (MB) |\n|------------------|-----------------|\n| Direct Inference | 1170.2          |\n| ATTA (Ours)      | 3388.6          |\n| Tent             | 12796.7         |\n\n## W3: Architectures without BN\n\nWhile we primarily target models with BN, it's worth noting that most modern networks utilize certain types of normalization techniques including Instance Normalization (IN) and Layer Normalization (LN). For the case of IN and LN, we note that they inherently offer more stability across domain shifts, thus saving special adjustments in our first cascaded stage.  This is because, unlike BN, they normalize individual samples, making them less sensitive to variations between domains [e1].  We have also empirically tested our anomaly-aware self-training on the mentioned Segmenter backbone [d] which employs layer normalization. Results shown in Table E2 demonstrate the enhanced performance when incorporating our method.\n\nStill, we note that, since our goal is to adapt at the test phase without retraining, and that BN is prevalent in most network architectures, it's practical to recognize BN's ubiquity and design accordingly, especially in our task, where many methods [5,21,42] are based on the Deep Lab V3+ structure, which utilizes BN. \n\nIn summary, our framework is broadly applicable across the vast majority of modern neural network architectures that employ some form of normalization.  The test-time adaptation of models without normalization layers remains an open UDA problem, which is out of the scope of this work.  \n\nTable E2. Results on the RoadAnomaly Dataset with Segmenter. We use the pre-trained model weights on the Cityscapes dataset provided by [d]. \n\n|              | AUC $\\uparrow$      | AP $\\uparrow$       | FPR $\\downarrow$     |\n|--------------|-----------|-----------|-----------|\n| Energy       | 95.43     | 75.60     | 19.76     |\n| +ATTA (Ours) | **96.49** | **84.77** | **18.56** |\n| Max logit    | 94.81     | 71.65     | 20.96     |\n| +ATTA (Ours) | **96.08** | **81.88** | **19.19** |\n\n## C2: Different weather and illumination \nWe have considered various domain shifts in our dataset, including different weather and illumination conditions (cf. the response to W1), which establishes a proper evaluation protocol. In particular,  we have actively considered foggy conditions (cf. Appendix B) and conducted extensive testing of our model under various weather and illumination scenarios (cf. General Response 1).  \n\nWe appreciate the reference to Sakaridis et al. [a]. However, it's not directly applicable to our task, as the foggy Cityscapes dataset lacks the novel class information needed for OOD detection evaluation. Additionally, the provided code to generate fog from [a] seems to require specific disparity and camera information, which may not be accessible in our OOD test dataset.\n\n## Limitation\nWe'd like to kindly note that we discuss some limitations of our method in the conclusion. We have provided a response to your concern in W3, demonstrating that our model can be applied to network architectures without BN.\n\n[e1] Shuaicheng Niu et al. Towards Stable Test-time Adaptation in Dynamic Wild World. ICLR, 2023."}]

Author rebuttal:
We thank all the reviewers for their time and constructive comments. In the following, we address some shared concerns in this general response and answer each individual question by replying to each reviewer. We also include an additional PDF containing a revised Figure 2, some examples showing the specific "domain shift" of the Road Anomaly datasets, and visualization results on the SMIYC validation set.

## 1. Results on the SMIYC benchmark

We thank the reviewers for the suggestion of evaluating our method on the SegmentMeIfYouCan (SMIYC) [1] benchmark, which particularly includes significant domain shifts such as variations in illumination and weather. We submit our outputs to the benchmark test set and present the results in Table G1. Our experiments on the RoadAnomaly21 and RoadObstacle21 datasets, both part of the SMIYC benchmark, demonstrate significant improvements over the previous SOTA method PEBAL [2]. Notably, in the RoadObstacle21 dataset, PEBAL's performance is hampered by a lack of robustness, whereas our method has increased the AUPRC score from 5.0% to 76.5%. This validates our method's adaptability to substantial domain shifts. We also present the visualization results on the validation sets in the attached PDF.

Table G1: Results on the SMIYC official test benchmark. The results for our model were obtained by submitting the model outputs to the benchmark organizer, as required. The results for PEBAL were taken from the benchmark's official website.

| RoadAnomaly21              |  |        |          |      |          |
|--------------|:--------------:|:------:|:--------:|:----:|:--------:|
| Methods             | AUPRC $\uparrow$         | FPR95$\downarrow$ | sIoU gt $\uparrow$ | PPV $\uparrow$ | mean F1 $\uparrow$|
| PEBAL        | 49.1           | 40.8   | 38.9     | 27.2 | 14.5     |
| PEBAL + ATTA  | **67.0**       | **31.6** | **44.6** | **29.6** | **20.6** |

| RoadObstacle21              | |        |          |      |          |
|--------------|:--------------:|:------:|:--------:|:----:|:--------:|
| Methods             | AUPRC $\uparrow$         | FPR95$\downarrow$ | sIoU gt $\uparrow$ | PPV $\uparrow$ | mean F1 $\uparrow$|
| PEBAL        | 5.0            | 12.7   | 29.9     | 7.6  | 5.5      |
| PEBAL + ATTA  | **76.5**       | **2.8**  | **43.9** | **37.7** | **36.6** |

[1] Chan Robin, et al. Segmentmeifyoucan: A benchmark for anomaly segmentation. NeurIPS, 2021.  
[2] Yu Tian, et al. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. ECCV, 2022.

## 2. Inference Time
We acknowledge the reviewers' concern about computational efficiency.   
- In this paper, we address a general dense out-of-distribution detection problem, where our method may be applied to various applications beyond self-driving, and thus real-time performance is not the primary focus. 
- In response to the reviewers' request, we have evaluated the average inference time for each image (see Table G2), revealing that our method is only 2.25 times slower than direct inference, and faster than another test-time adaptation method, Tent [1], and some ood detection methods with posthoc operations: ODIN [2],  Mahalanobis Distance [3] and Synboost [4]. 
- This efficiency is attributed to our design, which updates only once per image and confines learnable parameters to the classifier block (cf. lines 238, 278 of the manuscript). The latter design enables us to perform backward and the subsequent forward pass only on the classifier block, and is the main reason we achieve much faster inference than Tent. 
- Furthermore, in scenarios where data from the same domain are known, we can reduce the computation by performing domain-shift detection only once, maintaining the variable for subsequent images. To illustrate this, we show the inference time without domain-shift detection, which further reduces it to 1.25 times the direct inference speed. 
- Practically, our episodic training model allows for parallel inference across multiple processors. With ongoing hardware advancements, we anticipate a further reduction in the time gap.

Table G2: Comparison of inference time (seconds per image). We calculate the complete time from input to the final prediction and/or ood score (See [5] for details). Experiments are conducted on one NVIDIA TITAN Xp device, and results are averaged over all images in the FS Lost & Found validation set, with image size (1024 x 2048).
|  Methods  | Direct Inference | ATTA (Ours) | ATTA (Ours) w/o SBN | Tent | ODIN | SynBoost | Mahalanobis |
|:---------:|:----------------:|:-----------:|:-------------------:|:----:|:----:|:--------:|:-----------:|
|  Time (s) |        1.2       |     2.7     |         1.5         |  5.1 |  9.2 |    3.0   |    224.2    |

[1] Dequan Wang, et al. Tent: Fully test-time adaptation by minimization. ICLR, 2020.    
[2] Shiyu Liang, et al. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. ICLR, 2018.  
[3]  Kimin Lee, et al. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS, 2018.    
[4] Giancarlo Di Biase, et al. Pixel-Wise Anomaly Detection in Complex Driving Scenes. CVPR, 2021.   
[5] https://deci.ai/blog/measure-inference-time-deep-neural-networks/.  


Meta-review: The paper tackled the problem of OOD detection for semantic segmentation in the presence of domain shift. The paper tackles an interesting aspect of ML safety where TTA and OOD are nicely combined. Reviewers have raised many questions and the authors posted additional results, and clarifications. All reviewers concerns have been addressed, however 2 reviewers suggested that the paper should go through another round of peer review given the modifications made. To this AC, the additional results are mainly clarifications of some aspects and evaluation on more dataset, and hence the core contribution is still valid. I suggest that the authors incorporate all the extra analysis and results in main paper and appendix.
Decision: Accept (poster)