SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\bar{G}_i = P_\theta(Z_i^o = 1|x)$  denotes the calibrated outlier probabilities. Direct optimizing this loss function, however, turns out to be challenging due to the potential noisy estimation of the outlier probabilities. To mitigate the impact of pixels with unreliable estimates, we select a subset of pixels with high confidence in outlier estimation and adopt a pseudo-labeling strategy to optimize the loss function in an iterative manner. Concretely, in each iteration, we first compute the calibration parameters in Eqn (7) and choose a subregion  $D$  based on thresholding the outlier probabilities. Given  $a(x), b(x)$  and  $D$ , we then use the binarized outlier probabilities as the pseudo labels for the loss minimization as follows:
$$
\mathcal {L} _ {\theta} (x) \approx - \sum_ {i \in D} \left(\sum_ {c = 1} ^ {C} F _ {c, i} \left(1 - t _ {i}\right) \log \left(F _ {c, i} \left(1 - \bar {G} _ {i}\right)\right) + \lambda \cdot t _ {i} \log \bar {G} _ {i}\right) \tag {10}
$$
$$
D = \{i: \bar {G} _ {i} <   \tau_ {1} \text {o r} \bar {G} _ {i} > \tau_ {2} \}; \quad t _ {i} = 0 \cdot [   [ \bar {G} _ {i} <   \tau_ {1} ]   ] + 1 \cdot [   [ \bar {G} _ {i} > \tau_ {2} ]   ], \tag {11}
$$
where  $\tau_{1},\tau_{2}$  are the threshold parameters. In addition, we set  $\lambda = \sum_{i}[[t_{i} = 0]] / \sum_{i}[[t_{i} = 1]]$  in an image-dependent manner to automatically determine the class weight. Following the common practice in test-time adaptation [46, 7, 31, 47], we only update a subset of network parameters to avoid over-fitting. In this work, we choose to update the final classification block, which brings the benefit of faster inference after each update iteration.
We evaluate our method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, based on FS Static [1], FS Lost&Found [1], RoadAnomaly [29] and SMIYC [3]. Below, we first introduce dataset information in Sec. 4.1 and experiment setup in Sec. 4.2. Then we present our experimental results in Sec. 4.3, 4.4, 4.5.
Following the literature [44, 10], we use the Cityscapes dataset [9] for training and perform OOD detection tests on several different test sets, all of which include novel classes beyond the original
Table 1: We benchmark OOD detection methods on the corrupted FS Static dataset (gray rows) and compare with the results on the original dataset (white rows). Our ATTA method improves the model's robustness against corruption when combined with PEBAL.
MSP [17]	Entropy [24]	Max logit [15]	Energy [30]	Meta-OOD [4]	PEBAL [44]	+ Ours	+ TBN [36]	+ Tent [46]
AUC ↑	92.36	93.14	95.66	95.90	97.56	99.61	99.66	99.25	99.04
70.85	71.23	74.13	74.02	78.34	67.63	99.21	98.96	98.93
AP ↑	19.09	26.77	38.64	41.68	72.91	92.08	93.61	86.51	82.38
10.52	14.32	23.60	22.36	52.31	57.02	87.14	81.97	81.42
FPR95 ↓	23.99	23.31	18.26	17.78	13.57	1.52	1.15	2.33	4.09
100.0	100.00	89.94	89.94	100.0	97.17	2.94	4.26	4.43
Cityscapes labels. We note that these datasets may exhibit varying degrees of domain shift due to their source of construction. In the following, we provide a detailed overview of each dataset.
The Road Anomaly dataset [29] comprises real-world road anomalies observed from vehicles. Sourced from the Internet, the dataset consists of 60 images exhibiting unexpected elements such as animals, rocks, cones, and obstacles on the road. Given the wide range of driving circumstances it encapsulates, including diverse scales of anomalous objects and adverse road conditions, this dataset presents a considerable challenge and potential for domain shift.
The Fishyscapes benchmark [1] encompasses two datasets: Fishyscapes Lost & Found (FS L&F) and Fishyscapes Static (FS Static). FS L&F comprises urban images featuring 37 types of unexpected road obstacles and shares the same setup as Cityscapes [38], thereby rendering the domain shift in this dataset relatively minimal. On the other hand, FS Static is constructed based on the Cityscapes validation set [9] with anomalous objects extracted from PASCAL VOC [11] integrated using blending techniques. Consequently, this dataset exhibits no domain shift. For both datasets, we first employ their public validation sets, which consist of 100 images for FS L&F and 30 images for FS Static. Then, we test our method on the online test dataset.
The FS Static -C dataset is employed to investigate the impact of domain shift on existing OOD detection methods. We modify the original public Fishyscapes Static dataset [1] by introducing random smog, color shifting, and Gaussian blur [16], mirroring the domain shift conditions [8].
The SMIYC benchmark [3] consists of two datasets, both encompassing a variety of domain shifts. The RoadAnomaly21 dataset contains 100 web images and serves as an extension to the original RoadAnomaly dataset [29], representing a broad array of environments. On the other hand, the RoadObstacle21 dataset specifically focuses on obstacles in the road and comprises 372 images, incorporating variations in road surfaces, lighting, and weather conditions.
4.2 Experiment Setup
Baselines: We compare our method with several dense OOD detection algorithms [15, 17, 24, 26, 30, 4, 10, 14, 44] and test-time adaptation algorithms [36, 46]. To evaluate its generalize ability, we implement our method across different OOD detection backbones, including Max Logit [15], Energy [30] and PEBAL [44]. This allows us to examine its performance with varying capacities of OOD detection baselines and different OOD function forms.
Performance Measure: Following [4, 10, 14, 44], we employ three metrics for evaluation: Area Under the Receiver Operating Characteristics curve (AUROC), Average Precision (AP), and the False Positive Rate at a True Positive Rate of  $95\%$  (FPR95).
Implementation Details For a fair comparison, we follow previous work [1, 4, 44] to use DeepLabv3+ [6] with WideResNet38 trained by Nvidia as the backbone of our segmentation models. In our method, the confidence thresholds  $\tau_{1}$  and  $\tau_{2}$  are set to 0.3 and 0.6 respectively. Considering the standard practice in segmentation problem inferences, we anticipate the arrival of one image at a time (batch size = 1). We employ the Adam optimizer with a learning rate of 1e-4. For efficiency, we only conduct one iteration update for each image. The hyperparameters are selected via the FS Static -C dataset and are held constant across all other datasets. See Appendix B for other details.
Table 2: We compare our method on the OOD detection benchmarks: Road Anomaly dataset, Fishyscapes Lost & Found dataset, and Fishyscapes Static dataset. Our method consistently improves upon several OOD detection methods, with particularly significant improvements observed on the Road Anomaly dataset where domain shift is prominent.
Methods	OoD Data	Road Anomaly	FS LostAndFound	FS Static
AUC ↑	AP ↑	FPR95 ↓	AUC ↑	AP ↑	FPR95 ↓	AUC ↑	AP ↑	FPR95 ↓
MSP [17]	X	67.53	15.72	71.38	89.29	4.59	40.59	92.36	19.09	23.99
Entropy [24]	X	68.80	16.97	71.10	90.82	10.36	40.34	93.14	26.77	23.31
Mahalanobis [26]	X	62.85	14.37	81.09	96.75	56.57	11.24	96.76	27.37	11.7
Meta-OoD [4]	✓	-	-	-	93.06	41.31	37.69	97.56	72.91	13.57
Synboost [10]	✓	81.91	38.21	64.75	96.21	60.58	31.02	95.87	66.44	25.59
DenseHybrid [14]	✓	-	-	-	99.01	69.79	5.09	99.07	76.23	4.17
Max Logit [15]	X	72.78	18.98	70.48	93.41	14.59	42.21	95.66	38.64	18.26
+ ATTA (Ours)	-	76.60	23.96	63.49	93.53	17.39	40.69	95.48	41.23	20.89
Energy [30]	X	73.35	19.54	70.17	93.72	16.05	41.78	95.90	41.68	17.78
+ ATTA (Ours)	-	77.41	25.27	62.57	93.30	17.47	43.32	96.0	41.84	17.63
PEBAL [44]	✓	87.63	45.10	44.58	98.96	58.81	4.76	99.61	92.08	1.52
+ ATTA (Ours)	-	92.11	59.05	33.59	99.05	65.58	4.48	99.66	93.61	1.15
Figure 3: We present qualitative results on the Road Anomaly, FS Lost & Found, and FS Static datasets, where our method improves the previous state-of-the-art model, PEBAL [44], by effectively reducing domain shift and enhancing out-of-distribution detection. This improvement is particularly pronounced in the Road Anomaly dataset, which typically presents a higher domain shift compared to the Cityscapes training set.
4.3 Results on simulated FS Static -C Dataset
To investigate the performance of existing OOD detection method in wild, we benchmark several existing dense OOD detection methods [17, 24, 15, 30, 4, 44]. Then we integrate our method with the previous state-of-the-art model on the FS Static dataset, and conduct comparison to test-time adaptation techniques [36, 46].
As shown in Table 1, the introduction of corruption notably affects the performance of OOD detection methods, with an average decrease of  $30\%$  in AUC and an average increase of  $70\%$  in FPR95. This suggests that current OOD detection methods are highly susceptible to domain shifts. Notably, when combined with our test-time adaptation method, the performance of the state-of-the-art OOD detection method, PEBAL [44], remains more stable in the face of domain shifts, demonstrating a marginal  $0.4\%$  drop in AUC. In addition, we note that traditional TTA methods (TBN [36], Tent [46]) can result in performance degradation in the original FS Static dataset where no domain shift occurs (cf. rows in white). By contrast, our method consistently enhances the performance of OOD detection, demonstrating it greater adaptability in real-world scenarios where domain prior information is uncertain.
We present additional results on this dataset in Appendix C, including the results of combining our method with other OOD detection approaches, seen class prediction performance, and experiments under isolated domain shifts.
4.4 Results on existing dense OOD detection benchmarks
We then evaluate the performance of our method in existing dense OOD detection benchmarks and integrate it with several OOD detection methods. These include state-of-the-art techniques that do not require retraining or additional OOD data, such as Max Logit [15] and Energy [30], as well as methods that do, such as PEBAL [44]. As shown in Table 2, our ATTA method consistently improve upon previous state-of-the-art models, with particularly notable enhancement observed within the Road Anomaly dataset where the domain shift is more pronounced. Specifically, we enhance PEBAL's performance from  $87\%$  to  $92\%$  in AUC,  $45\%$  to  $59\%$  in AP, and reduce the FPR95 error from  $44\%$  to  $33\%$ . When compared to other methods in the benchmarks, our method in conjunction with PEBAL attains new state-of-the-art performance across all three datasets.
Figure 3 provides qualitative results of our method across these datasets. Our ATTA approach improves the performance of OOD detection backbones by mitigating the impact of domain-shift, and encourages the confidence of the model predictions for both inliers and outliers.
We also submit our method, in combination with PEBAL [44], to various online benchmarks. These include the SegmenMeIfYouCan [3] and the online Fishyscapes [1] benchmarks, where our method consistently outperforms PEBAL [44]. For further details, please refer to Appendix D.3 and D.4.
We further evaluate the efficacy of our model components on the Road Anomaly dataset using PEBAL [44] as the baseline model. We begin by analyzing the effectiveness of our proposed modules, Selective Batch Normalization (SBN) and Anomaly-aware Self-Training (AST), and subsequently delve into a more detailed examination of each module's design.
As illustrated in Table 3, the individual application of either SBN or AST contributes incremental enhancements to the baseline model shown in the first row. When these two modules are combined, the performance enhancement is further amplified.
Table 3: Ablation study of our two main modules: SBN and AST.
SBN	AST	AUC↑	AP↑	FPR95↓
X	X	87.63	45.10	44.58
X	✓	88.72	48.11	43.66
✓	X	90.84	55.81	37.48
✓	✓	92.11	59.05	33.59
We then scrutinized the internal design of each module in Table 4. We first replace SBN with versions that only employ batch-wise statistics (row #1) or training statistics (row #2), demonstrating that our SBN, which incorporates both types of statistics, outperforms these variants. Subsequently, we evaluated the design of our self-training module, initially replacing it with a closed-world entropy [46] calculated only on the seen classes (row #3), and then altered our GMM-based sample-specific normalization to a naive z-score normalization (row #4). We observed that abating each component resulted in performance degradation, which underscores the effectiveness of our original design.
Table 4: Ablation study of our internal design of each module.
Train	Batch	Entropy	Norm	AUC↑	AP↑	FPR95↓
X	✓	anomaly-aware	GMM	86.29	48.65	57.03
✓	X	anomaly-aware	GMM	88.72	48.11	43.66
✓	✓	seen-class only	-	90.46	54.64	39.28
✓	✓	anomaly-aware	z-score	91.25	56.65	36.33
✓	✓	anomaly-aware	GMM	92.11	59.05	33.59
We present the time and memory overhead of the proposed methods, as well as some other ablation results in Appendix E.
In this work, we propose the problem of dense OOD detection under domain shift, revealing the limitations of existing dense OOD detection methods in wild. To address the problem, we introduce a dual-level OOD detection framework to handle domain shift and semantic shift jointly. Based on our framework, we can selectively adapt the model to unseen domains as well as enhance model's capacity in detecting novel classes. We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, demonstrating significant performance gains on various OOD segmentation benchmarks, especially those with notable domain shifts.
Despite these promising results, there is still room for further refinement in our ATTA method. Similar to other self-training methods, ATTA relies on a well-initialized model. Our experiments show that our adaptation method yields strong results on state-of-the-art models, but the improvement is less noticeable with weaker backbones. Another practical consideration is the additional time needed for test-time adaptation methods. This is an area where future research could make substantial strides, potentially developing more efficient algorithms that could streamline the process and reduce inference time. All in all, our study introduces a more realistic setting of dense OOD detection under domain shift, and we hope it may offer some guidance or insights for future work in this field.
This work was supported by Shanghai Science and Technology Program 21010502700, Shanghai Frontiers Science Center of Human-centered Artificial Intelligence, and MoE Key Lab of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University).
[1] Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation. arXiv preprint arXiv:1904.03215, 2019.
[2] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tasknorm: Rethinking batch normalization for meta-learning. In Proc. of the International Conference on Machine Learning (ICML). PMLR, 2020.
[3] Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann. Segmentmeifyoucan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104.14812, 2021.
[4] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. arXiv preprint arXiv:2012.06575, 2020.
[5] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5128-5137, 2021.
[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proc. of the European Conference on Computer Vision (ECCV), pages 801-818, 2018.
[7] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In Proc. of the European Conference on Computer Vision (ECCV), 2022.
[8] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In Proc. of the European Conference on Computer Vision (ECCV), 2022.
[44] Yu Tian, Yuyuan Liu, Guansong Pang, Fengbei Liu, Yuanhong Chen, and Gustavo Carneiro. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 246-263. Springer, 2022.
[45] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proc. of IEEE conference on computer vision and pattern recognition (CVPR), 2019.
[46] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by minimization. In Proc. of the International Conference on Learning Representations (ICLR), 2020.
[47] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proc. of IEEE conference on computer vision and pattern recognition (CVPR), 2022.
[48] Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, and Alan L. Yuille. Synthesize then compare: Detecting failures and anomalies for semantic segmentation. In Proc. of the European Conference on Computer Vision (ECCV), pages 145-161, 2020.
[49] Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[50] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021.
[51] Fuming You, Jingjing Li, and Zhou Zhao. Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:2110.04065, 2021.
[52] Yuliang Zou, Zizhao Zhang, Chun-Liang Li, Han Zhang, Tomas Pfister, and Jia-Bin Huang. Learning instance-specific adaptation for cross-domain segmentation. Proc. of the European Conference on Computer Vision (ECCV), 2022.
We provide the pseudo code in Sec. A, more implementation details in Sec. B, additional experimental results in Sec. C and Sec. D, additional ablation results in Sec. E, visualization results in Sec. F and some additional discussion in Sec. G.
A Pseudo Code of ATTA
The pseudo code for our proposed method, Anomaly-aware Test-Time Adaptation (ATTA), is presented in Algorithm 1. In this algorithm, each test image  $x_{t}$  utilizes a same pre-trained network parameter  $\theta$  for initialization. The adaptation process consists of two stages: a domain-aware selective Batch Normalization (BN) updating stage and an online self-training stage based on an anomaly-aware entropy loss. The final output of the model includes the inlier probability map and outlier score map for each test image.
Algorithm 1: Anomaly-aware Test-Time Adaptation (ATTA)
Input: Test samples  $D_{\mathrm{test}} = \{x_t\}_{t=1}^T$ , seen-class classifier  $f_\theta$  and unseen-class detector  $g_\theta$  with pretrained  $\theta$ , number of iterations  $N$ , learning rate  $\eta$ , confidence thresholds  $\tau_1, \tau_2$ .  
Output: Inlier probability maps  $\{F_t\}_{t=1}^T$ , outlier score maps  $\{G_t\}_{t=1}^T$ .  
for  $x_t \in D_{\mathrm{test}}$  do  
Initialize parameters  $\theta_t \gets \theta$ .  
Compute the domain-shift probability based on Eq.(1).  
Update the BN statistics of the network using the above probability as in Eq.(2,3).  
for  $i = 1, \dots, N$  do  
Get the inlier probability map  $F_t \gets f_{\theta_t}(x_t)$  and outlier score map  $G_t \gets g_{\theta_t}(x_t)$ . Calculate the anomaly-aware entropy loss  $\mathcal{L}(F_t, G_t, \tau_1, \tau_2)$  as in Eq.(10,11). Update the model parameters:  $\theta_t \gets \text{Adam}(\nabla \mathcal{L}, \theta_t, \eta)$ .  
end  
Output the inlier probability map  $F_t \gets f_{\theta_t}(x_t)$  and outlier score map  $G_t \gets g_{\theta_t}(x_t)$ .  
end
B More Implementation Details
Experimental Results of Previous OOD Methods: All experimental results for the previous dense OOD detection methods in our paper are obtained by running the pretrained model weights provided by the official code repository or as officially reported in their original papers.
Implementation Details for Tent: For the Tent method [46], we follow the instructions provided in their original paper regarding their experiments on the segmentation task. This involves considering the parameters of the affine transformation in the Batch Normalization layers as the trainable parameters, using transductive batch normalization [36], and adopting the episodic training manner. To ensure a fair comparison with our method, we update the model only once for each test image, use Adam optimizer and tune the learning rate on the FS Static-C dataset. These steps allow us to align our experimental setup with that of the Tent method and facilitate a meaningful comparison.
Implementation Details for GMM Modeling in Our Method: During the process of fitting the Gaussian Mixture Model (GMM) in our method, we employ a peak-finding algorithm to identify the right-most peak in the distribution of OOD scores. By considering this peak as the initial mean value for the right GMM component before performing the Expectation-Maximization (EM) optimization, we aim to mitigate potential issues arising from the presence of multiple peaks in the inlier distribution. This approach leverages the prior knowledge that the right-most peak is more likely to represent the outliers, enhancing the robustness and accuracy of our GMM modeling process.
Details in Constructing FS Static-C Dataset: As described in Section 4.1 of our paper, we simulate the domain shift in the FS Static-C dataset by introducing random smog, color shifting, and Gaussian blur. Specifically, we incorporate the fog function from Hendrycks and Dietterich [16] and utilize the 'ColorJitter' and 'GaussianBlur' transformations provided by the torchvision library. Each transformation is independently applied with a  $50\%$  probability. By randomly combining these transformations, we generate the final transformed images for the FS Static-C dataset. This process
enables us to create a diverse and challenging dataset that encompasses various forms and levels of domain shift (cf. Sec. F). Consequently, we are able to comprehensively evaluate the performance of our method under different conditions and demonstrate its robustness in handling domain shift.
C Additional Results on Simulated FS Static -C Dataset
C.1 ATTA Combined with Other OOD Detection Methods
We extended our evaluation to include not only PEBAL [44] but also the remaining five OOD detection methods. The results in Table 5 below demonstrate that our method consistently enhances the robustness of these OOD detection techniques in scenarios with potential domain shifts. Specifically, we achieved significant performance gains on the FS Static-C dataset for all OOD detection methods, with an average increase of AUC of around  $20\%$  (cf. row #2), AP of  $10\%$  (cf. row #4), and a decrease of FPR95 of nearly  $80\%$  (cf. last row). Some results on the FS Static-C dataset are even better than the original performance of the corresponding methods on the non-domain-shift FS-Static dataset, such as MSP, Entropy, and Meta-OOD, as shown in the table. By combining our method with various OOD detection methods, we demonstrate its efficacy and general applicability across different scenarios.