Modulated Phase Diffusor: Content-Oriented Feature Synthesis for Detecting Unknown Objects
Authors: Aming WU
Cheng Deng
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised out-of-distribution object detection; OOD data synthesis; Modulated phase diffusion
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: To promote the safe deployment of object detectors, a task of unsupervised out-of-distribution object detection (OOD-OD) is recently proposed, aiming to detect unknown objects during training without reliance on any auxiliary OOD data. To alleviate the impact of lacking OOD data, for this task, one feasible solution is to exploit the known in-distribution (ID) data to synthesize proper OOD information for supervision, which strengthens detectors' discrimination. From the frequency perspective, since the phase generally reflects the content of the input, in this paper, we explore leveraging the phase of ID features to generate expected OOD features involving different content. And a method of Modulated Phase Diffusion (MPD) is proposed, containing a shared forward and two different reverse processes. Specifically, after calculating the phase of the extracted features, to prevent the rapid loss of content in the phase, the forward process gradually performs Gaussian Average on the phase instead of adding noise. The averaged phase and original amplitude are combined to obtain the features taken as the input of the reverse process. Next, one OOD branch is defined to synthesize virtual OOD features by continually enlarging the content discrepancy between the OOD features and original ones. Meanwhile, another modulated branch is designed to generate augmented features owning a similar phase as the original features by scaling and shifting the OOD branch. Both original and augmented features are used for training, enhancing the discrimination. Experimental results on OOD-OD, incremental object detection, and open-set object detection demonstrate the superiorities of our method. The source code will be released at https://github.com/AmingWu/MPD.

Published as a conference paper at ICLR 2024
MODULATED PHASE DIFFUSOR: CONTENT-ORIENTED FEATURE S SYNTHESIS FOR DETECTING UNKNOWN OBJECTS
Aming Wu, Cheng Deng" School of Electronic Engineering, Xidian University, Xi an. China amwu@xi di iian. wen.edu..com en, chdeng@mail xx xidiian aediinnnuu.... edu.
ABSTRACT
To promote the safe deployment of object detectors, task of unsupervised out- of-distribution object detection (OOD-OD) recently proposed, aiming to detect unknown objects during training without reliance on any auxiliary OOD data. To alleviate the impact of lacking OOD data, for this task, one feasible solution to exploit the known in in-distribution (ID) data to synthesize proper OOD information for supervision, which strengthens detectors' discrimination. From the frequency perspective, since the phase generally reflects the content of the input, in this paper, we explore leveraging the phase of ID features to generate expected OOD features involving different content. And method of Modulated Phase Diffusion (MPD) is proposed, containing a shared forward and two different reverse processes. Specifi- cally, after calculating the phase of the extracted features, prevent the rapid loss of content in the phase, the forward process gradually performs Gaussian Average on the phase instead of adding noise. The averaged phase and original amplitude are combined to obtain the features taken as the input of the reverse process. Next, one OOD branch is defined to synthesize virtual OOD features by continually enlarging the content discrepancy between the OOD features and original ones. Meanwhile, another modulated branch is designed to generate augmented features owning similar phase the original features by scaling and shifting the OOD branch. Both original and augmented features are used for training, enhancing the discrimination. Experimental results OOD-OD, incremental object detection, and open-set object detection demonstrate the superiorities of our method. The source code will be released https:/ggthub..co//mnnngguuu
INTRODUCTION
Detecting unknown objects is critical for the safe application of detection systems. Currently, most detection methods (Ren et al.j 2013) He et al.) 2017 Carion et al.) (2020) usually follow a closed-set assumption, i.e., the training and testing processes share the same category space. However, the real scenario is open and filled with unknown objects, presenting enormous challenges for closed-set assumption based detectors. To facilitate the deployment of object detectors, task of unsupervised out-of -f-ooiiiiiiibtioonnn object detection (OOD-OD) (Du et al. (202cc) is recently proposed, whose purpose detect unknown OOD objects during training without exploiting any auxiliary OOD data.
Due to lacking OOD data for training, the challenge of unsupervised OOD-OD (Du et al. 2022c mainly lies in how only leverage the known in-distribution (ID) data to enhance the ability of distinguishing OOD objects while reducing the impact on the performance of detecting ID objects. One feasible solution (Du et al. 2022c Reiss et al. 2022) is to synthesize series of proper virtual OOD features for supervision based on the ID data, which is conducive to promoting the object detector to learn clear boundary between ID and OOD objects. Particularly, the work (Du et al. 2022c first leverages ID data to estimate class-condee conditoanii distribution for each category. Then, virtual OOD features are sampled from the region that slightly deviates from the estimated distribution. However, to estimate the distributions accurately, it is important utilize abundant objects for each
"Corresponding Author

---

Published as a conference paper at ICLR 2024
Moddllated Pluaee Difftesion
OOD Testing Dala
Phase
Foslure Map
Amplitude
Reverse Inpul
ff(t)
iffG
&
x
ff
U-Net
Aug Phese
ID Fechures
ffe()
Modulator
*:: ODD Features
Aagment Feetumes
Delection Results
Figure I: Modulated Phase Diffusion for OOD-OD, which exploits the Phase information for content- oriented feature synthesis and consists of shared forward and two different reverse processes. To alleviate the impact of lacking unknown data, the first reverse branch (as shown in orange arrows) is to synthesize virtual OOD features that differ from the content of the ID features. Mcanwhile, another branch (as shown in green lines) aims to synthesize the augmented features of the ID features. By taking these synthesized features for training, our method could enhance the ability of discriminating OOD objects while reducing the impact on detecting ID objects (as shown in red and black lines). category, which limits its application in the case of few samples. Meanwhile, when the number of categories is large, estimating the distribution for each category may lead to an increase of computational costs.
In this paper, we still explore feature synthesis to alleviate the impact of lacking OOD data. In general, the extracted features of input images could be recognized as containing style and content information (Lee et al.) 2023). And the content of the OOD features should be different from that of the ID features. Besides, from the frequency perspective, recent researches (Lee et al., 2023 Chen al. 2021 Oppenheim & Lim 1981 have shown that the amplitude and phase could be separately regarded as the style and content of the input. To this end, we pay more attention to exploiting the phase information perform content-oriented feature synthesis, which is instrumental in addressing the key challenge of unsupervised OOD-OD (Du et al. 2022c
Specifically, as shown in Fig. 1 an approach of Modulated Phase Diffusion (MPD) is proposed, which is dedicated phase diffusion generator and mainly consists of a shared forward and two different reverse processes. After extracting the features of the input images, we first utilize Fourier transform to decompose the corresponding amplitude and phase components. For traditional diffusion models (Ho et al., 2020 Luo 2022, Rombach et al., 2022). the forward process to gradually add noise to progressively weaken the content of the input. However, experimental results show that directly adding noise into the phase could not boost the performance of discriminating OOD objects. The reason may be that the phase components generally describe sensitive angle-related information (Oppenheim & Lim] 1981). Adding much noise may rapidly destroy the content in the phase. Therefore, during the forward process, Gaussian Average gradually performed on the phase, alleviating the loss speed of content in phase. The averaged phase and original amplitude are input into the inverse Fourier transform, whose output taken as the input of the reverse process. Next, by continually enlarging the content discrepancy between the OOD features and ID features, one OOD branch (as shown in orange lines) is designed to synthesize expected virtual OOD features. Meanwhile, another modulated branch (as shown in green lines) is presented to generate augmented features owning a similar phase as the IDD features by scaling and shifting the OOD branch. Finally, these synthesized features are used for training, enhancing the discrimination ability. Experimental results on multiple datasets demonstrate the superiorities our method.
In summary, our contributions are mainly three-fold: (1) To alleviate the impact of lacking unknown data, we explore leveraging the phase information to perform content- oriieted feature synthesis, which is conducive to improving the discrimination. (2) To generate expected features, method of Modulated Phase Diffusion is proposed to synthesize virtual OOD features and augmented ID features, which is instrumental in addressing the key challenge of unsupervised OOD-OD. (3) In the
2

---

Published as a conference paper at ICLR 2024
experiments, our method is evaluated on OOD-OD (Du et al. 2022c). incremental object detection
(Kj et al., 2021), and open-set object detection (Han et al. 2022). Particularly, for OpenImages
dataset (Kuznetsova et al., 2020), compared with the bascline method (Du et al. 2022c). our method
significantly reduces FPR95 by around 8.78%.
2 RELATED WORK
OOD Detection. Discovering unknown objects is an important ability of human intelligence. To
simulate this ability, OOD detection (Hendrycks & Gimpel 2017]
Liang et al., 2017 Wu & Deng
2023a b) has recently attracted much attention, whose goal
to discriminate OOD data from ID
data. Currently, most methods (Geifman & EI- Yaniv
2019 Jeong & Kim 2020 Katz-Samuels
al., 2022 Malinin & Gales, 2018; Meinke Hein 2020 focus on OOD image classification and try to design regularization methods to improve the discrimination. Particularly, many methods (Hendryeks et al., 2019; Liu al. 2020, Bendale Boult, 2016; Wang et al., 2021 [un al.) 2022) aim to design some specific score mechanisms to regularize the model to produce different scores for OOD and ID data. And the score could be used to distinguish OOD data from ID data. Besides, reconstruction (Denouden et al. 2018 Zhou 2022) is also commonly used idea for OOD detection, which assumes that the reconstruction loss for OOD data usually larger than that for ID data. Though these methods have been shown to be effective, since object detection involves object localization and classification, these methods could not be directly applied to OOD-OD.
Recently, task of unsupervised OOD-OD (Du al. 2022c Wu et al. 2023 is proposed to localize and recognize unknown objects during training. To reduce the impact of lacking OOD data for supervision, Du et al. (Du al.] 2022c) proposed to use large-scale object samples estimate the distribution of each category, which is used to sample virtual OOD features that slightly deviate from the estimated distribution. However, when the number of categories large, estimating the distribution for each category may increase computational costs. Besides, the work (Du et al. (0226) presented learn unknown-aware knowledge from auxiliary videos. which does not match the setting of unsupervised OOD-OD. Finally, Du et al. (Du et al. (202aa) further presented to use sittance-based mechanism to shape the learned representations. Different from the above methods, in this paper, we explore leveraging the phase components to perform content-oriented feature synthesis, which is beneficial for improving the discrimination ability the object detector.
Diffusion Models. As popular generator, diffusion models (Rombach et al. 2022 Yang et al., 2022b) generally contain forward diffusion for gradually adding noise and reverse process to continually recover the denoised data. Particularly, Ho et al. (Ho et al. 2020) first propose Denoising Diffusion Probabilistic Models, accelerating the deployment of diffusion models. Based on this work, some methods (Rombach al., the attention mechanism (Vaswani et al. 2017 and Variational AutoEncoder (VAE) (Van Den Oord et al., 2017) into existing diffusion models, which produce stable diffusion models and generate high-quality images. However, diffusion models are rarely used for specific feature synthesis. In this paper, we propose method of Modulated Phase Diffusion, which is dedicated phase-based diffusion generator including a shared forward and two different reverse processes. By this diffusion model, we can obtain expected OOD features and augmented ID features, which strengthens the discrimination. Experimental results on multiple datasets demonstrate the superiorities of our method.
2022 Gu et al. 2022 Hu et al.. 2022) explore introducing
MODULATED PHASE DIFFUSION FOR FEATURE SYNTHESIS
For unsupervised OOD-OD (Du et al. OOD data. To this end, we design Modulated Phase Diffusion for synthesizing specific features. Concretely, we follow the settings (Du et al., 2022c) and only utilize the ID data that own a fixed number of categories for training. During inference, the object detector should distinguish ID objects from OOD objects accurately.
2022c). the key is how to overcome the difficulty of lacking
3.1 FORWARD DIFFUSION VIA AVERAGING PHASE
As shown in Fig. 2. we follow the baseline work (Du et al. object detector, i.e., Faster R-CNN (Ren et al. 2015 He al. 2017), the basic detection model.
2022c) and exploit the widely used
3

---

Published as a conference paper at ICLR 2024
O Region Proposals
C Concatenation
Fo
Fr
Lloc
Rol Feat Extractor
fft
ifft
RPN
Ampitude
Pr
Lels
Po
Po
Shared
Backboon
Sre Phase
Po
(F O)
ifft
Ag Plase
Kl.ddiverencce
OOD FoaL Phase
Rol Feat.
Avg Phase
O
a
Extractor
Density
Po
Lymoorty
B-
B
ifft
(Fo. O)
Modulator
Nogative Energy Score
Pr
Pr.
Inpul Image
Figure 2: The details of MPD for detecting unknown objects. MPD mainly contains three functions:
(1) The output of the forward process is used enhance object-related information in the features extracted by the backbone network, which is conducive to improving localization performance. (2)
To alleviate the impact of lacking OOD data, one OOD branch (as shown in orange lines) presented
to synthesize expected OOD features Fo that differ from the content of the original features Fo. (3)
Another modulated branch (as shown in green lines) is designed to generate augmented features Fo of the original features Fo, strengthening the discrimination of the object classifier.
Given an input image, a backbone network, e.g., ResNet (He et al. 2016), is utilized to extract the corresponding feature map Fo E wxxxxx, where w, h, and separately denote width, height, and the number of channels.
Then, Fo is input into fast Fourier transform (FFT) (Cochan et al. 1967 Lee al. 2023) to decompose the corresponding amplitude A E Ruxxxe and phase Po E Ik w xxxxxxe content-oriented feature synthesis, the diffusion operation is performed on the phase Po containing the content information of the input. In general, the forward diffusion (Yang et al., 2022b Ho et al. 2020) gradually adds noise progressively cover up the content of the input image. Unfortunately, experimental results show that adding noise into the phase of input features could not boost the performance. Taking PASCAL VOC (Everingham et al., 2010) and MS-COCO (Lin al., 2014) as the ID data for training and OOD data for evaluation, based on the FPR95 metric, the performance of using the traditional diffusion model (Ho et al. 2020) and adding noise into the phase is increased by 6.48%. The reason may be that the phase of the features describes sensitive angle-related information (Oppenheim & Lim 1981). Adding much noise may rapidly destroy the content in the phase, affecting the performance of the diffusion model.
Next, to attain
To this end, we explore replacing adding noise with Gaussian Average, which is beneficial for slowing down the loss speed of content in phase. Specifically, the forward process is fixed to Markov chain that gradually performs Gaussian Average on the phase Po:
2T (1+52)/202
P g(a),
((a)
where (ar) represents Gaussian Kernel with the variance a. (i.j) indicates the position in the kernel. t= 1, I..., P. wxxxc is the convolutional output. Meanwhile, the kernel size is set to Finally, it is worth noting that the forward process has no learnable parameters.
Since Gaussian Average is a local operation, compared with globally adding noise on the phase, this operation reduces the damage to the content involved in Po. Finally, after multiple iterations, the averaged phase Pr. E wxxxx and original amplitude are input to inverse fast Fourier transform (IFFT) (Cochran et al. the input of the following reverse stage to synthesize specific features.
1967 Lee et al. 2023) to acquire the output Fr E puxxxxe that is taken as
3.2 REVERSE PROCESS FOR OOD FEATURE SYNTHESIS
To reduce the impact of lacking OOD data, as shown in Fig. 2 one OOD reverse branch is presented to Fy synthesize expected OOD features that differ from the content of Fg. which is instrumental in improving the ability of discriminating OOD objects from ID objects. Concretely, Fig. and 2 taking Fr the input, U-Net model (Ronneberger al. 2015) is designed predict the
4

---

Published as a conference paper at ICLR 2024
feature Di-1 Ruxxxe Then, we perform FFT on Di-1 to decompose the corresponding phase Pt-1 E Ruxxxx, Next, P:-1 and A are input to IFFT, whose output is taken the input of the next step. The processes are as follows:
D =
= (Frs 1), P-I=I1((---),
i--= A),
(2)
where co(:.-) represents the learned U-Net model. And TT,,... ..., Fr- = Fr. Since the amplitude generally describes the style information, to alleviate the style impact and focus on content-oriented feature synthesis, we replace the amplituee from D.-1 with the original amplituee A. FT-1 E
_uxaxxe is the output the timestep t. Next, taking Ft-1 the input, we continually perform the above operations obtain the synthesized virtual OOD map Po Rxxxxe
During training, a loss function Lood is proposed to facilitate the designed U-Net model to possess the capability of OOD feature synthesis:
Lood E. f fSt(es O)) 6, = Pr Pr-
where e, describes the lost phase information in Po from the timestep T I. to T - Since the average operation is continually performed on Po. as the iteration increases, the information about Po involved in e, gradually decreases, which is conducive to promoting F6 to contain rich information that differs from the content in Fo. Finally, to further enlarge the gap between Fu and Fo. loss Lais defined to maximize the K L-divergence between the virtual OOD features and features, i.e., Edis = (L)g(Fo), {(Fo)]. where q()) represents the probability distribution.
3.3
MODULATED REVERSE PROCESS FOR AUGMENTED FEATURE SYNTHESIS
The modulated reverse process aims to recover the original features from the forward output Fr. which is taken as the augmentation for training and is instrumental in strengthening the discrimination of the classifier for ID objects. Specifically, shown in Fig. and taking Fr as the input, we still
employ the U-Net model of the OOD branch predict the corresponding feature D,-1 E IR wxxxe Since Dr- involves plentiful OOD-related information, we exploit the modulation mechanism (Perez
al. 2018
Wang et al. 2020) to perform a transformation on D.
D. =eo(Fe. (), Dz-1 3 D. + B,
(4)
where and |3 are learnable parameters for channel-wisdd scaling and shifting D:-1. 1. D.-1 E Rwxhxe and / = T, T,.... 1. Fr. Fr. Compared with employing a newly-designes U-Net model, leveraging the modulation mechanism (Perez al. 2018 Wang et al. 2020) does not introduce a large number of parameters. alleviating the overlitting risk. Then, FFT is performed on D,-1 to decompose the phase P E TR. Txxe which corresponds to the lost phase caused by the forward average. Next, we utilize the IFFT operation to calculate the input of the next iteration:
P = ff((D-)))
P.+
f-1=ifftPP:-. A),
(5)
where py= Pr. Pr-, wxxxe represents the recovered phase. Similarly, to reduce the impact of amplitude, P._ and original amplitude A are input to IFFT to acquire the output Fr_ e xxxxe Next, we continually perform the above operations in equation 4 and equation 5 to generate the augmented feature map Fo E Rwxkxe of the original feature map Fo-
During training, loss Lang is defined promote the modulated parameters to contain the ability of transforming OOD information:
Loug E, [lee -
f(( -
6 P - Pt.
(6)
where e, describes the lost phase information from the timestep - to By optimizing equation 6 the decomposed phase P. could be facilitated to contain rich information about the lost phase, which makes Fg retain plentiful content about Fo.
3.4 MPD-DRIVEN OOD OBJECT DETECTION
In general, object detection involves two subtasks, i.e., object localization and recognition. Therefore, enhancing object-related information in Fo is instrumental in detecting objects accurately.
5

---

Published as a conference paper at ICLR 2024
Algorithm I Modulated Phase Diffusion for Unsupervised OOD-OD
Input: ID data {X, YJ, randomly initialized detector with parameter s, randomly initialized U-Net with parameter 0, randomly initialized modulator with parameter y and 3, weight À for the K L-loss weight a for the loss LMPD. weight r for the uncertainty loss Luncerty.
Output: Object detector with parameter yp" and OOD detector C. while train do
Sample images from the ID dataset {x, [X,Y]. Perform the forward phase diffusion using equation to obtain Py and Fr. for T, 1 do
Di-1 =o((Ft, t), ff(D and ifft((P- A).
OOD map Fa
D co(Ft 1), and 1 =- ④ D.-1 + 8.
Pt- = ff((D--)), P = Pt Pt-1, and F = iff(( A).
# Aug map Fo
end
Calculate the overall training objective L using equation 3 and equation
equation 6
equation 7
equation g
Update the parameters sp, 0, and [3 based on equation 9
end
while eval do
Calculate the OOD uncertainty score using the left part of equation 10 Perform thresholding comparison using the right part of equation 10
end
To this end, as shown in Fig. 2 residual operation between F6 and Fr first performed. Since the content in the forward output Fr is blurry after multiple average operations, the residual result contains plentiful object-related related information. Then, the residual result is concatenated with Fo to obtain the enhanced result E E Rexxxe, i..,, E ([F,,, Fo Frl), where (-()) R xxxx2exx represents one-layer convolution to transform the number of channels.
Next, E is taken the input of the RPN module (Ren al. 2015 He et al. 2017 output of object proposals O. Meanwhile, based on O, Rol-Alignment followed by Rol-Feature extraction (e et al. 2017) is separately performed on E and augmented map Fo to obtain Oin E Rm xn and Oaug E IR XXx where m and denote the number of proposals and channels. Then, Oin input to the object classifier and regressor to calculate the classification loss Lele and localization loss Lioc:
Cin Lels + Lioe A KK[p((O..) p(Oaug)].
where À is byper-paramesd which is set to 0.001 in the experiments. The K L--iveereeece loss is to constrain the prediction consistency between On and Oaug. ameliorating the ability of the object classifier for discriminating ID objects.
Finally, to distinguish OOD objects from ID objects, based on O, Rol-Alignment followed by Rol- Feature extraction is performed on Fg to extract OOD features Oood E Rmxx, Qood and Oi are used compute uncertainty loss (Du et al. 2022c). regularizing the detector to produce low OOD score for the ID object features, and a high OOD score for the virtual OOD features:
exp E(u)
1
Luncerty Eu |-log + exp E(u)
Ep...na I-og
(8)
1+ exp E(v)
where E(-) the object-level energy score Du et al. 2022c Liu et al. 2020 The overall objective shown as follows:
L Lin + a LMPD Luncerty,
LMPD Lood + Laug Lais:
where a and are two hyper syper-parameters, which are 0.001 and 1.1 in the experiments.
3.5 INFERENCE FOR OOD OBJECT DETECTION
During inference, we only leverage the forward phase diffusion to acquire the feature Fy used to enhance object-related information. Meanwhile, only calculate the uncertainty score for OOD

---

Published as a conference paper at ICLR 2024
Table 1: The performance (%) unsupervised OOD-OD, 1 denotes larger values are better and represents smaller values are better.
In-distribution Data
Method
FPR95 1
AUROCT OOD: MS-COCO7
mAP (ID))
OpenImages
MSP (Hendrycks Gimpel
2017
70.99733.
83.45 81.91
48.7
ODIN [Liang al. 2017
63.12
82.20/ 82.59
48.7
Mahalanobıs Lee al 2018b
65,41
81.45 / 81,48
48.7
Gram matrices Sasiry & Oore
2020
67.75/67.42
79.88 / 77.62
48.7
Energy score Liu ct al. 2020
55.89
83.69 82.98
48.7
PASCAL-VOC
Generalized ODIN [T el al
2020
79.57
83.12 79.23
48.1
CST Tack et al. 2020)
59.91/57.41
81.83 82.95
48.1
GAN-synthesis (Loe ct al. 2018a
59.93
83.67 82.67
48.5
SIREN-vMF (Du
64.68 68.53
85.36 82.78
SIREN-KNN (Du el 2022a
50.45
89.67 88.80
VOS (Baseline) (1u et 2022e
/5..3/
88.70/ 85.23
48.9
MPD
41.28/ 46.45
90.54/ 88.03
49.2
[enddrcks Gimpel 2017
80,94 79.04
75.87 77.38
ODIN [Laang 2017
6
74.44 76.61
31.2
Mahalanohs ( Lee 2018h
55.74/47.69
85.71 88.05
31.2
Gram matrices Sastry & Dore
2020
60.93/77.55
74.93 59.38
31.2
Berkeley DeepDrive-1000 100k
Energy score Liu el al. 2020
60.06 64.97
77.48 / 79.60
31.2
Generalized ODIN (Hsu et al.
2020
$7.27 15.117
85.22 / 87.18
31.8
CSI Tack ct al.
2020
47.10/37.06
84.09 / 87.99
30.6
GAA--yynleesss Lee el 2018a
$7.03/
78. / 81.25
31.4
(Baseline)
Du ct al
2022e
44.27 35.54
86.87 88.52
MPD
37.24/ 26.76
88.56 92.23
31.4
pestess pedestriar
pedestri trit
DOO
00O
000
Figure Results on the 0OD images from MS-COCO. The first and second rows respectively
indicate results based on VOS (Du et al. (222c) and our method. The ID dataset is BDD-100k.
object detection (Du et al. 2022c Specifically, for a predicted bounding box b, the processes of distinguishing OOD objects are shown as follows:
S = exp B(b)
C(b) = { 0 if S<6 if S28
(10)
1 exp B(b)
For the output of the classifier C(-). we use the threshold mechanism Du et al. 2022ch to distinguish ID objects (the result is from OOD objects (the result is O). The threshold commonly set 0.95 so that high fraction of ID data is correctly classified. Finally, Algorithm shows the training and testing processes of our method.
EXPERIMENTS
For unsupervised OOD-OD, method is first evaluated two different benchmarks (Du et al. 2022c). Then, to further demonstrate the effectiveness, we evaluate our method on class incremental object detection (IOD) (Kj et al. 202T) and open-set object detection (OSOD) (Han et al. 2022
4.1 0OD-OD PERFORMANCE ANALYSIS
Table shows the performance of unsupervised OOD-OD. We can see that though different methods similar mAP performance, the ability of detecting OOD objects differs significantly. This
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance (%) analysis of class-incremental object detection. 'OD Ours' indicates that our method is plugged into iOD [Kj ct al. (021). Here, '50' and '75' separately represent that the mAP metric is calculated when the IOU threshold is set to 0.5 and 0.75.
10 10 setting
acro
sycle
bad
bost
boxil
bus
CAr
cut
chair
cow
lable
dog
hoese
bike
person
plant
sheap
sofa
train
mAP
OW-DEIR (20) (Cupta al
2022
61.8
09.1
67.8
45.8
47.3
78.5
78.4
75.6
36.2
71.5
57.5
75.3
76.2
77.4
79.5
40.1
06.8
06.3
75.6
64.1
65.7
ROSETTA (50) Yi 2012
74.2
76.2
64.9
54.4
57.4
76.1
84.4
68.
$2.4
67.0
62.9
63.3
79.8
72.8
78.1
40.1
62.3
61.2
72.4
66.8
66.8
(50) Kjetal X011
76.0
14.6
67.
55.0
57.6
75.1
85.4
770
43.7
70.8
90.
664
76.0
72.6
74.6
39.7
64.0
60.2
68.5
60.5
66.3
1DD + Ous
76.0
15.7
70.0
52.5
55.3
78.9
85.2
76.5
45.5
75.3
57.2
79.5
79.8
76.4
79.8
43.5
71.7
09.0
14.2
68.0
69.5
1DD (25) al 2021
39,0
26.5
28.4
19.4
24.2
47.2
56.7
41.
19.1
48.0
21.1
32.1
43.0
36.3
40L0
14.8
40.1
36.5
37.3
45.3
35.3
OD (277
42.2
41.3
29.6
22.3
22.8
53.7
58.0
41.4
21.5
42.6
24.6
32.9
39.8
41.4
58.5
15.5
44.2
36.5
35.0
45.6
36.5
setting
3210
eyele
hin
bost
bottle
CH
chit
COW
tbllc
dog
horsc
person
plant
sheap
train
MAP
OW-DFTR (50) (Cupto
77.1
76.5
69.2
79.8
84.2
K1D
49.7
79.6
79.0
85.4
65.1
73.9
69.4
ROSETIA (50) Yonk c1 2024
16.5
77.5
65.1
56.0
60.0
78.3
85.5
78.7
49.5
68.2
67.4
71.2
83.9
75.7
82.0
43.0
50.6
64.1
72.8
67.4
69.2
1DD (50) Ki 2021
18.4
19.7
66.9
54.8
56.2
17-2
84.5
19.1
47.2
15.0
51.8
74.7
$1.6
17.5
60.2
37.8
58.0
54.6
23.0
56.1
67.8
10D -ua
77.5
78.7
11.7
54.7
62.3
78.7
84.5
77.0
51.3
78.8
56,
79.7
79.8
71.0
77.8
18.9
65.0
61.8
74.9
67.1
70.5
(OD ak 2021
40.7
40.9
28.7
19.1
23.8
61.6
56.1
38.8
23.6
47.5
18.7
40.1
40.2
41.5
39.8
9.1
40.6
32.4
41.9
47.6
36.6
OD 0z
45.0
43.9
32.1
23.7
25.00
56.4
58.3
40.2
25.6
45.6
26.7
37.2
47.2
42.4
4016
15.3
43.1
20.2
45.7
48.2
38.8
194 setting
eyole
bid
chii
dog
person
plent
sheep
OW-DETR (50) (Supto
2013
13.8
19.4
ROSETTA (50) ora
84.6
49.2
71.0
76.1
1OD (50) (
7622
86.2
46.6
78.5
16.4
57.6
(OD (2l
78.3
77.9
73.5
57.4
84.7
M0.9
50.0
64.9
82.0
82.9
77.7
46.9
69.5
74.8
61.4
71.8
OD (75) Kjetal 2011
35.9
44.7
31.6
36.9
56.5
38.7
21.6
48.4
35.9
37.9
38.7
40.7
000 USS
40.9
45,0
38.7
23.2
32.0
56.2
62.6
40.3
24.9
48.9
28.7
463
41.8
42.4
41.5
18.4
44.0
37.9
44.6
48.2
40.3
Table 3: Performance analysis of OSOD. We report close-set performance (mAPk) on VOC, and both close-set (mnAPx) and open-set (WI, AOSE, APz) performance of different methods on VOC- COCO- (20. 40, 60). Here, '20', '40', and '60' indicate that the testing COCO images separately contain 20, 40, and 60 non- VOC classes.
Method
VOC
VOC-COCO-20
VOC-COC0-400
VOC-COCO-60
mAPkr
W1,
AOSE
mAPkr
APtr
WT
AOSE:
mAPer
APu-
W1
AOSE;
mAPxi
FR-CNN Ren al. 2015
80.10
18.39
15118
58.45
0
22:74
23391
55,26
0
18.49
25:12
55.83
0
FR-CNN Ren el al 2015
80.01
18.83
11941
57.91
0
23.24
18257
54.77
0
18.72
19566
55.34
0
PROSER 2hou
79.68
19.16
13035
10.92
24.15
19531
7.62
3:25
Douie m
79
18:18
12811
58.25
7.60
23.40
19752
$5.30
1.70
18.35
21415
0.53
DS (Minee at ai. 2016
80.04
16.98
12868
58.35
5.13
20.86
19775
55,31
3.39
17:22
21921
55.77
1.25
Openlleet Han et al. 2022
80.02
14.95
11286
58.75
14.93
18.23
16800
$5.83
10.58
14.24
18250
56.37
4.36
OpeniDet FOur
80.18
12,23
10160
59.88
15.89
13580
$7.51
11.82
12.08
16681
57.25
5,07
indicates that existing detection methods are easily affected by OOD objects. Meanwhile, based on FPR95 and AUROC metrics, our method significantly outperforms the compared methods. This not only demonstrates that synthesizing virtual features is feasible for OOD- OOD-OD but also shows that our MPD method could perform content-oriented feature synthesis effectively. Fig. shoow some detection results. Compared with the baseline method (Du et al., (022c). our method accurately localizes and recognizes OOD objects, which further demonstrates the superiorities of our method.
4.2 PERFORMANCE ANALYSIS OF IOD AND OSOD
To further demonstrate the effectiveness of our method, we verify our method on two different tasks, i.e. IOD (Kj et al. 2021) and OSOD (Han et al., 2022). We directly plug our method into the two tf methods (Kj et al. 2021 Han et al.] 2022). Meanwhile, we do not utilize the uncertainty loss. The training and testing processes are the same as the two baselines (Kj et al. [022]] Han et al. 2022). Table and show the detection results. We can see that plugging our method into the two baseline methods improves their performance significantly. This further demonstrates that leveraging the phase information indeed could synthesize expected features, which strengthens the discrimination ability of the object detector.
4.3 ABLATION AND VISUALIZATION ANALYSIS
In this section, we utilize PASCAL VOC as the ID data for training and MS-COCO as the OOD data to perform ablation analysis of our method.
Analysis of MPD. Our method mainly includes the module for synthesizing OOD features, the module for feature enhancement, and that for generating augmented features. In Table 4 we make an ablation experiment of our method. We can see that only synthesizing virtual OOD features could improve the ability of detecting OOD objects. This shows that leveraging the phase is beneficial
8

---

Published as a conference paper at ICLR 2024
(a) Input Image (b) Source Map
(c) Enhance
(d) Source Phase (c) Aug Phase (f) OOD Phase
Figure 4: Visualization of the Source map Fo, Enhanced map E (i.e., E ([ Po Fr])), Source
phase Po. Augmented phase Po. and OOD phase Pg based the OOD data (MS-COCO).
for obtaining the features owning different content. Besides, observe that performing feature augmentation and enhancement further boosts the performance, which demonstrates that our method could effectively synthesize the features enhancing the discrimination.
Reverse iteration number T. During reverse processes, we continually repeat the same oper- ations to obtain virtual OOD features and aug- mented features. Here, we analyze the impact of the iteration number. We do not change our method and training details. Table shows the results. Through multiple average operations, the content in the phase significantly weak- ened. It is difficult to utilize small number of iterations to acquire proper features involving plentiful content. We can that using more iterations is beneficial for synthesizing expected OOD and augmented features, strengthening the discrimination ability of the object detector.
Table 4: Ablation analysis of MPD for unsupervised OOD-OD. 'OOD' and 'Aug' separately indicate syn- thesizing virtual ood features and augmented features. 'Enhance' represents E = ([Fo.. Fo Fr]).
OOD Aug Enhance FPR95 | AUROCT mAP
44.15%
89.44%
48.9%
42.26%
89.91% %
49.1%
43.58%
89.62%
49.2%
41.28
90.54% %
49.2%
Analysis of modulated operations. In equation 4 to synthe- size augmented features effectively, we exploit the modulated operation to perform a transformation for the output of the OOD branch. Here, we use a newly-designed U-Net model to replace the modulated operation. We observe that the performance de- creases severely, e.g., the FPR95 value is increased by around 5.9%. The reason may be that using a new U-Net module increases the parameters, leading to overfitting. More ablation experiments are shown in Appendix.
Table Analysis of the iteration number T in the reverse stage.
T
FPR95 | AUROC 1 mAP
1
45.38%
89.31%
48.9%
2
43.94%
89.52%
49.1%
4
41.28%
90.54%
49.2%
Visualization analysis. Fig. we show some visualization examples. Compared with the original feature map Fo. the enhanced feature map E contains stronger object-related information. This shows that the object-related content in the averaged phase weakened during the forward process. Besides, the phase of OOD features is significantly different from that of original features and augmented features, demonstrating that our MPD method could effectively synthesize expected features that differ from the content of the original features and alleviate the impact of lacking OOD data.
5 CONCLUSION
For unsupervised OOD-OD, focus on leveraging the phase information to perform content-oriented feature synthesis and propose a new method, i.e., modulated phase diffusion. Specifically, the forward process gradually performs Gaussian Average the phase. Then, two different reverse processes separately designed to synthesize expected virtual OOD features and augmented features. Extensive experimental results three different tasks demonstrate the effectiveness of our method.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported in part by the National Key R&D Program of China (No. 2023YFC3305600) Joint Fund of Ministry of Education of China (8091B022149) 8091 B0207724))) National Natural Science Foundation of China (62132016, 62171343, 20071361, and 62102293), and Fundamental Research Funds for the Central Universities (ZDRC2102).
REFERENCES
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, pp. 1563-1572, 2016.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitude-phae phaseeeeppheeeeeee recombina- tion: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 458-467, 2021.
William T Cochran, James W Cooley, David L Favin, Howard D Helms, Reginald A Kaenel, William W Lang, George C Maling, David E Nelson, Charles M Rader, and Peter D Welch. What is the fast fourier transform? Proceedings of the IEEE, 55(10):1664-1774 1967.
Taylor Denouden, Rick Salay, Krzysztof Czarnecki, Vahdat Abdelzad, Buu Phan, and Sachin Vernekar. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprini arXiv:1812.000 1812.02765, 2018
Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In WACV, pp. 1021-1030, 2020.
Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects. In NeurlPS, 2022a.
Xuefeng Du, Xin Wang. Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't t know from videos in the wild. In CVPR, pp. 13678-13688, 2022b.
Xuefeng Du Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. ICLR, 2022c.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Yonatan Geifman and Ran Blniv..v. Selectivenet: A deep neural network with an integrated reject option. In ICML, pp. 2151-2159, 2019.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pp. 10696-10706, 2022.
Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer. In CVPR, pp. 9235-9244, 2022.
Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, and Gui-Song Xia. Expanding low- density latent regions for open-set object detection. In CVPR, pp. 9591-9600, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961-2969. 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and oust---dist examples in neural networks. ICLR, 2017.
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840-6851, 2020.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. In CVPR, pp. 10951-10960, 2020.
Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In CVPR, pp. 11502-11511, 2022.
Taewon Jeong and Heeyoung Kim. Ood-maml: Meta-learning for few-shot out-of-distribution detection and classification. NeurIPS, 33:3907-3916, 2020.
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. CVPR, pp. 5830-5840, 2021.
Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors their natural habitats. In ICML, pp. 10848-10865. 2022.
Joseph Kj. Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubrama- nian. Incremental object detection via meta-learning. IEEE Transactions on Pattern Analysis and Machine Inteligence, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International journal of computer vision, 128(7):1956-198) 2020.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting -iitt--ditti samples. ICLR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurlPS, 31, 2018b.
Sangrok Lee, Jongseong Bae, and Ha Young Kim. Decompose, adjust, compose: Effective normal-
ization by playing with frequency for domain generalization. CVPR, 2023.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of outtof-distriibtion image detection in neural networks. ICLR, 2017.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. NeurIPS, 33:21464-21471 2020.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. 11970, 2022.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. NeurlPS, 31, 2018.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. ICLR, 2020.
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. ICRA, pp. 3249-3249, 2018.
Alan Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE, 69 (5):529-541, 1981.
11

---

Published as a conference paper at ICLR 2024
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, volume 32, 2018.
Tal Reiss, Niv Cohen, Eliahu Horwitz, Ron Abutbul, and Yedid Hoshen. Anomaly detection requires better representations. arXiv preprint arXiv:2210.10773 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, pp. 91-99, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pp. 234 241, 2015.
Chandramouli Shama Sastry and Sageev Oore. Detecting -itoo--distibbution examples with gram matrices. In ICML, pp. 8491-8501 PMLR, 2020.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribttion detection with deep nearest neighbors. ICML, pp. 20827-20840, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. NeurIPS, 33:1 1839-11852, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 30, 2017.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't it know? NeurlPS, 34:2944:42290000 2021.
Renzhen Wang, Kaiqin Hu, Yanwen Zhu, Jun Shu, Qian Zhao, and Deyu Meng. Meta feature modulator for long-tailed recognition. NeuriPS, 2020.
Aming Wu and Cheng Deng. Discriminating known from unknown objects via structure-enhanced recurrent variational autoencoder. CVPR, pp. 23956-23965, 2023a.
Aming Wu and Cheng Deng. Tib: Detecting unknown objects via two-stream information bottleneck. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023b.
Aming Wu, Da Chen, and Cheng Deng. Deep feature deblurring diffusion for detecting out-of- distribution objects. In ICCV, pp. 13381-13391, 2023.
Binbin Yang. Xinchi Deng. Han Shi, Changlin Li, Gengwei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiaodan Liang. Continual object detection via prototypical task correlation guided gating mechanism. In CVPR, pp. 9255-9264, 2022a.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.000 200009099,, 2022b.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pp. 2636-2645. 2020.
Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In CVPR, pp. 4401-4410, 2021.
Yibo Zhou. Rethinking reconstruction autoencoder-basee based out-of-distribution detection. CVPR, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
For unsupervised OOD-OD, to overcome the limitation of lacking OOD data for supervision and improve the discrimination ability of the object detector, this paper proposes a new feature-level generator, i.e., Modulated Phase Diffusion, aiming to exploit the phase information to perform content-oriented feature synthesis. In the appendix, we provide implementation details, additional analyses, various ablation studies, and more visualization results.
A.1 EXPERIMENTAL SETUP
Implementation Details. We utilize Faster R-CNN (Ren et al. 2013) with Rol-Alignment layer (He al. 2017) as the basic detection model. ResNet-50 (He et al. (016) is taken as the backbone. For the forward process, we perform four Gaussian Average operations. a in equation is set to 0.8. For the reverse stage, the encoder and decoder in the U-Net model Eg all consist of three convolutional layers. All the experiments are trained using the standard SGD optimizer with a learning rate of 0.02.
Datasets. For unsupervised OOD-OD, we adopt PASCAL VOC (Everingham al., 2010) and Berkeley DeepDrive (BDD- 100k) (Yu et al. (020)) the ID data for training. Meanwhile, MS- COCO (Lin et al., 2014) and Openlmages (Kuznetsova et al., 2020) are taken as the OOD datasets to evaluate the trained model. And the OOD datasets are manually examined to guarantee they do not contain categories. Besides, PASCAL- VOC includes the following categories: Person, Car, Bicycle, Boat, Bus, Motorbike, Train, Airplane, Chair, Bottle, Dining Table, Potted Plant, TV, Sofa, Bird, Cat, Cow, Dog. Horse, Sheep. BDD- 100k contains the following classes: Pedestrian, Rider, Car, Truck, Bus, Train, Motorcycle, Bicycle, Traffic light, Traffic sign.
For IOD, we follow the standard evaluation protocol (Kj et al., 202T) and evaluate our method on PASCAL VOC (Everingham et al. 2010). We initially learn 10, 15, or 19 base classes, and then introduce 10, 5, or new classes as the second task. Finally, for OSOD, we follow the work (Han al. 2022) and utilize 20 VOC classes and 60 non-VOC classes in COCO to evaluate method under different open-set conditions.
Metrics. For OOD-OD, report: (1) the false positive rate (FPR95) of OOD objects when the true positive rate of ID objects is at 95%; (2) the area under the receiver operating characteristic curve (AUROC); (3) mean average precision (mAP). For OSOD, use Wilderness Impact (WT) (Dhamija et al., 2020) to measure the degree of unknown objects misclassified to known classes. And we also use Absolute Open-Set Error (AOSE) (Miller et al., 2018) to count the number of misclassified unknown objects.
A.2 MORE EXPERIMENTAL DETAILS OF IOD AND OSOD
To further demonstrate the effectiveness of method, we verify method IOD and OSOD. Here, we directly plug our method into two baseline methods and do not calculate the uncertainty loss. The training details are the same the baselines.
To effectively exploit the synthesized virtual OOD features, we train a binarized classifier, i.c., the output of the known category is 1, and the output of the virtual OOD features is 0. Meanwhile, still utilize a L---ivereeece to constrain the prediction consistency between the original features and augmented ones. By these operations and minimizing the cross-entropy loss, the discrimination ability of the object classifier could be further strengthened.
A.3 FURTHER DISCUSSION OF VISUALIZAATION RESULTT
To alleviate the impact of lacking OOD data, we focus on phase-based content-oriented feature synthesis. Fig. 5 shows more visualization examples. We can see that compared with the original feature map Fo. the enhanced feature map E contains stronger object-related information and weaker background-relevan
information. This shows that after the multiple averaging operations, the object- related content of the forward output is gradually damaged, meeting the purpose of the forward diffusion. Besides, we can observe that the OOD phase information is significantly different from the phase of the augmented features. This further indicates that the content of the synthesized virtual
13

---

Published as a conference paper at ICLR 2024
(a) Input Image (b) Source Map
(c) Enhance
(d) Source Phase
(c) Aug Phase
(f) OOD Phase
Figure 5: Visualization of the Source map Fo, Enhanced map E (i.e., E = ([Fo. Fo Frl)), Source
phase Po. Augmented phase Po. and OOD phase Po based on the OOD data (MS-COCO).
OOD features differs from that of the input features, which attains the motivation of this paper and demonstrates the effectiveness of our method.
A.4 MORE ABLATION EXPERIMENTS OF HYPER- PARAERTRAAMSESS AND MPD METHOD
For our method, utilize the hyper-parameter for the K L-divergence loss (equation 7 the hyper-parameter for the loss LMPD (equation 9) and the hyper-parameter r for the loss Luncerty (equation D. Since the uncertainty loss Luucerty I8 directly related the current task, the value of should be set larger than and a. Meanwhile, if X and are set to small value, the role of the two corresponding losses will be weakened optimization. Thus, it meaningful to set proper values for these
Here, we take PASCAL VOC as the ID data and MS-COCO as the OOD data to perform an ablation analysis of pyree--arameters. And we only change these hyper- prree-parameters and keep other modules unchanged.
Analysis of The hyper-parameter aims to minimize the K L-divergence between the prediction probabilities from O, and Oaug. In the experiments, we observe that when À is set to 0.01, 0.001, and 0.0001, the performance FPR95 is 42.94%, 41.28%, and 42.16%.
equation is to balance the detection loss and the loss that
Analysis a. The goal the hyper-parameter a in equation 9 is to weigh the importance of the module of MPD. In the experiments, find that when is set to 0.01, 0.001, and 0.0001, the corresponding FPR95 performance 43.25%, 41. 41.28%, and 41.94444 11.92%.
Analysis of In this paper, the Luncerty- the experiments, we observe that when is set to 0.5, 0.1, and 0.01, the corresponding performance of FPR95 is 43.89% 41 28%, and 43.13%
in equation 9 is to constrain the uncertainty loss
14

---

Published as a conference paper at ICLR 2024
GOD
OOD
OOD
DOD
OOD
ODD
OODL
OOD
OOD
OOD
OOD
JOO
000
OOD
OOD
COF
OOOD
800
OOD
000
ODD
ODO!
8
NERRY
OOD
OOD
OOD
ODD
Figure 6: OOD detection examples based on our method. Here, we use BDD-100 as the in- distribution data and MS-COCO as the OOD data. We can see that our method accurately distinguishes OOD objects, which shows the effectiveness of our method.
15

---

Published as a conference paper at ICLR 2024
person 59%
person 100%
bird 60%
bicycle 100%
boat 100%
bicycle 100%
horse person 93%
person 100%
horse 100%:
bird 100%
100%
person 100%
MIL
dog 100%
bicycle 100 Vo
sheep 100%
car 100%
bicycle
airplane 100%
train 100% MMM
person 99% 100%
airplane 100%
bicycle 90%
horse 100%
person 100%
person 100%o
100
horse 100%
bird 99%
COW 100%
car 100%
51%
1009617% %
boat 96%
dog 100% %
dog 100%
dog 99%
person 10
bont 79%
100%
bird 100%
bird 99%
motorcycle 100%
Figure 7: Detection results based PASCAL VOC. We that method accurately localizes and recognizes objects these images, e.g., the dog, car, cow, and person, which shows that our method is effective for in-distribution data.
16

---

Published as a conference paper at ICLR 2024
Table 6: Definitions of notations used in our method.
Notations
Definition
Fo
The feature map extracted by the backbone network.
A. Po
The corresponding amplitude and phase of Fo-
(
Gaussian Kernel with the variance a. The convolutional output of the t-th Gaussian Average operation.
Pj
e.()))
The learned U-Net model.
D1-1
The predicted result of the U-Net model.
PI-1
The decomposed phase based on Di-1- The IFFT output of P and A. The synthesized virtual OOD map. The oulpul of the modulation mechanism. The decomposed phase based on D;-1- The augmented feature map.
F--1
Fo
D
Pi-1
Fo
Analysis of the Kernel Size. In equation [ during the forward process, exploit the Gaussian Kernel to perform the average operation. Here, we make an ablation analysis of the kernel size. And other modules are kept unchanged. Taking PASCAL VOC ID data for training and MS-COCO as OOD data for evaluation, when the kernel size is separately set to x 3. 5 x 5, and 7 x 7, the corresponding performance is 41.76%, 41.28%, and 42.84%
In Fig. 6 and 7 we show more detection results from our method. We can see that our method could distinguish ID objects and OOD objects accurately, which further demonstrates the effectiveness of our phase- based diffusion method.
A.5 DEFINITIONS OF NOTATIONS
Table gives the definitions of notations used in our method.
A.6 FURTHER DISCUSSION OF OUR METHON
For unsupervised OOD-OD, since there no OOD data available, one feasible solution is to leverage the known ID data synthesize expected OOD features that differ from the content of ID features. To this end, in this paper, we explore leveraging the phase component to generate expected OOD features, which is instrumental in improving the ability of discriminating OOD from ID objects. In the following, we will give more discussions about our method.
On performing Gaussian Average. During the forward process, existing diffusion methods (Ho) ctt al. 2020 Luo 2022) are fixed to Markov chain that gradually adds Gaussian noise to the input. And notable property of the forward process is that it admits sampling the diffusion results at an arbitrary timestep due to the characteristic of Gaussian noise (Ho et al. 2020).
Since method is to leverage the phase components to generate expected features, directly adding noise to the phase may rapidly destroy the content in the phase. affecting the performance of the diffusion model. To this end, replace adding noise with gradually performing Gaussian Average. Compared with globally adding noise, Gaussian Average is local operation and could reduce the damage the content. The overall forward process can be described as follows:
(P.(P.,1) Pt-1 G(o),
(11)
where P: represents the i-th output phase of the forward diffusion process.
Meanwhile, the forward process owns a notable property that it admits averaging the phase Pt at an arbitrary timestep t:
Pt = Pt-1 ((o) = (Pr-2 G(()) ((o) = Po G(a)
G(a) = Po G(a.),
(12)
where o, -i= Like traditional diffusion methods (Ho et al. 2020 Luo 2022), this property admits acquiring the diffusion results at an arbitrary timestep t. Experımental results on multiple datasets demonstrate the effectiveness of this operation.
17

---

Published as a conference paper at ICLR 2024
Baseline
Ours
0.7
0.6
loss
0.5
0.3
0.2
19
2017
4015
6012
8010
10008
12006 14003
16001 17999
iteration
Figure 8: The training loss of the baseline (Du al. 2022c and our method.
On the reverse process. The reverse process aims to generate virtual OOD feature Fo from an averaged result Fr:
po(Frro) = p(
II pa (F.,11P.) t=
(13)
where Pr Fr. An U-Net network is taken as the decoder to predict the output F, involving OOD-related information.
The reverse loss Creverse can be represented as follows:
Lreverse -EgDxxx((((PF)))))((( Σ D
log (Fol Polli fi))
(14)
where the first term is a constant during training and can be ignored. By minimizing the last two terms, the gap between the synthesized OOD features and ID features could be enlarged, promoting the synthesized OOD features to contain plentiful content that differs from the ID input. Concretely, for the second loss term, we explore making the output of the decoder gradually contain less ID- relevant content to promote the decoder to own the ability of generating OOD samples (as shown in equation 3). Meanwhile, for the last loss term, maximize the loss Lais to further enlarge the gap between the OOD and ID features. Extensive experimental results and visualization analysis demonstrate that our method could synthesize expected features effectively, improving the ability of discriminating OOD objects.
A.7 VISUALIZATION OF THE TRAINING LOSS CURVE
In Fig. we show the training loss curves of the baseline Duu et al. 2022c and our method. The decreasing speed of the loss value is around consistent with the baseline method. We can that the loss curve our method is significantly lower than that of the baseline, which further demonstrates the effectiveness of our method.
18


Reviews:
[{"review_id": "q1THcRYq4K", "reviewer": "Reviewer_PJVM", "summary": "This paper focuses on the out-of-distribution object detection (OOD-OD) task, and proposes a method named MPD to tackle it from the frequency perspective. Following the previous method VOS that alleviates the OOD-OD problem by adaptively synthesizing virtual outliers, MPD also attempts to synthesize suitable virtual OOD features as well as generate augmented features for supervised training. Different from VOS that assumes a class-conditional multivariate Gaussian distribution of the feature space, MPD in this paper tries to add noise to the phase domain in the diffusion way. Moreover, the authors find that the Gaussian Average for processing each step is better than directly adding noise. Many experiments and ablation studies have verified that MPD is superior than previous methods for dealing with OOD-OD.", "questions": "Overall, the method proposed in this paper is innovative and effective. Please go back to the two questions I mentioned in the weaknesses of items 2 and 3. Let me shorten these two questions as below:\n1. How the proposed MPD increases the number of network parameters?\n2. How about using advanced basic detectors instead of the outdated Faster R-CNN?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I think this paper has at least the following several major contributions:\n\n1. The authors approach the OOD-OD problem from the phase domain of the extracted image features, which is an aspect that is interesting and rarely studied.\n\n2. Introducing diffusion to generate different features is a new attempt in OOD. After discovering that simply and directly adding noise according to the original method was problematic, the authors proposed their own effective improvement strategies.\n\n3. Quantitative experimental results prove the advancement of MPD in many OOD-OD benchmarks.", "weaknesses": "Similarly, we summarize the weaknesses of this paper as follows:\n\n1. Actually, studying the processing of features from a phase perspective is not the first of its kind in this paper. In other words, the method [1] has proven that phase-related features are content-oriented in the DG field, which is very similar to OOD. This paper directly uses such a conclusion of DG in OOD and cannot be regarded as a complete innovation.\n\n2. Generally speaking, there are quite a few steps in the continuous transformation of the Diffusion model, such as dozens or even hundreds of steps. The method in this paper seems to only use up to 4 interations (T=4 in Table 5). Why not try more steps? Is it because more parameters are introduced (such as the U-Net model for predicting feature maps, and two branches for generating new features in OOD) that it is inconvenient to increase T to a too large number? If so, the authors need to explain clearly how the new method MPD increases the number of parameters compared to the original basic detector, such as the used Faster R-CNN.\n\n3. As we all know, Faster R-CNN is a classic but outdated detector. It gives a weak baseline of detection comparing to recent new ones. The actual value of OOD-OD is to achieve robust and generalizable object detection in real applications. Thus, using advanced basic detectors such as YOLOv5 [2], YOLOv8 [3], TOOD [4] and DETRs is more meaningful. And it will be important to see if the proposed MPD works or not on these superior detectors. It may not be practical to do more experiments. The authors could give similar explanations and discussions. For example, is MPD universal to these superior detectors?\n\n[1] Decompose, Adjust, Compose: Effective Normalization by Playing With Frequency for Domain Generalization, CVPR 2023\n\n[2] https://github.com/ultralytics/yolov5, YOLOv5 2020\n\n[3] https://github.com/ultralytics/ultralytics, YOLOv8 2023\n\n[4] TOOD: Task-aligned One-stage Object Detection, ICCV 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjps01holv", "reviewer": "Reviewer_3rox", "summary": "This paper focuses on the OOD object detection problem and propose to detect unknown object without relying on auxiliary OOD data. This paper exploits ID data to generate OOD data by considering the phase information in frequency spectrum. A modulated phase diffusion (MPD) is designed, with some detailed forward and reverse compuation. Experiments on several tasks show the effectiveness.", "questions": "1. How about the conventional unknown object detection based on a simple threshold, such as entropy based.\n2. Minimizing the KL between ID and OOD is strange in Eq. 7.\n3. In Eq. 9, there are many losses, which makes the training not easy.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+The proposed technical framework sounds good, by leveraging frequency information, U-net and augmented features.\n+ Experimental results are comparable to previous SOTA models.", "weaknesses": "- This paper writting has a large space for improvement and not easy to follow. Although the authors presented the main motivations of this paper, there are still many places unclear. The phase information of ID features is used to generate OOD features. Since phase represents more the content, the amplititude information may be more important for different styles (OOD featuers).\n-The motivation on the augmented features (ID or OOD?) is not estabilished.\n-What is the difference between OOD-OD and open-set OD? The authors seem list them as different. But as I see, they are the same problem.\n-The forward and reverse process are not clear due to the poor writting.\n-Fig.1 and Fig.2 are redundant which evens show similar objective about the proposed MPD. Also, the designed method seems complex and not easy to follow.\n-I also concern about the claim \"lacking unknown data\". Since OOD-OD is problem setting, it is rational to suppose some categories are unknown, such as open-set OD.\n-Lacking the visualization results of phase based OOD data synthesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZaHDjQkprb", "reviewer": "Reviewer_A444", "summary": "This manuscript attempts to use the Diffusion model to solve the unsupervised out-of-distribution object detection (OOD-OD) task. This method uses two inverse processes to synthesize the phase information of unknown and known samples, respectively. The experimental results validate the effectiveness of the method.", "questions": "1. Why choose to generate phase instead of the original image or amplitude? What are the phase advantages?\n2. Why choose a 5x5 kernel for the Gaussian average? Has the author tried other types or sizes of kernels?\n3. If Gaussian noise is replaced by the Gaussian average, is the diffusion model still valid? Can you provide the formula derivation?\n4. The description of the OOD phase in Figure 2 is confusing, the meaning of approximately equal and not equal symbols is unclear.\n5. What are the shortcomings of this method? Does its training and reasoning time have any advantages compared to previous methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Gradual phase averaging: Instead of adding noise, MPD gradually performs Gaussian averaging on the phase of extracted features. This helps to prevent rapid loss of content in the phase, ensuring that important information is preserved during the process.\n\n2. Experimental superiority: MPD has demonstrated superior performance in various tasks, including OOD-OD, incremental object detection, and open-set object detection. The experimental results validate the effectiveness and advantages of the MPD method in promoting safe deployment of object detectors.", "weaknesses": "The idea of replacing Gaussian noise in the diffusion process with Gaussian average operation seems to be an experimental result, lacking theoretical explanation and formula derivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)