TIB: Detecting Unknown Objects via Two-Stream Information Bottleneck
Authors: Aming WU
Cheng Deng
Venue: Submitted to ICLR 2023 (2023)

Abstract: Detecting diverse objects, including ones never-seen-before during model training, is critical for the safe application of object detectors. To this end, a task of unsupervised out-of-distribution object detection (OOD-OD) is proposed to detect unknown objects without the reliance on an auxiliary dataset. For this task, it is important to reduce the impact of lacking unknown data for supervision and leverage in-distribution (ID) data to improve the model's discrimination ability. In this paper, we propose a method of Two-Stream Information Bottleneck (TIB), which consists of a standard Information Bottleneck and a dedicated Reverse Information Bottleneck (RIB). Specifically, after extracting the features of an ID image, we first define a standard IB network to disentangle instance representations that are beneficial for localizing and recognizing objects. Meanwhile, we present RIB to obtain simulative OOD features to alleviate the impact of lacking unknown data. Different from standard IB aiming to extract task-relevant compact representations, RIB is to obtain task-irrelevant representations by reversing the optimization objective of the standard IB. Next, to further enhance the discrimination ability, a mixture of information bottlenecks is designed to sufficiently capture object-related information. In the experiments, our method is evaluated on OOD-OD and incremental object detection. The significant performance gains over baselines show the superiorities of our method.



Under review as a conference paper at ICLR 2023
TIB: DETECTING UNKNOWN OBJECTS VIA TWO-STREAM INFORMATION BOTTLENECK
Anonymous authors
Paper under double-blind review
ABSTRACT
Detecting diverse objects, including ones never-seen-bete before during model train- ing, is critical for the safe application of object detectors. To this end, a task of unsupervised out-of-distribution object detection (OOD-OD) is proposed to de- tect unknown objects without the reliance on an auxiliary dataset. For this task, it is important reduce the impact of lacking unknown data for supervision and leverage in-distribution (ID) data to improve the model's discrimination ability. In this paper, propose method of Two-Stream Information Bottleneck (TIB), which consists of a standard Information Bottleneck and a dedicated Reverse In- formation Bottleneck (RIB). Specifically, after extracting the features of ID image, we first define a standard IB network to disentangle instance representa- tions that beneficial for localizing and recognizing objects. Meanwhile, we present RIB to obtain simulative OOD features to alleviate the impact of lacking unknown data. Different from standard IB aiming extract task-relevant compact representations, RIB is to obtain task-irrelevant representations by reversing the optimization objective of the standard IB. Next, further enhance the discrim- ination ability, mixture of information bottlenecks is designed to sufficiently capture object-related information. the experiments, our method evaluated OOD-OD and incremental object detection. The significant performance gains over baselines show the superiorities of our method.
INTRODUCTION
With the rejuvenation of deep neural networks, for object detection, many advances |enn et al. (205); Redmon et al.] (016)) Carion et al.) (2020): [Chnn et al.) (2022) have been achieved. Most existing methods often follow a close-set assumption that the training and testing processes share the same category space. However, the practical scenario is open and filled with unknown objects, presenting significant challenges for object detectors trained based on the close-set assumption. To this end, a task of unsupervised out-of cut-of-diitribution object detection (OOD-OD) Du et al. (2022b) is recently proposed, whose goal is to accurately detect the objects never-seen-beie before during training without accessing any auxiliary data. Obviously, addressing this task is helpful for promoting the safe deployment of object detectors in real scenes, c.g., autonomous driving.
The main challenge of unsupervised OOD-OD is lacking supervision signals from OOD data during training Du et al. (2022b). In particular, as shown in the left part of Fig. [ an object detector is typically optimized only based on the in-distribution (ID) data. During inference, the detector could accurately localize and recognize ID objects but easily produces overconfident incorrect predictions for OOD objects. The reason that the object detector could not learn clear discrimination bound- ary between ID objects and OOD objects in the case of lacking OOD data for supervision. Thus, for this task, one feasible solution is extract simulative OOD data based on the ID data. And the simulative OOD data could be used to improve the discrimination ability of the object detector.
In order to obtain simulative OOD data, it is general to leverage generative methods, e.g., gener- ative adversarial networks Lee et al. (2018a) 8a) and mixup Zhang et al. 20018), synthesize OOD images. Though these methods have been demonstrated to be effective, using a large number of synthesized images may increase computational costs. Meanwhile, it is difficult to synthesized images to cover the overall object space, which may weaken the discrimination performance for certain unknown objects.

---

Under review as a conference paper at ICLR 2023
ID Training Data
TD Training Data
Two-Streem Information Rottleneck
OOD Testing Data
ID Feat.
* OOD Feat.
IB Foat.
RIB Feat.
Object Detector
Maximize Discrepancy
Backbone
Rol Feat. Extractor
x
x
Figure I: Two-Stream Information Bottleneck for OOD-OD. 'RPN' is with Rol Alignment. The green boxes are OOD objects. The red and black lines separately indicate the decision boundary between ID and OOD objects and that between ID objects belonging to dif- ferent categories. Due to lacking unknown data for supervision, the traditional object detector could not distinguish ID objects from OOD objects effectively, Our method aims to generate simulative OOD features by maximizing the prediction discrepancy between the features extracted by the IB module and that extracted by the RIB module, which enhances the discrimination ability.
Network
In this paper, we explore employing Information Bottleneck (IB) Tishby al. 20000; Alemi al. (2017) to obtain a series of simulative OOD features for training. Particularly, we propose a method of Two-Stream Information Bottleneck (TIB) to improve the discrimination ability of the object detector, which mainly consists of standard Information Bottleneck and dedicated Reverse Infor- mation Bottleneck (RİB). Specifically, as shown in the right part of Fig. [[] given an ID image as the input, backbone network, e.g., He et al. (2016). is used to extract the corresponding repre- sentations. Then, standard variational Alemi al. (2017) is defined to decompose an Instance map from the backbone representations, which is instrumental in localizing and recognizing objects accurately. Besides, standard IB struggles to extract maximally compressed features of the input while preserving much task-relevant information as possible Lee et al. (202I). Whereas, OOD features could be considered irrelevant to the current task. Thus, we present RIB to obtain an OOD map used extract task-irrelevant representations via reversing the optimization objective of the standard IB. Concretely, by maximizing the discrepancy between the predictions from the Instance map and that from the OOD map, and simultaneously minimizing the classification loss, the OOD) map could be promoted to contain plentiful object-irrelvant information, which is beneficial for extracting simulative OOD features and improves the discrimination ability.
Furthermore, recent research |Schulz et al. (2020) has shown that IB is an effective mechanism to capture object information. Inspired by this idea, we explore designing mixture of information bottlenecks to purify object-related information from multiple different facets. Finally, by com- bining the information, the discrimination ability could be further enhanced. the experiments, our method is separately evaluated on OOD-OD and incremental object detection Kj et al. 2021 Extensive experimental results demonstrate the superiorities of our method.
The contributions of our work are summarized follows:
We propose method of Two-Stream Information Bottleneck consisting of standard IB and a dedicated RIB. Particularly, RIB aims to obtain simulative OOD features by maxi- mizing the prediction discrepancy between ID features and OOD features, which reduces the impact of lacking unknown data for supervision.
We design a mixture of information bottlenecks to purify object-related information from multiple different facets, which is beneficial for enhancing object-related information in the features for classification and improves the detection performance.
Experimental results show that our method could effectively improve the performance of OOD-OD and incremental object detection. Particularly, for PASCAL VOC Everingham ct (2100), compared with the baseline method Du et al. (2022b), our method significantly reduces FPR95 by around 10.42%.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
OOD detection. To promote the safe application of models in practical scenarios, OOD detection Pimentel et al. (2014); Yang al. (20Tb) has attracted much attention, whose goal is to distinguish ID data from OOD data. Most existing methods [ee et al. (2018b); Hendrycks et al. (2019): Leee al. 8a) Liang et al. (017)) Lee et al. (2018b) focus on OOD detection for image classification and exploit regularization operation or auxiliary dataset to address this problem. Particularly, Bendale et al. [Bendale & Boult (016) developed the OpenMax score for OOD detection based on the extreme value theory. Mcanwhile, there exist some methods that explore utilizing temperature scaling Liang et al. (0177) generative models Lee et al. (2018b); |Serrà et al. (2019), or ensemble methods Vaas al. (2018); Choi et al. (2018) to calibrate the distribution of the softmax score. Besides, Liu et al. Liu et al. (2020) proposed to leverage the energy-based idea to address OOD detection, which opens a new solution. Yang al. (Yang et al.] (2021a) and Zhou Zhou (2022) separately leveraged the idea of semantically coherent and that of reconstruction to distinguish OOD data. Though these methods have been shown be effective, since object detection involves object localization and recognition, these methods could not be directly used for OOD-OD.
Recently, unsupervised OOD-OD Du et al. (2022b) is proposed to determine whether detected ob- jects belong to al. Du al. (2022b) proposed to synthesize virtual outliers reduce the impact of lacking un- known data for supervision. The work Du et al.] (2022a) explores learning more aaknown-relatee knowledge from an auxiliary video dataset, which could not used for unsupervised OOD-OD. Harakeh et al. Harakeh & Waslander (2021) mainly focused uncertainty estimation for the lo- calization branch, which could not well address OOD object detection that includes localization and classification. Different from the above works, in this paper, we propose method of Two-Stream Information Bottleneck to reduce the impact of lacking unknown data and improve the discrimina- tion ability via reversing the optimization objective of standard information bottleneck. Extensive experimental results demonstrate the effectiveness of our method.
or not without accessing an auxiliary dataset. For this task, Du
Information bottleneck. Recent research Wang al. (022);; Lee al. (02 )); Schulz et al. (2020)
has shown that IB Tishby et al. (2000) is a promising mechanism to reveal the principle of neural
networks through the lens of information stored in encoded representations of inputs. Given two
random variables X and Y, the optimization objective of IB can be described as follows:
max (TTYY) (I(;; T
(1)
where I(T; and Y) are the mutual information of representation T towards inputs X and labels Y, respectively. 3 controls the tradeoff. By this optimization objective, the intermediate rep- resentation T can be promoted to contain compact relkvaele information. Recently, some works Ahuja et al. (2021): Li al. (2022) have shown that IB is helpful for extracting domain-invariant rep- resentations, which improves the generalization ability. Besides, there exist some works Schulz et al.) (2000); Kim et al. (202T) that indicate using IB could capture object-related information, which is beneficial for boosting the performance and enhancing the interpretability. In this paper, we explore exploiting IB to obtain simulative OOD features for training and design mixture of IB to further enhance the object-related information. Extensive experiments on OOD-OD and incremental object detection have shown that our method is instrumental in improving the discrimination ability.
TWO-STREAM INFORMATION BOTTLENECK
For unsupervised OOD-OD, the object detector is trained based on the ID data X denotes the set of ID images, Y is the label set, and B indicates location information. During inference, give an image x* including OOD objects, the trained object detector should accurately distinguish ID objects (the output is 1) from OOD objects (the output is 0).
where
3.1 OBJECT-RELATER INFORMATION EXTRACTION
Since object detection involves two subtasks, i.e., object localization and classification, for OOD- OD, the model should first localize OOD objects and ID objects. Then, the model could accurately distinguish ID objects from OOD objects, while correctly classifying ID objects. To attain this goal,
3

---

Under review as a conference paper at ICLR 2023
Uncertainty:
Uncertainty Loss
MIB: Mixture of IB
Gating
Agg: gggnogaton Operation
Gating: G Gating Operation
MIB
Lac
Means
Pin
Infor Enc:
IH
RPN
O
Rol Feat.
Extractor
Agg
Variances
Instance Map
Infor Eren
Qn
E
cls
Backbone
Shared
Shared
OOD Map
L
dis
Meens
Input Image
R
Rol Foot.
P Pood
Density
RIB
RPN
Extractor
uneertainty
Negative Energy Score
Variances
Figure 2: The architecture of our method. 'Infor Enc' indicates information enhancement. Our method mainly consists of an IB branch and a RIB branch. Particularly, the IB branch aims to capture plentiful object-related information by optimizing the objective of information bottleneck. Meanwhile, after obtaining the high-level features from the detection head, mixture of IB is de- signed to further enhance object-related information, which is beneficial for improving the discrim- ination ability. Besides, to alleviate the impact of lacking unknown data for training, we propose RIB to generate simulative OOD features by maximizing the loss Ldis. Here, is worth noting that during the RPN process, for the OOD map, we only perform Rol Alignment based on the proposals O extracted from the Instance map. Finally, the ID features from the MIB and the simulative OOD features are all used to calculate the uncertainty loss Luncertainty.
is important to extract plentiful object-related information. As IB owns the advantage of captur- ing compact task-relevant information Alemi et al. (2017), we explore exploiting IB to compress ibjeettirrelevant information (e.g.. the background information) in the extracted features, which is beneficial for improving the discrimination performance.
Concretely, as shown in Fig. 2 we follow the baseline work Du al. (2022b) and adopt the widely used object detector, i.e., Faster R-CNN Ren et al. (2015), as the basic detection model. Given an input image, we first employ the backbone network, e.g., ResNet He et al. (2016), to extract the corresponding feature map F E R=xxxe, where w. h, and separately denote width, height, and the number of channels. To obtain rich object-related information, we exploit the constraint of vari- ational information bottleneck (Alemi et al. (2017) to further encode the feature map F. Specifically, we separately define a convolutional network W μu and Wo1 to estimate the corresponding means and variances. The encoding processes of F are shown follows:
#1 = Wp1 F, 01 = Wo F. Z = + cxx((())
where E wxxxc and 0 E wxxx are the estimated means and variances. c iidicaaees Gaussian noise sampled from (000 I). '*' represents the convolutional operation. The encoding output is denoted as the Instance map Z E Rexhxe.
Next, Z is taken the input of the RPN module to extract series of object proposals O. Based on O, Rol-Alignment operation followed by Rol-Feature extraction is performed on to obtain the output Pln =x= where z and respectively denote the number of proposals and channels. Since the object proposals usually contain much object-irrelevan information (c.g., the background information) that may weaken the discrimination performance, to this end, we design a mixture of information bottlenecks consisting of multiple branches to further enhance object-related informa- tion. For each branch, based on Pin. we first define two fully-connected networks to separately estimate the corresponding means and variances. Then, we perform encoding operation of Pi::
P. (Pin):
P. = (P,m),
Q = P. + exx((?))
(3)
where i = 1, ..., 'n' is the number of IB. & and Φy represent two different fully-connected networks. Pld R= and p E ==* are the estimated means and variances. Q, E x=*x denotes the output encoding results of the current branch.
Since P., contains the information belonging to multiple different objects and much background in- formation, exploiting multiple branches of information bottlenecks is beneficial for purifying object- related information from multiple different facets. Next, we first define gating operation to aggre- gate the information from different IB. By means of the residual operation between the aggregated
4

---

Under review as a conference paper at ICLR 2023
information and the imput Pmn the object-related information in Pi could be enhanced. The overall enhancing processes are shown as follows:
P
n A
G
Σ G. Qi: i=1
== A= a. Pins
Qi
where Pin R and Q R separately represent the average results of Pm and Qi. G3 is the calculated gating weight.
Rax indicates the aggregated results. Here, a E Rax* denotes the learnable sigmoid weight, i.e., v (Pim,,, where V is fully-connected network. Finally, E E R indicates the output enhancing result.
As shown in Fig. 2 during training, E is taken as the input of the classifier and regressor to calculate the classification and localization losses. The joint training objective shown as follows:
Σ KLIp(QU(P..) ==1
LIB Lils + Dloc + 3. (KLIp(2)F),
(5)
where Les and Lioe separately denote the classification and localization losses. is a hyper- parameter. In the experiments, / is set to 0.0001. Following the information theories for deep learning Alemi et al. (017). we define r(.) as a prior marginal distribution, which is modeled as standard Gaussian N (0. (0, Obviously, by minimizing the task loss and the K KT--divereence loss, the dependence between and and that between Q; and Pmn are reduced, indicating that Z and Qi encode plentiful object-relevant information from the input and Pm. which is instrumental in improving the discrimination performance.
3.2 SIMULATIVE OOD FEATURES GENERATION
For unsupervised OOD-OD, one of the major challenges lies in lacking unknown data for super- vision, which is prone to producing overconfident incorrect predictions for OOD objects. To this end, we propose RIB method to generate simulative OOD features by reversing the optimization objective of the standard IB Alemi et al. (2017), which reduces the impact of lacking unknown data and improves the ability of distinguishing OOD objects.
Concretely, shown in Fig. based on the feature map F from the backbone network, first define a convolutional network Wwz and Wa2 to estimate the corresponding means and variances and leverage variational information bottleneck Alemi et al. (2017) encode F:
W,2 F.
= Wa2*
R 12 + exp(o2),
(6)
where R E =xxx indicates the encoding OOD map. Next, based on the object proposals O, Rol- Alignment operation followed by Rol-Feature extraction performed on R obtain simulative OOD features Poud E R To promote Pood to contain plentiful task-irrelevant information, we
explore reversing the optimization objective of the standard IB (as shown in equation T):
[(R, Pona); ); F) AI(Poodi Y): R.Poad
max
where \ controls the tradeoff. The goal of equation 7 to enforce the extracted and Pood from the input F to encode much less task-related information, which is beneficial for obtaining simulative OOD features for supervision. To attain this goal, we explore maximizing the prediction discrepancy Lais between E and Pood. The processes are shown follows:
1 K
Lais
Ip(Pond)) p(F)ss.
(8)
where p( ((oodd) and p(E), denote the prediction probability for class k. respectively. K is the num- ber of ID categories. Meanwhile, the classifier with shared parameters is used to produce p( P(ooo)) and pEE). Besides, since Rt is directly encoded based on the input F, R could be considered to be related to F. Here, we do not use a mutual information constraint to enhance the dependence between R and F. By maximizing the loss Lais while minimizing the task loss, the gap between Pond and will be enlarged, which promotes Pood to contain plentiful information unrelated to the ID objects. Finally, achieve the goal of distinguishing OOD objects from ID objects, Pood and E
5

---

Under review as a conference paper at ICLR 2023
used calculate uncertainty loss Du et al. (2022b). which aims to regularize the detector to produce a low OOD score for the ID object features, and a high OOD score for the simulative OOD features. The processes are shown follows:
exp ((a)
Cuncertainty E 1log + exp S(u) Pood l-log exp E(v)
(9)
where E() is the object-level energy score Du et al. (202bb; Liu et al. (2020) During training, we can only access the ID data. The overall training objective shown follows:
L - LIB X
diss + T Luncrttintty
(10)
where À and 7 two hyper-parameters, which
0.001 and 0.1 in the experiments.
3.3 INFERENCE OOD OBJECT DETECTION
During inference, we use the output of the uncertainty loss for OOD object detection. Specifically, for predicted bounding box b, the detection processes are shown as follows:
exp E(b) E(b)
{
score(b) <: if score(b) 2 2.
score(b) + exp E(b)
D(b)
(11)
For the output of the classifier D(-), the commonly used threshold mechanism is leveraged dis- tinguish the ID objects (the result is 1) from the OÔD objects (the result 0). The threshold ry is set 0.95 that high fraction of ID data is correctly classified.
EXPERIMENTS
In the experiments, for unsupervised OOD-OD, follow the settings of the work et al. (2022b) and do not use any auxiliary dataset for training. And our method evaluated on two different benchmarks. Furthermore, to further demonstrate the effectiveness of our method, verify our method on the task of class-incremental object detection Kj et al. (202T), i.e., new classes are se- quentially introduced to the object detector.
4.1 IMPLEMENTATION DETAILS AND BENCHMARKS
Implementation Details. We use Faster R- R-CNN Ren et al, 2015) with the Rol- Aol-AAligmmet layer He et al. (2017) as the basic detection model. The backbone is ResNet-50 et al. (2016). The param- eters pre-traincs the Instance map (cquation 2 and OOD map (equation 6). we separately utilize two convolutional layers define WA1, W.1. W μ22 and W.2. For each branch of MIB (equation 3 respectively utilize two fully-connected layers to define and Φω And the number of the IB branches is set to 8. All the experiments trained using the standard SGD optimizer with learning rate of 0.02.
ImageNet Russakovsky et al. (2015) for initialization. For the generation of
OOD-OD Benchmarks, PASCAL Everingggg et al. (010)) and Berkeley DeepDrive (BDD- 100k) Yu al. (2020) datasets are taken as the ID training data. Meanwhile, we adopt MS-COCO Lin et al. (2014) and Openlmages Kuznetsova et al. (2020) as the OOD datasets to evaluate the trained model. And the OOD datasets are manually examined to ensure the OOD images do not contain ID categories.
Metrics. To evaluate the performance of unsupervised OOD-OD, we report: (1) the false positive rate (FPR95) of OOD objects when the true positive rate of ID objects is at 95%; (2) the area under the receiver operating characteristic curve (AUROC); (3) mean average precision (mAP).
4.2 PERFORMANCE ANALYSIS OF UNSUPERVISED OOD-OD
Table ] shows the performance of unsupervised OOD-OD. We can see that though these methods own similar detection performance, the ability of distinguishing OOD objects differs significantly. This shows that these detection methods easily affected by OOD objects. Thus, detecting OOD objects meaningful for promoting the safe application of object detectors. We can see that com- pared with baseline methods, method obtains the best performance of OOD object detection.

---

Under review as a conference paper at ICLR 2023
Table 1: The performance (%) of unsupervised OOD-OD. All methods trained based ID data
and do not use any auxiliary data. 1 denotes larger values are better and denotes smaller values are
better. 't' indicates that we directly run the released code obtain the results. 'Bbone' and 'R50' separately represent backbone network and ResNet-50.
ID Data
Method
Bbone (Purams)
FPR95
AUROC 1
mAP (ID)
OOD: MS-COCO/ Op0enlmaeees
Hesdrrccysk Gimpel
81.91
ODIN Liang et 2017
59.82 63.14
82.20/ 82.59
48.7
Mahalanobis (Lee ct ai. 2018b
67.73 65.41
81.45/ 81.48
48.7
Gram murrices Sastry & Dore
2020
62.75
79.88
PASCAL- VOC
Energy Lu (4.44
A100
56.89/ 58.69
83.69 82.98
48.7
Generalized OSu et
2020
59.57 70.28
83.12/79.25
48.1
Tack 2020
59.91
81.83 82.95
GAN synliness
201
60.93 59.97
83.67 82.67
48.5
VVOS (Baseline Du al.
20226
R50 (41.4M)
51.97 56.81
87.55 83.37
48.1
Two-Stream IB
R30 (54.7M)
41.55 47.19
90.36 88.09
49.2
MSP MSP Hendrycks & Gimpel
201
80.94 79.04
75.87 77. 38
31.2
ODIN Lian el 201
62.85 58.92
74.44 76.61
mobil Loe at 700
10
55.74 47.69
85.71 88.05
31.2
Gram matrices Sastry & Lore
2020
60.93 77.55
74.93 /0.0999.30
31.2
BDD- 100k
Energy ODIN
54.97
Generalized
2020
57.27 50.17
85,22 87.18
31.8
Tack et al. 2020
47.10 37.06
84.09
87000/8880000 .9
30.6
Gynlssyll Jee
50.61
!VOS (Baseling
R50 (41.4M)
50.25 41.06
83.92 86.88
Two-Stream IB
R50 (54.7M)
36.85 24.00
88,47 92.54
313
pedestrian
padestran
OOD
SGID
OBO
000
XXX
Figure 3: Detection results the OOD images from MS-COCO, The first and second rows respec- tively indicate results based on VOS Du et al. (2022b) and our method. The in-distribution dataset BDD-100k. Blue boxes represent objects detected and classified one of the ID categories. Green boxes indicate OOD objects. We can see that our method accurately determines OOD objects.
Particularly, compared with VOS Du et al. (022b) that aims to synthesize virtual outliers, based on FPR95, for PASCAL VOC Everingham 10.42% and 9.62%. For BDD- 100k Yu et al. (2020). our method outperforms VOS by around 13.4% and 17.06%. This shows that our method of Two-Stream Information Bottleneck is beneficial for ex- tracting simulative OOD features, which reduces the impact of lacking unknown data for supervision and improves the discrimination ability of the object detector.
al. (2010), our method outperforms VOS by around
detection examples. For these images, the baseline method Du et al. (2022b) does not detect all OOD objects accurately. Taking the sccond column as an example, VOS Du al. (2022b) misclassifies the dog into the Pedestrian category. We can that method correctly localizes and recognizes OOD objects, further demonstrating its effectiveness.
In Fig. 3 show
4.3 PERFORMANCE ANALYSIS CLASS-INCREMENTAL OBJECT DETECTION
To further demonstrate the effectiveness of our method, we evaluate our method on class-incrementee object detection Kj et al. (2021) and follow the standard evaluation protocol Kj et al. (202T). We initially learn 10, 15, or 19 base classes, and then introduce 10, 5, 1 new class the second task. Meanwhile, we directly plug our method into the baseline method Kj et al. (202 and do not calculate the uncertainty loss (equation 9. The training details the same the baseline Kj al.

---

Under review as a conference paper at ICLR 2023
Table 2: Performance (%) analysis of class-incremental object detection hased on PASCAL VOC Everinnhgm ct al. (210). We consider adding 10, 5, and classes (highlighted in blue) to detector trained on the rest of the classes. 'iOD + Ours' indicates that our method is plugged into iOD Kj ct al. (222T) Here, 'P50' indicates that the mAP metric is calculated when the IOU threshold is set to 0.5.
10 10 setting
sero
cycle
bind
boat
bottle
bus
car
cal
chair
cow
table
dog
boese
bike
person
plant
sheep
SoG
Irain
TV
|P5
All 20
79.4
83.3
73.2
59.4
62.6
61.7
$6.6
83
56.4
61.6
71.9
83
85.4
$1.5
$2.7
49.4
74.4
75.1
79.6
73.6
75.2
Rist 10
78.6
78.6
72
54.5
63.9
81.5
87
78.2
55.3
84.4
73.4
Sul Training
35.7
9.1
16.5
733
9.1
18.2
9.1
26.4
9.1
6.1
57.6
57.1
72.6
67.5
73.9
33.5
53.4
61.1
66.5
57
57.3
Shmelkov et Shmelkov al
2017
69.9
70.4
09,4
54.3
48
68.7
73.9
68.4
45.5
58.1
59.7
12.7
73.5
12.2
66.3
29.5
63.4
61.5
69.3
62.2
63.1
Frotey I1on Ponieerr 20200
72.8
75.7
71.2
60.5
61.7
70.4
83.3
76.6
53.1
72.3
36.7
76.9
66.8
67.6
66.1
34.7
63.1
48.1
57.1
43.6
62.2
ORE
63.5
70,9
58.9
42.9
34.1
76.2
50.7
76.3
34.1
66.1
56.
70.4
50.2
72.3
$1.
42.7
71.6
68.1
77
67.
66.6
OW OW- 10
61.8
88.1
67.8
45.8
47.3
78.3
7866
36.2
71.5
57.5
75.3
76.2
77,4
79.5
40.1
66.8
66.3
756
64.1
65.7
RUSETTA
76.1
67.0
63.3
72.4
66.8
(OL/K] 2021 1OD
74.6
67.5
55.9
75.1
70.8
66.4
76
74.6
39.7
68.5
60.5
66.3
71.5
M0.6
77.1
75.4
79.4
79.4
73,0
70.4
15 5 setting
ae7o
cycle
bind
boat
bottle
bus
CAE
CUI
chair
COW
table
dog
boose
bike
person
planz
sheep
5003
Irain
1V
P50
20
79.4
K3.3
73.2
59.4
62.6
81.7
26.6
K3
56.4
81.6
71.9
$5.4
$1.5
$2.7
49.
74.4
75.1
79.6
73.6
75.2
Past 15
78.1
52.6
74.2
61.8
63.9
80.4
87
81.5
57.7
80.4
75.1
30.8
$5.8
$1.6
$3.9
53.2
Sul Training
12.7
0.6
9.1
9.1
3
0
8.5
9.1
0
3
9.1
0
3.5
2.1
9.1
37.6
51.2
57.8
51.5
59.8
16.8
Shmelkow al SSmelllw ot el
2017
70.5
70.2
68.8
59.1
53.2
75.4
79.4
78.8
46.6
504
59
75.8
71.8
78.6
69.6
13.2
61.5
63.1
71.7
62.2
665.0
Fretex Polieerrrr in 2020
66.5
78.1
71.8
54.6
61.
68.4
82.6
82.7
52.1
74.1
63.1
78.6
90.5
78.4
$0.4
36.7
61.7
59.5
67.9
59.
67.9
ORE Josep-ccenn Leeim RML
75.4
81
67.1
$1.9
55.7
77.
55.8
81.3
46.1
76.2
55.4
76.7
56.2
78.5
$2.1
32.8
63.6
54.7
77.3
64.6
68.5
OW- DW--ETT: (inu 1
77.1
76.5
69.2
51.3
61.3
79.8
84.2
81.0
49.7
79.6
58.1
79.0
$3.1
67.8
85.4
33.2
65.1
62.0
73.9
65.0
69.4
BA 1ung
20244
76.5
77.5
05.1
56.0
60.0
78.3
85.5
78.7
49.5
68.2
67.4
71.2
53.9
75.7
$2.0
43.0
60.6
64.1
72.8
07.4
69.2
10 KJ 65 gl. 2021
76.4
79.7
06.9
54.8
56.2
77.7
84.6
79.1
47.7
75
61.8
74.7
$1.6
77.5
90.2
37.8
58
54.6
75
56.1
67.8
1OD
78.0
78.7
13.5
53.8
63.6
80.1
85.6
N3.1
50.8
M0.9
06.5
$1.0
$3.1
77.2
77.7
34.8
06.6
63.0
71.8
06.0
71.1
setting
acro
cycle
bird
bottl
bus
CAr
chair
COW
table
dog
hossa
bike
person
pluzt
shoop
5003
trinn
|50
K3.3
73.2
62.6
N1.7
56.6
H1.6
71.9
93
55.4
$1.5
52.7
79.6
77.3
68.4
59.7
81.4
55.3
M0.3
47.8
78.1
77.5
53.5
76.2
71.2
65.8
Training
16.6
9.1
9:1
9.1
9:1
8.3
9.1
22.3
9.1
9.1
13.7
9:1
9.1
9.1
15.4
50.7
14.3
Shmelkaw al /Smeeloww al
2017
604
79.3
69.5
78.4
80.5
45.7
76.3
77.2
50.8
77.5
70.1
76.7
62.7
Fider (1,0 Pebmerr R0G0
64.2
74.7
73.2
55.5
53.7
70.8
$3.9
82.6
51.6
79.7
58.7
78.8
$1.8
77.4
43.1
61.7
693
61
68.6
JasceJeeepee kall 2001
67.3
76.8
60
48.4
58.8
81.1
36.5
75.8
41.5
79.6
56.5
72.8
85.9
$1.7
$3.4
44.8
75.8
68.2
75.7
60.
68.9
OW-DETRRUmD Litiaa 2
2022
70.5
77.2
73.8
51.
55.6
7940
80.8
WIL6
43.2
W0.4
53.5
77.5
89.5
$2.00
74.7
43.3
71.9
66.6
79.
62.0
70.2
ROSETTA Tang ceie
75.3
77.9
03.3
56.2
55.3
79.6
34.6
72.9
49.2
73.7
08.3
71.0
78.9
71.7
80.7
44.3
69.6
68.5
76.1
68.3
69.6
(OL(K etal. 2021
76.1
77.5
09.4
55
56
78.4
84.2
79.2
46.6
79
63.2
78.5
82.7
79.1
79.9
44.1
73.2
66.3
76.4
57.6
70.2
:00 1100
77.4
78.3
73.6
58.2
62.3
77.5
85.2
81.4
51.5
78.4
62.3
81.9
84.2
74.7
71.2
49.7
75.6
67.6
74.3
8788
71.5
Table 3: Performance (%) analysis of incremental object detection based PASCAL VOC Everingham et al. (201)). Here, the IOU threshold is set to 0.75.
10 10 setting
Jaero cycle
bird boat
bottle bus
car
cat
chair
cow
table
dog
horse
bike
person
plant
sheep
sofa
train
P75
at..
2021
36.5
28.4
19.4
47.2
56.7 41.0
19.1
48.0
43.0
36.3
40.0
14.8
4011
36.5
35.3
F0D+
16.0
44.7
31.4
25.8
26.0
62,9
61.8 45.0 22,4
46.6
19.3
33.7
37.1
39.8
39.3
15.8
47.5
34,1
40.0
45.0
38.2
setting
Jacro cycle bird boat
hus
car
cat chair cow
dog
horse bike person plant sheep
Irain
P75
G0 BuoK) ct 2021
40.7
40.9
28.7
19.1
23.8
61.6
56.1 38.8 23.6
47.5
18.7
40.1
40.2
41.5
39.8
9.1
40.6
32.4
41.9
47.6
36.6
Foars
43.8
44.2
34.6
20.1
32.1
56.
62.7
45.4 25.3 49.0
30.0
42.2
50.4
44.5
42.3
11.6
44.2
36.0
48.0
42.3
40.3
19 setting
Jaera cycle
bird boat bottle hus
car
cat chair cow table dog hore bike person plant sheep
sofa
train
P75
at
44.7
22,4
52.0 56.5 38.7
21.6
21.2
37.9
38.7
1722
38.5
F
43.5 46.6
36.1
22.0
32.0 60.3
63.4
45,2 24.4
48.2
30.5
41.9
50.1
41.5
42.2
17.5
43.7
36.8
49.5
47.5
41.2
2021I. Table 2 and 3 separately show the results based on the metric of mAP50 and mAP75. We can see that plugging our method improves the detection performance significantly. Particularly, for the '19+1' setting and the mAP75 metric, employing our method boosts the performance by around 5.2%, which further shows that our method could indeed enhance the discrimination ability.
4.4 ABLATION AND VISUALIZATION ANALYSIS
In this section, we utilize PASCAL VOC as the ID data for training and MS-COCK COCO as the OOD data to per- form an ablation analysis of our method.
Table 4: The performance (%) of only using the IB branch and only using the RIB branch.
Method
FPR95 +
AUROC t
mAP (ID)+
Analysis of IB and RIB. In this paper, we define
vos
51.97
87.55
48.1
an IB branch and RIB branch to separately extract
instance-level features and simulative OOD features.
IB
46.71
89.17
48.9
Here, we make an ablation analysis of our method. Ta- ble 4 shows the performance. Here, 'IB' includes the designed mixture of information bottlenecks and uses
RIB
45.14
89.94
49.1
TIB
41.55
90.36
49.2
vOS Du et al. (2022b) to synthesize virtual outliers for training. We can see that employing the IB branch could improve the detection performance, which shows that using the IB is indeed helpful for compressing object-unrelated contents. And we observe that exploiting the mixture of information
8

---

Under review as a conference paper at ICLR 2023
(a) Input Image
(h) Instance
(c) OOD
Input Image
(e)
Instance
(f) OOD
Figure 4: Visualization of the Instance map and OOD map based on the OOD data (MS-COCO), For each feature map, the channels corresponding to the maximum value are selected for visualization.
bottlenecks reduces FPR95 by around 2.8%, indicating that this module is beneficial for enhancing object-related information and improving the discrimination ability. Furthermore, observe that only using the RIB module could obtain superior performance compared with the baseline method Du al. (2022b). This indicates that reversing the optimization objective of the information bot- tleneck could extract object-unrelated information, which is helpful for obtaining simulative OOD features and improving the discrimination ability.
Analysis of the branch number in MIB. In this pa- per, design a mixture of information bottlenecks (MIB) to purify object-related information from mul- tiple different facets (as shown in Fig. 2}. We make an ablation analysis of the branch number. Here, we only change the branch number and keep other modules un- changed. Table 5 shows the detection results. We can see that the number IB branches indeed affects the performance of OOD detection. When the number of IB branches is small, the MIB module does not obtain better performance. The reason may be that the proposal features involve much information be- longing to different objects. Using fewer IB branches could not sufficiently capture object-related information, which weakens the discrimination. Instead, using IB branches introduces more parameters, which may lead to overfitting and reduces the performance of OOD detection. We observe that for our method, the performance of using branches the best.
Table 5: The performance (%) of using different number of IB branches.
Num (n)
FPR95
AUROC
mAP (ID)
2
43.64
90.02
49.3
4
42.33
90.37
48.8
8
41.55
90.36
49.2
12
|
42.53
90.04
49.0
Visualization analysis. In this paper, we separately extract an Instance map and an OOD map based on the backbone output. In Fig. 4 make a visualization analysis. We can see that the Instance map contains plentiful object-related contents and less object-irrelevnt information, which is in- strumental in improving the localization and recognition performance. Meanwhile, we observe the extracted OOD map is significantly unrelated to the current object, which is beneficial for obtaining simulative OOD features and alleviates the impact of lacking unknown data supervision.
5 CONCLUSION
For unsupervised OOD-OD, this paper proposes method of Two-Stream Information Bottleneck consisting of an IB branch and a RIB branch. Specifically, the IB branch aims to extract object- related information that is helpful for improving localization and recognition performance. Mean- while, the RIB branch to extract simulative OOD features to alleviate the impact lacking un- known data for training by reversing the optimization objective of the information bottleneck. Ex- tensive experimental results on OOD-OD and class incremental object detection, and visualization analysis indicate the superiorities of method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Sean-Chrittophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 14:3438-3450 2021.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. International Conference on Learning Representations, 2017.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
the
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language mod- eling framework for object detection. Proceedings of the International Conference on Learning Representations, 2022.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint /8000013. 2018.
Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't t know from videos in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, 2022a.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by vir- tual outlier synthesis. Proceedings of the International Conference on Learning Representations, 2022b.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: speldworld detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9235-9244, 2022.
Ali Harakeh and Steven L Waslander. Estimating and evaluating regression predictive uncertainty in deep object detectors. arXiv preprint arXiv:2101.00 050066, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 5830-5840, 2021.
10

---

Under review as a conference paper at ICLR 2023
Junho Kim, Byung-Kwan Lee, and Yong Man Ro. Distilling robust and non-robust features in adversarial examples by information bottleneck. Advances in Neural Information Processing Systems, 01-17148-771555 2021.
Joseph Kj, Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubra- manian. Incremental object detection via meta-learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-198) 1981, 2020.
Jungheom Lee, Jooyoung Choi, Jisoo Mok, and Sungroh Yoon. Reducing information bottleneck for weakly supervised semantic segmentation. Advances in Neural Information Processing Systems, 34:27408-27421, 2021.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrate classifiers for detecting out-o--distribution samples. International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018b.
Bo Li. Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao In- variant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7407, 2022.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distributt image detection in neural networks. International Conference on Learning Representations, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy- based out-of distribbuti detec- tion. NewrIPS, 33:21464-21475. 2020.
Can Peng, Kun Zhao, and Brian C Lovell. Faster ilod: Incremental learning for object detectors based on faster renn. Pattern Recognition Letters, 140:109-115, 2020.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal processing, 99:215-249, 2014.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10428-10436, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211-222,20) 2015.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pp. 8491-8501. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. International Conference on Learning Representations, 2020.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and oitoof-distribution detection with likelihood- based generative models. Interna- tional Conference on Learning Representations, 2019.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object de- tectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision, pp. 3400-3409, 2017.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 33:11839-1185, 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore Willke. Out-of-distribttion detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Zifeng Wang, Shao-Lun Huang, Erean E Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. Pac- bayes information bottleneck. International Conference on Learning Representations, 2022.
Binbin Yang. Xinchi Deng, Han Shi, Changlin Li, Gengwei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiaodan Liang. Continual object detection via prototypical task correlation guided gat- ing mechanism. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 9255-9264, 2022.
Jingkang Yang. Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent duttrf-distribuutin detection. In ICCV, pp. 8301-8309, 2021a.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2 2021b.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. 1dd: A diverse driving dataset for heterogeneous multitask learn- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636-2645, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
Yibo Zhou. Rethinking reconstruction autoencoder-baat based out-of-distribution detection. arXiv preprint 22XX:22003.02194 2022.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Here we provide additional analyses, various ablation studics, and more visualization results.
A. FURTHER DISCUSSION ABOUT RIB
In this paper, to alleviate the impact of lacking unknown data for supervision, design a RIB mod- ule to extract simulative OOD features via reversing the optimization objective of the information bottleneck. The reversed objective is shown as follows:
max I(R,, Pood); ); F) (Poood Y). R. RoPd
(12)
Proposition Assume T(Pood; ; Y) = 0, then we achieve the features that contain plentiful out-of- distribution information.
Proof. Note that I I(Pood: = O implies Pood and the labels Y are independent. By minimizing the classification loss, the in-distribution features P., be promoted to be related to the labels Y. Hence, [(Pood; Y) 0 enforces Pood to be irrelevant to Pin. which promotes Pood to contain rich out-of-distribution information.
A.2 TRAINING AND INFERENCE PROCESSES
Algorithm Two-Stream Information Bottleneck for Unsupervised OOD-OD
Input: ID data {_X, Y, B}. randomly initialized detector with parameter 6, weight 3 for the K L- loss. weight for the loss Ldia. weight r for the uncertainty loss Luncertainty.
Output: Object detector with parameter 0*, and OOD detector D. while train do
Sample images from the ID dataset {X, B).
Calculate the Instance map Z and OOD map R using equation and equation
Calculate the enhancing result E using equation Jand equation
Calculate the loss LIR using equation the loss Lais using equation the uncertainty loss
Luncertaiuty using equation 9
Update the parameters 0 based on equation 10
end
while eval do
Calculate the OOD uncertainty score using the left part of equation 11
Perform thresholding comparison using the right part of equation 11
end
A.3 ABLATION ANALYSIS OF HYPER-PARE PARAMETERS
In this paper, we use the hyper-parameter 3 to control the L--loss (equation 5 and use À and 7 to separately control the loss Lass and Cuncertainty (equation [Op. Since Lele, Lioe. and Luncertainty are directly related to the task, 9 and should be set a smaller value than Meanwhile, if B and are set to a very small value, K T-lloss and Ldis may play small role in optimization. Thus, it is meaningful to set proper values for these syper-parametent Next, we utilize PASCAL VOC as the ID data and MS-COCO as the OOD data to perform an ablation analysis of sayeer-paraeetrs. And we only change the value of
and keep other modules unchanged.
Analysis of tlenecks. When 8 separately set to 0.001, 0.0001, and 0.00001, for FPR95, the corresponding performance is 43.84%, 41.55%, and 42.83%.
B. We use 33 to control the K L-divergence loss in information bot-
Analysis of hyper-parameter X. We use to constrain the reversing optimization objective of the information bottleneck. When is separately set to 0.01, 0.001, and 0.0001, for FPR95, the corresponding performance is 44.15%, 41.55%, and 43.26%.
Analysis of hyper-parameter and use the same setting for т. When 7 is separately set to 0.15, 0.1, and 0.05, the corresponding performance is 44.58%, 41.55% and 43.73%.
For the uncertainty loss, follow the work Du et al. (022b)
13

---

Under review as a conference paper at ICLR 2023
A.4 MORE EXPERIMENTAL RESULTS
Results on RegNecX-4.0GE To further verify the effectiveness of our method, we evaluate our method on another backbone network, i.e., RegNetX-4.0GF Radosavovic et al. (2020). Here, we take PASCAL VOC as the IDD training data and MS-COCO as the OOD data for evaluation. Based on FPR95, AUROC, and mAP, the performance of our method is 44.90% 90.08%, and 51.5%, which significantly outperforms the VOS's performance, i.e., 50.81 %. 88.42%, and 50.8%. This shows that our method of Two-Stream Information Bottleneck is able to strengthen the instance- related information and extract proper simulative OOD features, which reduces the impact of lacking unknown data for supervision and improves the discrimination ability.
Further Analysis about RIB. For RIB (sec equation 7. since the OOD map R is extracted based on the input feature map F, R could be considered be related to F. Hence, we do not mutual information constraint to enhance the dependence between R and F. Here, we make an ablation analysis of adding the mutual information constraint. We take PASCAL VOC the ID training data and MS-COCO as the OOD data for evaluation. We observe that adding the mutual information constraint increases the performance of FPR95 by around 1.1% Meanwhile, we replace the variational operation with the traditional convolution operation. The performance of FPR95 is increased by around 4.8%. These analyses show that using the variational operation helpful for capturing distribution-relace information, which is better to extract cut-of-distribution information.
A.5 MORE VISUALIZATION EXAMPLES
In Fig. we give more visualization examples. We can see that the extracted Instance map con- tains plentiful object-relevant information, which helpful for localizing and recognizing objects accurately. Meanwhile, we can also see that the calculated OOD map is significantly different from the Instance map. which conforms to the meaning of the OOD, i.e., the OOD data deviates from the in-distribution. This further shows that our proposed reverse information bottleneck could indeed extract simulative OOD features, which is beneficial for alleviating the impact of lacking unknown data for supervision and improving the discrimination ability.
In Fig. and and recognizes ID objects and OOD objects. Particularly, compared with ID objects that contain fixed number of categories, the category distribution of OOD objects is diverse, presenting signif- icant challenge for the object detector. Our method attempts solve this problem from the feature perspective, which has been demonstrated to be effective.
7 we show more detection results. We can see that our method accurately localizes
A.6 COMPUTATION OF MUTUAL INFORMATION ON 3D FEATURE MAPS
In this paper. we compute K L-divergence loss to approximate the mutual information. The pro- cesses are shown as follows:
p(x) g(x)
KL(p)14) Σ p(x) log
(13)
where p(-) and q(-) represent two probability distributions. For example, given two 3D feature maps H xx and E Rwxxxe, we first perform softmax operation on and C. Then, we separately take the corresponding elements between the two processed results the input of equation 13 calculate the K L. result. The mean of the KL results from all corresponding elements taken the output K Lddveegencce loss.
A.7 PERFORMANCE ON HIGHER LEVELS OF CONTAMINATION FROM OOD CLASSES
We select 8K images from COCO and synthesize some OOD objects these images to perform further evaluation. In Fig. we show some synthesized images. Through experiments, we observe that method still outperforms VOS Du al. (2022b) significantly. Particularly, compared with vOS Du et al.] (2022b). our method reduces FPR95 by around 10. 10.1% and improves AUROC by around 3.2%, which further demonstrates the effectiveness of our method.
14

---

Under review as a conference paper at ICLR 2023
(a) Input Image
(b) Instance
(c) OOD
Input Image
(c) Instance
(f) OOD
Figure 5: Visualization of the Instance map and OOD map. For each feature map, the channels corresponding to the maximum value are selected for visualization. We can see that the Instance map contains plentiful object-relevant information. Meanwhile, the OOD map involves sufficient object-irrelevant information, which is helpful for extracting simulative OOD features to alleviate the impact of lacking unknown data for supervision.
15

---

Under review as a conference paper at ICLR 2023
DOLF
OOD
00D
OOD
OOL
0OD
000
ODD
DOD
00D
0OD
OOD
O0D
000
OOD
OOD4
OOD
OOD
OOL
00C
OOO
OOD
OOD
OOD
OOD
OOD
Figure 6: OOD detection examples based on our method. Here, we use BDD-100k as the in- distribution data and MS-COCO as the OOD data. We can see that our method accurately distin- guishes OOD objects, which shows the effectiveness of our method.
16

---

Under review as a conference paper at ICLR 2023
100%
person 100%
sheep 100%
100%
bicycle 100%
Lite
horse 1person 99%
bus 100%
person 67%
bicycle 100%
motorerson YLIE
person 100
person 100%
bus 100%
motorcycle 100%
horse 100%
sheep 100%
sheep 10
94%
car 100%
dog 100%
bird 100%
bird 100%
bird 100%
couch 100%
person 100%
person YLL
person 100
train 100%
bicycle 100%
motorcycle 100%
100%
HITH
bicycle
bird
person 100cc
COW 100% 100%
boat 100%
motorcycle
person 100%
person 99%
bird 100%
horse 100%
bicycle 100% TE
dog 100%
tv 100%
boat 100%
IV
dog 100%
bicycle 100%
boat
100%
Doa
Figure 7: Detection results based on PASCAL VOC. We that method accurately localizes and recognizes objects in these images, e.g.. the dog, train, cow, and person, which shows that our method is effective for in-distribution data.
17

---

Under review as a conference paper at ICLR 2023
Figure 8: To further evaluate our method, we synthesize some images that contain OOD classes, e.g., the Dinosaur, Panda, and Camel. Mcanwhile, also collect some images from real scenarios.
Table 6: Definitions of notations used in our method.
Notations
Definition
F
The feature map extracted by the backbone network.
The encoded Instance map.
Pm
The ID object features extracted based on Z.
Q
The oulput encoding result of the i-th IB branch for information enhancement.
G
The calculated gating weight of the i-th IB branch.
A
The aggregaied results.
The output enhanced results.
R
The encoded OOD map.
Pood
The OOD object features extracted based R.
A.8 MORE ABLATION STUDIES ABOUT EQUATION
In equation A is to aggregate the results of each IB branch. By this operation, A contains plen- tiful object-related information. The learned sigmoid weight to balance A and P., during the enhancing process.
We make an ablation analysis about A and a. Firstly, we replace the gating operation (as shown in the left part of equation 4 with the simple mean operation and keep other modules unchanged.
We observe that compared with our method, the mean operation increases FPR95 by around 2.7%, which shows the effectiveness of the gating operation. Next, replace the learned sigmoid weight with a manually set value and keep other modules unchanged. We set multiple different values and observe that 0.6 corresponds to the best performance. However, compared with method, this
18

---

Under review as a conference paper at ICLR 2023
operation increases FPR95 by around 1.3%, which indicates that using the learned weight is much better for balancing A and Pm-
A.9
DEFINITIONS OF NOTATIONS
Table 6 gives the definitions of notations used in our method.
A.10 LIMITATION
To promote the safe deployment of object detectors, propose use information bottlenecks to strengthen object-related information and generate simulative OOD features to alleviate the impact of lacking OOD data for training. Since we only verify our method on two benchmarks, do not clear about the performance of the proposed method in practical scenes, which may be the limitation of our proposed method.
19


Reviews:
[{"review_id": "uqCaA50A5Fd", "reviewer": "Reviewer_UuoW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper proposed a new information theoretic approach to tackle out-of-distribution object detection (OOD-OD). \n- Authors leverage Variational information bottleneck (IB) to suppress object-irrelevant representations to improve discriminability of in-distribution class (ID). \n- Next, to simulate OOD features to train an OOD detector in an unsupervised fashion, authors reverse the IB equation, called Reverse Information Bottleneck (RIB). In contrast to IB, which aims to extract task-relevant compact codewords/features, the goal of RIB is to simulate task-irrelevant OOD style features to train a strong OOD detector. \n- Extensive experiments on standard OOD-OD datasets and incremental object detection datasets show the effectiveness of the proposed approach for OOD-OD.", "summary_of_the_review": "- Overall I'm very happy with the paper and its writing quality. \n- Authors leverage Information bottleneck to simultaneously enhance object relatedness and suppress (with reversal) object irrelevance in features to improve OOD-OD. \n- The proposed method is simple and very effective. The experiments support all the claims made by the authors. A few more explanations to understand the working of Reverse IB are required which I believe the authors can provide in reasonable time. \n- I'm willing to improve my ratings depending on the author's response.", "strength_and_weaknesses": "Strengths\n- The proposed method is very simple and effective and is one of the major strengths of this work.\n- The use of information theoretic methods, the IB, for object detection is new and fresh.\n- All the claims made in the paper have been properly addressed with adequate experiments.\n- The paper tackles a very important practical problem with object detectors and achieves impressive performance on standard benchmarks.\n\nWeaknesses\n- I believe the comparisons in Table-1 are unfair as the current method introduces more trainable parameters than VOS. Not to diminish the impressive performance, I believe it is imperative to add a column with backbones and trainable parameters introduced for a more fair comparison.\n- Authors claim that the OOD map simulates outlier samples and the visualizations of the map doesn't seem to do that. After careful observation one can notice that the maximum of the OOD map still occurs either inside or around the object of interest. Additionally, these maps are not interpretable, it is not clear to me why these maps should look like noise. Can these maps be interpreted as adversarial noise (on the features) since they are obtained by maximizing the discrepancy between logits of the right class? (Like noise that can be added to make the feature lie closer to a different ID class different than the one in the feature map).\n- In the standard OOD-OD setup the number of OOD samples coming from COCO/Open-Images are usually less than 20% (930 OD samples for COCO with 4952 samples of ID val set). It is not clear how the proposed model does with higher levels of contamination from OOD classes. \n- The contribution of the first stage of IB is not clear from the ablation experiments. What happens if one were to simply pass F through two different convolutional blocks and keep everything the same? In Table 4, authors either keep the full branch or remove it entirely, but why do one need the first IB/RIB stage? One can apply the same L_dis, L_uncertainty losses on this architecture even without computing mean and standard deviation and combining them. I understand that the loss becomes uninterpretable but how does it fare against the proposed approach?\n\nMinor questions/comments to improve the paper (Not dependent on the final decision):\n- It is not clear how Eq. 5 is related to Variational Information Bottleneck. From the paper there should be a negated term and the second term is the KL divergence. In Eq. 5 there are two KL divergence terms and it is not clear how the authors end up with this equation. This step requires some more information to help the readers understand the math.\n- How is Eq. 8 related to Eq. 7. Do the authors mean to say that Eq. 8 approximates Eq. 7. In that case why is it hard to compute Eq. 7 directly without any approximation?\n- It would be beneficial to researchers implementing this work, if information about computing mutual information (I(.,.)) on 3d feature maps is provided in the supplementary. \n- Is the purpose of experiments on Incremental object detection just to show that IB can improve overall object detection performance?. If uncertainty loss is not computed for these experiments, then why would one need to perform RIB?\n- Kindly add the baseline to Table 4, readers would have to move back and forth to understand what the baseline is. \n- Line before \"Metrics\" in Section 4.1 \".... manually examined the OOD images to ensure ..\" -> \".... manually examined to ensure ... \".\n- What is the confidence threshold used to generate the results in Fig. 3. If its the same value for VOS and TIB?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1I0knkof-Fe", "reviewer": "Reviewer_D2nY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to solve OOD problem in object detection.\nIt tries to reduce the impact of lacking unknown data for supervision and leverage in-distribution data to improve the model’s discrimination ability with both Information Bottleneck and Reverse Information Bottleneck.\nIt first uses a standard IB network to disentangle instance representations that are beneficial for localizing and recognizing objects, and then use a RIB to obtain simulative OOD features to alleviate the impact of lacking unknown data.\nExperiments demonstrate good performance in incremental object detection. ", "summary_of_the_review": "Overall, this is a solid paper with detailed experiments and visualizations. The major concern is the scalability in really-world data. I will increase score if the author can provide my evidence. ", "strength_and_weaknesses": "Strength\n+ The motivation is clear, and the solution is novel.\n+ Good performance gain on incremental object detection setup.\n\nWeakness\n- The major concern is the scalability of such method. The author only uses PascalVOC and BDD as in-domain data, which is relatively small and simple. To verify its ability of handling real-world data, I would suggest the author can demonstrate a more complex setup, for example, use COCO as ID and Openimages as evaluation set.\n- Figure 2 is not very illustrative. It would be better to align equations into the flow. Or teardown to multiple small figures for better clarity.\n- In Figure 4, the visualization of OOD is confusing. It is more like white noise. How does simulate OOD feature extracted from it help?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-h0yB5JAINS", "reviewer": "Reviewer_ajPF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of out-of-distribution object detection (OOD-OD), where an object detector has to detect unknown objects without relying on an auxiliary dataset. The authors propose a framework of Two-Stream Information Bottleneck (TIB), with a standard IB network to extract object-related information from in-distribution (ID) data, and a Reverse Information Bottleneck (RIB) to obtain OOD features. By exploiting IB, the method manages to obtain enhanced object-related information from ID data. At the same time, TIB obtains simulative OOD features by maximizing the discrepancy with the ID features. Experimental results show the method could significantly improve the performance of OOD-OD and incremental object detection over the baselines.", "summary_of_the_review": "The idea in the paper is well-motivated and described. The paper addresses the problem of out-of-distribution object detection by exploiting Information Bottleneck, which has been carefully studied by prior work and shown promising results. Therefore, the idea has solid theoretical reasoning. However, when implementing the IB in terms of loss functions, not all properties are guaranteed. Also, lacking of a visual explanation of the OOD map with OOD instances significantly reduces the convincingness of the proposed approach. \n", "strength_and_weaknesses": "**Strengths**\n+ The idea of using IB to extract object-related and OOD feature problems is novel and interesting. It seems that the idea could be applied to other object detection problems. The empirical results are encouraging.\n+ The paper is well-written and easy to follow. The key design choices in the paper are carefully described.\n\n**Weaknesses**\n+ The task of OOD-OD is not formally described in the paper.\n+ Does background differ from OOD instances?\n+ Is the OOD bounding box included when performing inference?\n+ The correlation between Information Bottleneck and the method is not clearly illustrated. Particularly, the KL terms in Eq. (5) do not express the IB where Z should have low mutual information with F but ignore the high mutual information with the label. So do the Q and P_in. Eq. (8) also has that problem. Also, we usually minimize a loss (with a non-negative value) not maximize a loss as in Eq. (8).\n+ The enhancing process (Eq. 4) appears to be unclear. Both A and α are related to P_in, and the role of each term is not well described. An ablation study about this process may help to clarify.\n+ Regarding the datasets, how many classes are in the ID and OOD data?\n+ Images in Figure 4 only contain objects from ID classes, which makes the visualization of OOD maps uninformative. What do the OOD maps look like with objects from  OOD classes, and in the cases of mixed ID and OOD classes? It would definitely demonstrate the effectiveness of the proposed approach on OOD-OD.\n+ There is no discussion about the choice of hyper-parameters in Section A.3. The paper should include an explanation for each set of hyper-parameters.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject