Dual Representation Learning for Out-of-distribution Detection
Authors: Zhilin Zhao
Longbing Cao
Venue: Accepted by TMLR ()

Abstract: To classify in-distribution samples, deep neural networks explore strongly label-related information and discard weakly label-related information according to the information bottleneck. Out-of-distribution samples drawn from distributions differing from that of in-distribution samples could be assigned with unexpected high-confidence predictions because they could obtain minimum strongly label-related information. To distinguish in- and out-of-distribution samples, Dual Representation Learning (DRL) makes out-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network exploring strongly label-related information to learn label-discriminative representations, DRL trains its auxiliary network exploring the remaining weakly label-related information to learn distribution-discriminative representations. Specifically, for a label-discriminative representation, DRL constructs its complementary distribution-discriminative representation by integrating diverse representations less similar to the label-discriminative representation. Accordingly, DRL combines label- and distribution-discriminative representations to detect out-of-distribution samples.  Experiments show that DRL outperforms the state-of-the-art methods for out-of-distribution detection.

Published in Transactions on Machine Learning Research (08/2023)
Dual Representation Learning for Out-of-distribution Detection
Zhilin Zhao
zhaozh17@hotmail.cm com
Data Science Lab, Macquarie University, Sydney, NSW 2109, Australia
Longbing Cao
longbing. castongg--oooooooo Qan..aa.......m edut
Data Science Lab, Macquaric University, Sydney, NSW 2109, Australia
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- PHAr3g49h6
Abstract
To classify in-distribution samples, deep neural networks explore strongly label-related formation and discard weakly label-related information according the information bot- tleneck. Out-of-distributtin samples drawn from distributions differing from that of in- distribution samples could be assigned with unexpected high-confidence predictions be- cause they could obtain minimum strongly label-related information. To distinguish in- and out-of-distribution samples, Dual Representation Learning (DRL) makes oul-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network explor- ing strongly label-related information learn label-discriminative representations, DRL trains its auxiliary network exploring the remaining weakly label-related information to learn distributioon-diierimnnat resentation, DRL constructs its complementary Distribution-disernnnnatt by integrating diverse representations less similar to the label-discriminativ representation Accordingly, DRL combines label- and iistribation-ddiiininatiee representations to detect out-of-distribution samples. Experiments show that DRL outperforms the state-of-the-art methods for out-of-distrriie of-distribution detection.
representations. Specifically, for label-discrininative rep- representation
1 Introduction
Deep neural networks demonst.ate signíficant classification generalization ability Krizhevsky et al. (2012) on samples drawn from an unknown distribution (i.e., in-distribution, ID), benefiting from their powerful ability to learn representations Bengio et al. (2013): Gong et al. (2022). From the information-thcoretiee view- point Saxe al. (2018) for an ID sample, pretrained network effectively learns the sbeeldiisrrmmintiiee representation by exploring strongly label-related information and discarding weakly label-related informa- tion. However, test samples could be drawn from distributions different from that of ID samples Ren et al.) (2019) ; Haas et al. (2023) (i.e., out-of-disst distribbiiunnnn, OOD), and the pretrained network not sensitive to such OOD samples and make unexpected high-confidence predictions on them. Tt causes that the pretrained network cannot distinguish ID and OOD samples. This over-confidence phenomenon on OOD samples makes networks vulnerable to atlack Goodfellow et al. (2015) and causes distributional vulnerability Zhao et al. (2022).
We arguc some fundamental causes of the above problem in pretrained network include: (1) the label- discriminative representations from strongly label-related information focus on capturing the ID input-label mapping, while (2) the network overlooks the learning of the weakly label-related information that can distinguish ID and OOD. This argument can be explained by the information bottleneck principle Alemi et al. (2017): Wang al. (2022) of learning tradeoff between input compression and its label prediction. As result, the pretrained network only mainly focuses
representations from

---

Published in Transactions on Machine Learning Research (08/2023)
Confidence 1.00.
Confidence 0.98.
DoI
Day
Bap
ID
0OD
Figure 1: Different informativeness properties of ID and OOD samples. Red and yellow boxes capture the information for learning label- and distribution-dicciinnnniii representation, respectively. The label- and iestrbbttion-disrimnnati representations of the ID sample correspond to the same label (dog). Conversely, the label- and distribution-discrimnnnatt
representations of the OOD sample correspond to different labels (dog and bag). However, the OOD sample is assigned with high-confidence prediction for the wrong label (dog) since it obtains minimum strongly label-related information about the label. Tt causes networks cannot distinguish between ID and OOD samples.
on strongly label-related information but discards weakly label-related information. Strongly and weakly label-related information are complementary, and their correspouding label- and distributionndiccciinnniw representations contain labeling information for ID sample. An ID sample has both strongly and weakly label-related information, and an OOD sample with minimum strongly label-related information still receives nigh-confidence prediction from the pretrained network. For example, if a pretrained network merely focuses on head (strongly label-related information) and ignores body (weakly label-related information) to recognize dog images, an OOD sample with minimum information about 'head' could have a high-confidence prediction about the dog class, as shown in Figure [
Accordingly, both the label- and distribution-lieeimnnnaie representations of an ID sample correspond to the same label, and the cnnrrbaatoon-issrimiaatiii representation of an OOD corresponds to another label even none of any labels. Therefore, the label-discriminative representations are sufficient for the classifi- cation task but insufficient for OOD detection, and the labeling consistency between label- and distribution- discriminative representations could distinguish ID and OOD samples. This inspiration from the different informativeness properties of ID and OOD samples can be summarized as:
By dually learning label- and iettrbutiionddisermmnnttt representations from strongly and weakly label-related information, respectively, a new perspective is to couple both representa- lions to detect OOD samples.
Several efforts have been made in the literature to improve the OOD sensitivity of a pretrained network when training OOD samples are unavailable Hendrycks & Gimpel 2017 Sastry & Oore (2020
bs Lee et al.
(2018a): ; Shalev et al. 20188. including post-hoc and confidence enhancement methods. In post-hoc methods, a pretrained net work is solely trained on ID samples without considering the predictions for OOD samples, an OOD detector then trained for the pretrained network. The OOD detector does not acquire new knowledge from training ID samples, making the OOD detection performance heavily dependent on the knowledge about label-discriminative representations learned by the pretrained network Hendrycks et al. (0119a) To address
this problem, the confidence enhancement methods improve the OOD sensitivity of pretrained network by retraining finetuning it with prior knowledge about OOD samples Lee et al.
(2018a); Hendrycks ct al. (2019b): Tack et al. (2000); Hsu et al. (2020). They regulate the output distribution of the network to encourage high- and low-confidence predictions on ID and OOD samples, respectively Malinin & Gaaees (2018). This approach enforces the network to extract more information about ID samples through retraining, while it relies heavily on prior knowledge about OOD samples and may not be applicable to unknown OOD samples. As result, the existing post-hoc and confidence enhancement methods suffer from limited OOD detection performance. To address the limitations of these two methods, learn OOD-sensitive representations (i.e.,
2

---

Published in Transactions on Machine Learning Research (08/2023)
distributoon-diccimnnnntt representations) from the weakly label-related information of ID samples and distinguish ID and OOD samples according to their different informativeness properties.
By exploring the different informativeness properties of ID and OOD samples, we propose a dual represen- tation learning (DRL) approach to learn both label- and strongly and weakly label-related information, respectively. For the generality and flexibility of DRL, we assume a pretrained network is given which only focuses on learning the aabel-disrimmnnative representa- tions to classify ID samples. We introduce auziliary network to learn the complementary distribution- discriminative representations corresponding to the label-discriminative representations. Therefore, the pre- trained network is trained solely on ID samples, while the auxiliary network is trained on both ID samples and the corresponding label-discimminative representations from the pretrained one. The pretrained and aux- iliary networks aim to extract label-related information from inputs to learn representations. Specifically, the label- and distributionddieeinnnati representations learned from the two networks are strongly and weakly related to labeling, respoctively. To ensure the above properties of the two representations, given an ID sam- ple, we set a constraint so that its label-discrmminative representation and sensitive to the same label. An implicit constraint is incorporated into the auxiliary network, which integrates multiple reprosentations different from label-discrimnnative representation into a complementary distributionnddicrimnnattt representation. After learning the auxiliary network depending on the pretrained network, DRL then couples the two representations to calculate an OOD score for each test sample. Accordingly, the OOD scores are then used to distinguish ID and OOD samples for OOD detection.
representations from
representation is significantly different from its
The main contributions of this work include:
Taking the information bottleneck principle, we reveal that learning a label-diseriminaiive represen- tation by a pretrained network alone may not sufficiently capture all labeling information of an ID sample. There may exist a complementary distribution-diecimmnaiii representation capturing the remaining labeling information.
We infer the information bottleneck principle to learn the complementary distribution-ddccriinaatt representations. Accordingly, we train an auxiliary network to learn representation by integrating multiple representations different from a label-discrininatie represen- tation.
By exploring the different informativeness properties of ID and 00D samples, the label and instribbttoon-discimmnaiiii representations are coupled to form OOD scores for distinguishing ID and OOD samples.
The paper is organized as follows. Section 2 reviews the related work. Section mothod. The experimental results are shown in Section and Section makes some concluding remarks.
g presents the proposed DRL
2 Related Work
The test samples for a DNN could be ID or OOD, and the task of OOD detection Ren et al. 22019 is to differentiate ID and OOD samples. Unlike outlier detection Zong et al. 2018); Schmier et al. (2023) which filters out training samples deviating from the majority through training a detector, OOD detection detects samples different from training ID samples in the test phase for trained networks. In contrast to the open-set recognition Zhang et al. (2020) whose training dataset contains samples from known and unknown classes and which refuses samples from unknown classes in the test phase, OOD detection only accesses a training ID datasot with the corresponding pretrained network Icarned from this datasct. For a pretrained network, the existing methods of improving the OOD sensitivity include post-hoc and confidence enhancement methods. Post-hoc methods train an OOD detector measuring data relationships sensitive to OOD samples according to the network outputs from a pretrained network, while confidence enhancement methods retrain pretrained network to improve the OOD sensitivity with prior knowledge about OOD samples.
3

---

Published in Transactions on Machine Learning Research (08/2023)
2.1 Post-hoc Methods
A baseline method in Hendryeks & Gimpel (017)) treats the confidence represented by maximum softmax outputs as the OOD score for a test sample, where ID and OOD samples are expected to own high and low scores, respectively. An out-of-Distrbution detector for Neural networks (ODIN) Liang et al. (2018) enhances the baseline method by considering the adversarial perturbation on inputs Miyato et al. (2019) and temperature scaling on the softmax function, and the basic assumption is that the two operations can enlarge the confidence between the ID and OOD samples. Energy-basse based Detoctor Liu et al. (020)) (ED) applics the
negative energy function in terms of the denominator of the soflmax activation for OOD detection, and the log of the confidence in the baseline method is particular case of the negative energy function. Apart from
utilizing the network outputs from the last layer, Mahalanobis Distance-based Detector Lee et al. (2018b)
(MDD) adds a small noise perturbation to input and combines its Mahalanobis distances of latent features from different network layers as the OOD score. Rectified Activations (RA) Sun et al. (2021) truncates
activations on the penultimate layer of a network to reduce the negative effect of noise. Based on MDD,
Gaussian mixture based Energy Measurement MMrteza & Li (2022) is scoring function that is proportional to the log-likel of the ID samples and extended to deep neural networks by applying the features extracted from the penultimate layers to calculate scores. However, these detectors are merely trained from the outputs of the network without learning any new knowledge from training ID samples, which indicates that the OOD detection performance heavily depends on the knowledge learned by the pretrained network.
2.2 Confidence Enhancement Methods
To explore more information from ID) samples to enhance the OOD sensitivity of a pretrained network, another approach
retrain the pretrained network with prior knowledge about OOD samples. One special idea is to generate OOD samples from training ID samples and apply them to restrict network behaviors on OOD samples. For example, the Joint Confidence Loss (JCT.) Lee et al. (2018a) utilizes a generative adversarial network Goodfellow ct al. (2014) to generate samples around the boundary of training ID samples and encourage their label brobabilities to satisfy a uniform distribution. Instead of generating OOD samples, Superviue Learning (SSL) Hendrycks et al. (2019b) augments an ID sample by rotating it 90" 180%, 270", respectively, and learns the rotation angles and the labels of augmented ID samples simultancously. Applying the same augmentation method, the Contrasting Shifted Instances (CSI) Tack et al. (2200) treats the original and augmented IT) samples as positive and negative samples in a contrastivve loss, respectively. Although these confidence enhancement methods introduce extra knowledge to encourage networks to learn more information sensitive to OOD samples from ID samples, the prior knowledge about OOD samples could be misleading due to the complexily of unknown test samples. From a model perspective, DeConf-C et al. (2020) believes the softmax function causes a spiky distribution over classes. It addresses this problem by using a divisor structure that decomposes confidence. DeConf-C also applies augmented ID samples that are perturbed by gradient directions. However, it is hard to select an appropriate specific divisor structure for a training ID dataset due to the lack of knowledge about data characteristies. Based the experimental observation that the OOD detection performance declines as the number of ID classes increases, MOS Huang& & Li (2021) groups training ID samples according to label concepts. However, the laxonomy of the label space is usually unavailable, and applying K-Means clustering on feature representations and random grouping to divide the ID datasct cannot ensure that the samples in a group have similar concepts. Based on the assumption that OOD samples are relatively far away from the ID samples, KNN+ Sun et al.. (202)) computes the k-th nearest neighbor distance between the embedding of each test image and the training set as the OOD score. Compactness and Dispersion Regularized learning Ming al. (2023) makes trade off between promoting large angular distances among different class prototypes and encouraging samples to be close to their class prototypes.
3 Dual Representation Learning
In this section, we presont the proposod DRL method. We assume cach ID sample (x, y) follows the joint dis- tribution P) where x is an input and y is its corresponding ground truth label. The number of classes is K, and thus have y E [K]. We assume pretrained network ge with the parameter transforms an
4

---

Published in Transactions on Machine Learning Research (08/2023)
Lose-nnstiiinatte Represertation
Distribution disriminzeee Represpetation
9
fo
Input
Pretrained Network
Aualiaary Network
(i]
distio-dittibutionn Score
Figure 2: The DRL learning process. It includes: (i) learning pretrained network training ID samples for label-discrmminative represenlations; (ii) learning an auxiliary network on training TD samples and their corresponding label-diseriminative representations for distribution-discriiinn representations; and (iii) calculating the OOD scores by combining these two representations.
input X E X into a labei-discriminative representation (x(x) d E D. We assume an auxiliary network fo with the parameter / transforms this sample with its label-diseriminative reprosentation d into the com- plementary distributiooddiiriminatee representation fa(x, d) = c E Both the pretrained and auxiliary networks own the same network structure. For each input x, we combine the two representations calcu- late an OOD score S(x). Different from traditional ensemble methods Guyon al. (017 that combine the outputs from different independent networks, DRL integrates two complementary representations from two dependent networks. Specifically, the auxiliary network learning complementary representations depends on the pretrained network learning label-discriminative representations, as shown by the learning process in Figure 2
Recall that a pretrained network merely learns label-diseriminative representations from strongly label- related information, which causes that this pretrained network assigns unexpected high-confidence predic- tions OOD samples. Therefore, DRL applies an auxiliary network to learn complementary distribution- discriminative representations from weakly label-related information to improve the OOD sensitivity of the pretrained network. In the following, we firstly reveal the existence of complementary information which is weakly related to labeling. We then describe the information bottleneck principle of extracting weakly label-related information. Based on the information bottleneck principle, we propose the learning method of the auxiliary network. Accordingly, we combine both label- and distributionnddscciiinnatt representations calculate the OOD scores for test samples.
3.1 Learning Principle of Label-Discriminative Representations
The information bottleneck principle Alemi et al. (017)) makes tradeoff between the compression of input and the prediction of its label, and the mutual information Belghazi et al. (2018) Gonzalez-Lopez et al. (2020) measures the shared information between variables. Accordingly, for a network, the process of extracting label-related information from inputs to learn the corresponding representations can be interpreted from the information-theoretiicc view [aax et al. (2018).
Given a dataset X' and its label set V, ID sample contains all the information about its corresponding label y. The information bottleneck limits the information to predict y by compressing to learn its label- discriminative representation d. Therefore, the information of the label-discriminative representations D shares with the labels y' should be maximized while the information between D and the inputs X' should be minimized, i.e.,
max MAxI(DD)) -
(1)
where I(::-) refers to the mutual information, and 3p controls the trade-off between learning more informa- tion from the labels y and retaining less information of the original inputs X for learning label-discriminative
5

---

Published in Transactions on Machine Learning Research (08/2023)
representations D. Specifically, I(D:)) determines how much label information is accessible from the label- discriminative representation, and [(X; D) denotes how much information the label-discriminatiee represen- tation can acquire from the original input.
Note that x contains all the information of because is a representation learned from x. Therefore, we have
(
(2)
where I(Z, (D)) denotes the shared information between the labels y and the union of X and D. According to the chain rule Federici al. (2020) of the mutual information, we have,
I(X:))- I(D:)) + I(X))
(3)
where M(D), representing the shared information between X' and y given the label-iiscr reprosentations D, greater than or equals Accordingly, we have I(X;))) 2 I(D: Thercfore, for an ID input x, its label-diseriminative representation d is insufficient for y because d does obtain all the labeling information about y.
Accordingly, we assume there exists another representation for an ID sample x different from the label- discriminative representation and containing the remaining information about y, i.e., a complementary representation c. The distribution dinirbbtioo-disciinnatiee representations also satisfies Eq. (2) and Eq. (3) since c also a representation the input x. We can thus obtain the following equation by simply replacing in Eq. (3] with C.
= ([C)))
(4)
Since insufficient for sclocting y and these no shared information between and d, we thus assume M(D:) I(C:)) and have the following equation according to Eq. and Eq.
I(X:)) = I(D:)) + =I:
(5)
According to Eq. (), both the label- and cnstributiion-discrmmmna representations about the same label co- exist for an ID) sample, and a label-discriminative representation alone cannot contain all the label information of an ID sample. Specifically, for ID samples, its label-discriminative representation is strongly related to labeling, while its distribution-dicciiinaaiiee representation learned from the remaining label information is weakly related to labeling. Recall that an OOD sample with high- discriminative representation that is sensitive to a label and a high-confidence prediction owns a label- representation that corresponds to other labels or even none of any labels. Accordingly, we can distinguish ID and OOD samples according to different informativeness properties by exploiting the dabel-discimmnntivee representations and exploring the distrrbutioondiiciiinnateee representations.
3.2 Learning Principle of Distribution- Disiiiiiioniiseriiinaaaiiie Representations
We present the information bottleneck principle of learning discriutiionddieriinnateee representations. Re- call that, for an ID sample x, the pretrained network gø learns its label-discriminative representation d to predict the label y based on the information bottleneck principle in Eq. ( Furthermore, the correspond- ing distrbbution-diicininntevee representation e also contains the information about the label because d is insufficient to contain all label information, which indicates that C also follows the learning principle of Eq. ① Further according to Eq. (5) there no shared label information between the two kinds of representations, ie., the mutual information I(D, C) is expected to equal to zero. However, in practice, it is impossible to separate the information of the two representations entirely. Basod on Eq. 11, we thus minimize the amount of shared information T(D,C) and have
max -cI(x()) aI
(6)
Similar to 3D. Be controls the amount of information propagated from X to C. A larger de will lead to more information being extracted from X, which also indicates more label-unrelatd information will be excted
6

---

Published in Transactions on Machine Learning Research (08/2023)
d
Label-discimmnative Representation
Component Represettation
101
(M )
Distibutioo-iocrinnnaaiive Representation
Weighted Sum
Constructing distribution-dierimiaattee representation. A thicker black line indicates a larger weight or vice versa. A tations where a component representation less similar to the label-diseriminative representation given a higher weight.
Figure 3:
representation consists of multiple component represen-
to reduce the overlap information between C and y. Also, is difference coefficient controlling the trade-off between extracting label-related information from the original inputs and enlarging the difference between the label- and distribttion-iieiiiinaiie representations. A larger a will lead to less overlap information between and C but less label information to be extracted, or vice versa.
Both Eq. and Eq. (6] are learning principles for learning representations by finding a trade-off between learning label-related information and retaining less information of the original inputs. They are significantly different. Recall that Eq. (L) forces network to explore the strongly label-related information by compression for learning the label-diserimnnatie representations. Compared with Eq. (L. Eq. (6) considers a constraint I(D;C) which forces a network to explore the label-related information discarded by -ibei-diser representations and use them to learn the distribation-dierrnnnaaie representations. Therefore, based on the two learning principles, the label-related information of label- and distiibttiorddicciinnaavee representations is different and complementary.
To find restriction for learning distribution-disrimmnaati reprosentations according to I(D:C), we quantify this mutual information term and have
I(D:C) = EP(D) KL (PCCI))))PP()))
(7)
where P(D), P(C) and P(CID) denote the respective probability distributions, and KL(-I-) represents the Kullback-Leibler (KT.) divergence. However, measuring Eq. 7 is intractable because we cannot obtain an analytic expression for P(C). According to the variational inference Gal & Ghahramani (2016). can solve
this problem by applying a tractable proposal distribution Bishop (2006) Q(C) to approximate P(C). We thus have
I(D:C) EP(D) KL (P(CI)))I) - KL (Q(C))IP(C) S Ep(D) [KL (P(CI))II((E))
(8)
where the inequality is due to the nonnegative property of the KL divergence. According to Eq. g we know that the distebbutioodiisciniaatvee representations depend on the corresponding label-discrinnnatie representations, and the distribution of to the proposal distribution Q(C). However, we have to decide Q(C) according to prior knowledge which is expected be similar to the unknown P(C). Therefore, difficult to construct an explicit constraint, e.g., representation lcarning due to the unknown proposal distribu-
representations PCC(D) should be close
regularizer, for the distributtion-diccimnntttt tion Q(C) in Eq. (8). The unknown proposal distribution drives explore an implicit constraint to ensure representation is complementary to its dibeliiicrimiitiive representation, representation contains weakly label-related information discarded by the
that i.c., label-discrmiinative representation.
The three sources of mutual information I(C:)), C(X;C) and I(D;C) in Eq. 6 can be modeled by a loss function, a network, and a constraint, respectively. Both the loss function and the network are not explicitly inferred from their mutual information terms. Specifically, the loss function ensures that the distribution- discriminative representations contain label-related information, and the network compresses the information
7

---

Published in Transactions on Machine Learning Research (08/2023)
from inputs to learn their representations due to the down-sampling nature. Accordingly, based on I(D;C), we implicitly restrict that distribution-disrimnnaal label-dscriminative representation in the learning process. This implicit constraint forces the network to explore weakly label-related information since the strongly label-related information has been explored by label-discriminative representations D. Without this constraint, networks will explore the same strongly label-related information DD from inputs to learn representations according to max I(C;)) - BcI(X;;) C). Because I), C(x;C) and ((D;C) in Eq. (6) are implicitly modeled, it unnecessary explicitly set the prper-parameters Se and a in Eq. 88)
representation is different from its corresponding
The information bottlenock principle offers a principled approach to encourage the learning of more abstract and invariant representations from input data for a given task, which forces networks to capture the underlying structures and dependencies in the data that are relevant to this task. Accordingly, using different constraints, we can obtain the learning principles for label- and disribbtioonddiccminnaave representations, as shown in Eq. (T) and Eq. (6). respectively. Specifically, dabbl-discriminative representa- tions aim to caplure the strongly label-related information, while aim capture the remaining weakly label-related information. However, the information bottleneck princi- ple finds trade-off between compression and loss of information. Therefore, exploring all the label-related information is impossible, and some essential details in the data may be discarded.
Discussion:
representations
3.3 Learning Auxiliary Networks
Following the information bottleneck principle in Eq. C apply auxiliary network fo with implicit constraint to learn complementary disttribution-dieciiinnaiii representation for its corresponding label- discriminative representation, i.e.,
(9) Directly modeling fo(x, d) by a network cannot ensure that and c are different, i.c., ensuring and c are learned from strongly and weakly label-related information, respectively. This is because is difficult to design an implicit constraint for fo(x, d) due to the unknown proposal distribution Q(C) in Eq. (8) We develop an indirect modeling method by an implicit constraint. The basic idea is to decompose the distribution-diccrinnnnttt reprosentation into multiple component representations {zı,..., 2o0 } where component representation z; less similar to the libei-discrininattie repireeiiiinniieeeerreennnaiiooi d is assigned with higher weight Wi, as shown in Figure
c = fe(( d).
3.3.1 Decompose cineibutiionDiisciminnatt Representations
We firstly decomposc Eq.
g into a lincar combination,
WiZi,
(10)
1=1
We assume z is drawn from the Gaussian distribution W(uz, Σz) without loss of generality. Inspired by the determinant point processes Kulesza & Taskar (2012); Anari et al. (2016) selecting diverse samples according to the kernel-based distance Kulesza & Taskar (2011), we construct by integrating diverse component representations (21,..., (Zx) where component representation z less similar with the corresponding label- discriminative representation d has higher weight. Simulating the idea of the attention mechanism |aawann ct al. 2017D, the inner product is adopted as the similarity metric. We thus define the weight w; as
= 1- 11--zi ez d.
(11)
where e is a small perturbation coefficient to ensure that the weight positive. Substituting Eq. Eq. (10). we obtain an implicit constraint on c,
into
Ey (z- ex. dz) Fzz [z] e,ay, [[z (((2-)z)TTbl dlz 1z2] <zzdpz uz (Ezd nzddz))
(12)
8

---

Published in Transactions on Machine Learning Research (08/2023)
3.3.2 Estimate Distribution-Diccriinnee Representations
According to Eq. [2), we can estimate a distribution-diiciininattee representation с by estimating the expectation jzz and the covariance matrix Σz. Following Bayesian neural networks Gal & Ghahramani (2016); Ardywibowo et al. (2022), we estimate μ2 by applying an component network z8 which maps an input to the component reprosentation expectation. i.c.,
zØ(x) = Uz.
(13)
Note that fo and share the same network backbone and parameter 0 because indirectly construct fo by Eq. 9 Eq. 12 and Eq. [3] Accordingly, the relationship between fo and 2ø is
fo(x, za(x) - (Zgd + 2x(x)" dz)dz))())
dzzxx)
(14)
which is the implicit constraint for learning distributionddisciminatvee representations, i.e., constructing a distributiorddicciinnatiee representation by integrating multiple component representations differing from its corresponding label-discriminative representattions. The indirect construction method considers the im- plicit constraint ensure that the Intributioo-diicrmmnnati rosponding label-discrmnnnttive representations, which ensures that label- and distrbutiion-dserimiaatti represenlations are learned from strongly and weakly label-related information, respectively.
representations are different from the cor-
However, the covariance matrix Vg is still unknown. We can define the covariance matrix according to our prior knowledge. This is because Σz which represents the dispersion degree of component representation representations. Since the covariance
z does not play a decisive role in learning matrix Ep of ID samples can be easily estimated by the pretrained network go- We thus assume Σz ED without loss of generality.
3.3.3 Objective Function
Following the learning principle of label-discriminativ representations, we apply the cross-entropy loss model ](C:)) and assume h(:-)) is the softmax function. Accordingly, the objective function for learning distiibutoondiieiinnnaiie representations is,
min P(x.)) lo,
8
(15)
s.t. c= fo d). d = (x
The parameter à is fixed for learning the parameter 0 in fo because go pretrained network. Based Monte Carlo Wu & Gleich (2019), we apply the stochastic gradient descent optimization algorithm Shalev-
Shwartz & Ben-Davit (2014) to estimate the gradient of Eq. 15], where the batch size is B.
3.4 Calculating Out-of-distribttion Scores
For an ID sample, both its label- and distribuution-dieriinnnaii representations contain information pointing to the same label. For an OOD sample with high- prediction, its label-diseriminative representation is sensitive to a label, while its duatribttiooddisiinnnatvo representation is sensitive to other labels or even none of any labels. Therefore, the labeling information in the label- and distributiorrddicriiinatee representations are complementary for ID samples but inconsistent for OOD samples. Therefore, we can detect OOD samples by combining these two representations. We obtain a softmax output of label y for input by simply averaging the softmax outputs of the two representations, i.c.,
O(x, v) h(c. v) + h(d, v)
y) + h(go(x).
(16)
We classify ID samples according to the softmax outputs (0(x, I),... O(x, K)}. Following the baseline method Hendrycks & Gimpel (2017) of detecting OOD samples which uses the confidence as the OOD score, we calculate the OOD score for input x by
S(x) = max 0(x, y),
(17)
vE[L.K]
9

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm Dual Representation Learning (DRL)
Input: pretrained nelwork 9á. perlurbation coefficient e, covariance Ep, batch size B
while convergence do Sample
(XB, from P(X,))
4: Receive d; = (x(x,), . vi [B]
5: Calculate C; = fx( d:), Vi E [B]
Estimate the objective function:
[(0)-
Σ
log ((c, V6)
B
Obtain gradients Vol(0) to update parameters
end while
Calculate out-of-distribution score:
S(x) max (h(c, h(d, vE(I,K]
10:
Output: S(x)
where an ID sample is expected to have a higher score, whereas an OOD sample expected to have a lower score. The pocudo-code of the DRL training procedure is summarized in Algorithm 1
4 Experiments
In this section, demonstrate the effectiveness of the proposed DRL method We compare DRL with post-hoc, confidence enhancement and ensemble methods. Furthermore, we analyze the effect of the hyper- parameters and net work backbones in DRL, run a set of ablation study experiments, and show the sensitivily of labeling information of label- and distribttionddicciinnatve representations.
4.1 Setups
We adopt the ResNet18 architecture He et al. (2016) for all the networks in the experiments and implement it in PyTorrc. The learning rate starts at 0.1 and is divided by 10 after 100 and 150 epochs in the training phase, and all networks are trained for 200 epochs with 128 samples per mini-batch. If not specified, we set = 0.001 and adopt the same network architecture for pretrained and auxiliary networks in the proposed method. For the pretrained network, we train it with a cross-entropy loss on an ID dataset. For learning the auxiliary network, we adopt Σz = Ep for constructing distributionddieeimnnaay representations not specified. Recall that Ep is the covariance matrix of label-discriminative representations of ID samples from the pretrained network.
We adopt CIFARIO Krizhevsky 2009) and Mini-Imagenee Deng et al. 2009 ID datasets to train neural networks. The numbers of classes of the two ID datasets are 10 and 100, respectively. The resolution ratios are 32 x 32 and 224 x 224, respectively. For data augmentation methods, we apply random crop and random hor-
izontal flip for CIFAR10 and resizing and random crop for Mini- Iini-Immgeeet. We adopt CIFAR100 Krizhevsky 2009), CUB200 Wah et al. (2011), StanfordDogs122 Khosla et al. 20011) OxfordPets37 Parkhi et al. (0012) Oxfordflowers 02 Nilsback Zisserman (200)), Caltech256 Griflin el al. (0066,, DTD47 Cimpoi el al.
(0014)
and COCO Lin al. (2014) OOD datasets to evaluate the OOD detection performance in the test phase.
In the test phase, each method calculates an OOD score for each test sample. The score gap between ID and OOD samples is expected to be large, which indicates we can clearly separate the two kinds of samples. Accordingly, to evaluate the detection performance of OOD samples, we adopt the following three metrics:
The source codes are httpe://ggttabb.co///autttttt-
10

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: OOD detection performance of six post-hoc methods and DRL. All the reported values are averaged AUROC over five trials. The subscript values denote the standard deviation. 'Ave.' represents the averaged AUROC across all eight OOD datasets. Boldface values represent the relatively better detection performance.
In-dist
Out-of-dist
Baseline
ODIN
MDD
RA
GEM
DRL
CTFAR100
87.0+b.22
86.0=b.0
86.5
85.8+b.11
07.4
p7.
0..
CUB200
61.5
56.0=o.o
57.1=b.u
66.5
62.9
6.2
63.7 fo.4
SlanfordDogss22 120
68.0
64.9....0
66.9
72.4
69.4
71.3
73. 73.1
OxfordPets37
12.
60.2
61.6=
65.
63.0
66.3=
67.8
CIFAR1O
Oxfordflowers102
88. 88.1
87.5=..0
87.3=o.0
88.9
88.4#0.0
90.4=0.1
91.2
Caltoch256
86.0
86.4+o.0
85.6
85.2+
86.5
84.1=
88.0+b..1
DTD47
89.4
91.0=b.0
90.6=b.u
89.2
89.0#t.0
88.4=o.u
92.6+t.11
COCO
87.3
87.8
87.6
86.6+o.0
07.4
85.2=o.0
t..
Ave.
78.7
78.7
77.9
79.9
79.2
79.4
81.9
CIFAR100
84.7
85.8
84.0
84.1
84.6
05
86.4
CUB200
71.8+0.4
71.5
70.
77.1
72.1
74.4=0.1
73.8
StanfordDooss10
65.6
63.6
64.0
62.0
63.5
65. 0..
67.
OxfordPets37
10.4
68.6
69.6
64.5
67.4
68.9
72.2
Mini-Imagena
Oxfordflowers102
79.5
80.4
78.0=o.u
76.5
77.6
79.8=o.0
81.4
Caltech256
78.
79.8
78.5
71.2
75.4
78.3=o.0
79.8
DTD47
72.8+1.5
73.9
71.9
76.1
72.6
78.6
74.5
COCO
78.9
79.2=
78.0
73.9
73.9
76.1 16.1
79.9
Ave.
75.2
75.3
74.2
73.1
75.1
75.9
76.9
the area under the receiver operating characteristic curve (AUROC) Davis & Goadrich (2006). the true negative rate at 95% (FPR95) Liang et al. (2018) and the detection accuracy (Detection) Liang et al. (2018) A higher AUROC score and a lower FPR95 and Detection score indicate better detection performance. Specifically, AUROC indicates the probability that a higher score assigned to an ID sample than an OOD sample. FPR95 indicates the probability that an OOD sample is declared to be an ID sample when the true positive rate is 95%. Detection indicates the misclassification probability when the true positive rate is 95%. In addition to OOD detection performance, Calibration Error (ECE) Guo et al. (2017). ECE uses the difference in expectation between confidence and accuracy to measure calibration. A network is considered calibrated if its predictive confidence aligns with its misclassification rate. In our experiments, the nunber of bins in ECE is set to 20.
also evaluate the classification Accuracy and Expected
4.2
Comparison Results
4.2.1 Comparison with Post-hoc Methods
We compare DRL with six post-hoc methods that train an OOD detector according to the outputs from a pretrained network. DRL does not modify the pretrained network, which is the same as the post-hoc meth- ods. Rather than training an OOD detector, DRI. improves OOD sensitivity by training an auxiliary network for the pretrained network to explore OOD-sensitive information. All the compared methods are based on pretrained networks. The pretrained networks used in the post-hoc methods the same as that in DRL.
11

---

Published in Transactions on Machine Learning Research (08/2023)
OOD detection performance seven confidence enhancement methods and DRL. Each value is averagod across all eight OOD datascts. The symbol indicates a larger value better, and the symbol & indicates a lower value is better. The boldface value represents the relatively better detection performance.
Table 2:
Dataset
Metric
JCL
CSI
SSL
DeConf-C
MOS
KNN+
CIDER
DRL
AUROC f
77.5
79.2
78.1
78.4
77.8
79.2
80.2
81.9
CIFAR10
FPR(95)
73.9
67.8
62.1
64.8
68.2
67.8
66.9
65.0
Detection +
26.7
24.6
26.6
25.9
27.5
25.3
24.6
22.5
AUROC f
72.8
75.2
75.6
75.3
75.3
74.5
75.4
76.9
Mini-Imagent
F9)(
86.5
85.9
82.5
85.8
86.3
86.2
85.3
83.2
Detection +
32.0
29.9
28.9
29.7
29.1
29.6
29.0
28.3
The considered post-hoc methods include the Baseline Hendrycks & Gimpel (0177) ODIN Liang et. al. (2018h, Energy-based Detector (ED) Liu et al. (2020), Mahalanobis Distance-based Detector (MDD) Lee
ct al. (0188)) Rectified Activations (RA) Sun al. (2021), and Gaussian mixture based Energy Mcasure- ment Morteza & Li (002).. For fair comparison, we add the scores from different layers without training
logistic regression
validation OOD dataset in MDD because the other compared methods do not access OOD samples in the training phase. For the other comparison methods, we follow the same setups as their
original ones.
The comparison results are summarized in Table 1 When the training ID) dataset is CIFAR1O, DRL achieves the best OOD detection performance on seven of the eight OOD datasets. Furthermore, when the training ID dataset Mini-Imagenet, a more complex dataset with more classes and higher resolution, DRL achieves the best detection performance on six of the eight OOD datasets and obtains the second-best. deteccioo performance the rost two OOD datasets. Overall, DRL achieves the best averaged OOD detection performance across diverse OOD datasets, which indicates that DRL outperforms the state-of-the-art post- hoc methods. The reason is twofold. The post-ho- hoc methods based on pretrained networks can only access the label-liscrminntivee representations that are sensitive to labeling. However, DRL trains an auxiliary network for a given pretrained network to explore the complementary distribationndiieiimnnaii representations that are sensitive OOD samples.
4.2.2 Comparison with Confidence Enhancement Methods
We compare DRL with the state-of-the-arl confidence enhancement methods in terms of AUROC, FPR(95) and Detection. Confidence enhancement methods retrain a pretrained network to improve the OOD sen- sitivity by modifying loss functions and training procedures. The adopted methods include JCL Lee et al. (2018a). CSI Tack et al. (2020). SSL Hendrycks et al. 2019b), DeConf-C C Hsu et al. (2020), MOS Huang & Li (0221) KNN+ (Sun et al. (2022), and Compactness and Dispersion Regularized learning Ming et al. (2023). The settings of all the comparison methods follow the original ones. For a fair comparison, following the grouping method in MOS, we apply K-Means clustering on feature representations to divide the training ID datasets into groups with similar concepts.
The comparison results are summarized in Table 2 We observe that DRL obtains significant improvement (4.64% on CIFAR10 and 2.04% on Mini- (maieemmge over the other state-of-the-at confidence enhancement methods in terms of AUROC, Likewise, DRL achieves significantly improved performance (13.90% Cl- FAR10 and 5.12% on Mini-Imagenet) in terms of Detection. Although DRL does not obtain the best results in terms of FPR(95), its performance is close lo the best one with a narrow gap (2.9 on CIFAR10 and 0.7 on Mini-Iman mini-Imagenet). Overall, the proposed DRL method outperforms the state-of-the- ment methods on both datasets terms of AUROC and Detection. The results demonstrate that exploring inrrbbuttoondisciimmatiee representations from weakly label-related information and integrating label- and distribution inniiutiiooniiiiimnnaiiiiii representations form effective mechanism to detect OOD samples. It is because
confidence enhance-
12

---

Published in Transactions on Machine Learning Research (08/2023)
Table 3: OOD detection performance on near and far OOD samples terms of AUROC. The boldface value represents the relatively better detection performance.
Dataset
Metric
JCL
CST
SST
DeConf-C
MOS
KNN+
CIDER
DRL
CIFAR100 (Near)
84.3
87.6
88.2
89.2
88.4
85.6
89.1
89.5
CUB200 (Near)
60.9
61.2
62.1
62.4
60.9
62.9
63.0
63.7
CIFAR10
Oxfordflowers102 (Far)
84.6
85.4
90.4
90.1
88.7
89.6
90.5
91.2
DTD47 (Far)
88.6
88.6
90.5
91.2
90.3
90.3
91.6
92.6
the distributionnddicrriniin discriminative representations coupled with the corresponding label-discriminative represen- tations contain more label-related information than any of them, which reduces the prediction confidence for OOD sample owning minimum label-related information and enhances the prediction confidence for an ID sample owning all the labeling information, It enlarges the confidence gap between ID and OOD samples. Therefore, DRL improves OOD) detection performance by leveraging more label-related information, which indicates that DRL may fail if the training ID samples contain little label-related information with numerous label-unrelated information. This is because training on these ID samples causes no remaining label-related information that the auxiliary network of DRL can explore.
Following the setups of Winkens et al. Winkens et al. (2020), also evaluate the OOD detection perfor- mance on near and far samples. Near and far samples represent the out-of-distribution samples slightly and significantly different from ID samples, respectively. For the ID dataset CIFAR1O, we adopt CIFAR100 and CUB200 as near OOD datasets because they contain some classes having similar semantics to that of CIFAR10. Furthermore, we adopt 0xfordflowers102 and DTD47 as far OOD datasets because the classes of the two datasets are vastly different from that of CIFAR10. The experimental results are summarized Table 3 All the considered methods can effectively detect far OOD samples due to the large semantic gap of classes between ID and OOD datasets. Furthermore, DRL achieves 2.62% improvement over the compared methods. Conversely, it is difficult to distinguish near OOD samples from ID) samples since the performance of each method is lower than that on far OOD samples. However, DRL still achieves the best detection performance and obtains more significant improvement, i.e., 2.94%. This is because DRL improves OOD sensitivity by exploring more label-related information from original inputs. Therefore, DRL can leverage more dotails to describe ID samples, which can more effectively differentiate from OOD samples with different information.
4.2.3 Comparison with Ensemble Methods
We compare DRL with ensemble methods that integrate independent networks with different initialization parameters. Specifically, we apply the traditional deep ensemble method which combines the output pre- dictions by calculating the average. Furthermore, we incorporate the adversarial samples into the deep ensemble method Guyon et al. (2017). Note that the traditional ensemble method degrades into the baseline method when the number of networks equals one. All the methods utilize multiple networks to explore more information from the ID samples to improve OOD sensitivity.
The comparison results are summarized in Figure For the two ensemble methods, the OOD detection increases the number of networks increases both CIFAR10 and Mini-Imagenet datasets. Furthermore, considering adversarial samples for the deep ensemble method leads to improved and declined performance on CIFAR10 and Mini-emagenet, respectively. Overall, DRL achioves the best detection performance on both datasets, and the performance of the two ensemble methods is close to that of DRL when the number of networks is five. Therefore, DRL only containing two dependent networks outperforms the ensemble methods containing more independent networks. The results verify that the proposed DRL method differs from the ensemble method substantially. Further, DRL combines label- and ieteributiio-discimmnnnn representations to fetch the OOD-sensitive information from training ID samples.
13

---

Published in Transactions on Machine Learning Research (08/2023)
CIFAR 10
Mini-Imagenet
82
77
81.5
76
81
AUROC
AUROC
80.5
75
80
DRL
74
DRL
79.5
Ensemble
Ensemble
Ensemble Adversarial
Ensemble Adversarial
79
73 1
2
4
2
# networks
# networks
Figure 4: OOD detection performance of two ensemble methods and DRL. The ensemble networks gradually increase independent pretrained networks with randomized initialization, and DRL only owns two depent networks, Le, pretrained and auxiliary network. Each point indicates the average AUROC across all eight OOD datasets.
95
90
Ep
85
AUROC
80
75
CIFAR100
70
CUB200
StanfordDogs122
65
OxfordP ets
Oxfordflowers 102
60
0.0001
80
80.5
81
81.5
AUROC (b)
(a)
Figure 5: (a) Effect of the perturbation coefficient on CIFAR10. Each curve represents the detection performance on an OOD dataset, and each point indicates the AUROC for the corresponding perturbation coefficient expectation E; (b) Effect the covariance matrix Ez on CIFAR10. Each bar indicates the average AUROC across all eight OOD datasets.
4.3 Parameter Analysis
4.3.1 Effect of perturbation coefficient
adopt small perturbation coefficient to ensure positive weight We also empirically verify that small perturbation coefficient is more suitable for the proposed DRL method. We select the value of the perturbation coefficient expectation in {0, 0.0001, 0.001, 0.01, Note that DRL degenerates to deep ensemble methods containing two independent networks when e= 0.
According to Eq. [11
The experimental results are shown in Figure 5a We observe that inercasing c can gradually improve the detection performance, although the effect is drastically reduced when sufficiently large (e > 0.01). Therefore, DRL prefers perturbation coefficient. that is small but larger than According to the
14

---

Published in Transactions on Machine Learning Research (08/2023)
AUROC
Accuracy
ECE
84
96
0.03
95.5
0.025
80
78
95
0.02
74
94.5
0.015
72
70
0.01
D
D
D+C
C
Figure Results of the ablation study on CIFARIO. The three subfigures report the results of AUROC, Accuracy and ECE, respectively. D indicates the label-discriminative representations from a pretrained net- work. C indicates the complementary D C indicates the combination of label- and distribbttion-diierinnnaat representations, i.e., the proposed DRL method.
cerribatioondisciiinnntte representations from auxiliary network.
D D+C
ECE 2.85%
ECE =2.78%
ECE =1.26%
0.8
Accuracy
0.6
Accuracy
0.6
Accuracy
0.6
0.2
0.2
0.2
0
0
5
10
15
20
10
15
20
5
10
15
20
Confidence
Confidence
Confidence
Figure 7: Calibration results on CIFAR10. The confidence is equally divided into 20 intervals, and each bar represents the expected accuracy of samples whose confidence values are in the same interval. The red dotted diagonal indicates the perfect calibration.
weight w measuring the dissimilarity between a label-discriminative representation d and a component representation z, applied to ensure that is positive. However, large cannot satisfy the criteria. From another point of view, the expression of the distributionndiscriinnatt is similar to adversarial samples Goodfellow et al. (2015) where the coefficient affects the portion of the perturbation Σzd + 1zzdzz uzz. According to the idea of adversarial learning, small coefficient is sufficient to alter the predicted label of a test sample. Similarly, in DRL, a small is sufficient to construct a instibutioondiicimnnattvo ropresentation c significantly differing from the corresponding label-diseriminatio representation, and a large can cause an unstable insstribttion-ddisiiiinaiiii representation which cannot. capture the weakly label-relatod information from the training ID samples.
representation in Eq. 12)
4.3.2 Effect of covariance matrix Ez
We empirically verify that DRL is not sensitive to the choice of covariance matrix Σz, and it more suitable to adopt the covariance matrix Ep estimated from label-discrimmnatiee representations to establish the implicit constraint Eq. (1) for learning the corresponding distrbbution-discimmnatii representations, i.e,, Σg = Ep. We compare Ep with different choices of covariance matrix Ez, including the identity matrix Σz, the Gaussian random matrix Eg. and the uniform random matrix Eu-
The experimental results are shown in Figure 5b Overall, the performance gap (180.7,81.8) among the four different covariance matrices is significant. This indicates that the covariance matrix Σz does not play decisive role in learning distributiion-dieriiinne representations, and DRL not sensitive to the
15

---

Published in Transactions on Machine Learning Research (08/2023)
original
21
strongly label-related information
weakly label-related information
malamute
ferret
carrier
vase
slot
orange
Figure 8: Heat maps of Grad-CAM for label- and iustribttionddiscimmnatii representations on Mini- Imagenct. Red regions correspond to high scores for class, while blue regions correspond to low scores. The first row represents the original images, the second row represents their strongly label-related informa- tion for learning label-discriminative representations, and the third row represents their weakly label-related information for learning distributionn representations. The figure is best viewed in color.
choice of Σg. Specifically, the two random matrices Eg and En achieve relatively poor performance, while the identity matrix achieves relatively better performance. We expect that this because networks tend decouple the features within an output representation. Therefore, the identity matrix, which assumes all features are independent, more appropriate than the two random matrices assuming random feature relationships. Adopting Ep achieves slightly better performance than Σz. This is because Ep is obtained from the pretrained network, and the pretrained and auxiliary networks and are trained on the same ID dataset. Therefore, Ep is the most appropriate estimation for Σz. Note that the covariance matrix only represents the dispersion degree of representations. Accordingly, assum- ing the pretrained and auxiliary networks own the same covariance matrix docs not indicate their output representations satisfy the same unknown distribution.
the same network structure
4.4 Auxiliary Network Backbone Analysis
To verify the effect of auxiliary network backbones, for the pretrained network with ResNet18 backbone He ct al. 2016)), we adopt Lightweight Network (LN), VGG19 Szegedy et al. (2015) and DenscNet100 Huang et al. (217)) for its corresponding auxiliary network. LN is a shallow multi-layer perceptron which is fully- connected and has two hidden layers 128 ReLU units. The experimental results on CIFAR10 are shown in Table 4 LN obtains the worst detection performance since the shallow network merely explores limited weakly label-related information. Conversely, the rest three network architectures that are more powerful achieve better results. Specifically, DenseNet100 with the most significant number of parameters achieves
16

---

Published in Transactions on Machine Learning Research (08/2023)
Table 4: Effect of auxiliary network backbones on CIFARIO. Each value averaged across all eight OOD datasets. The boldface value represents the relatively better detection performance.
LN
VGG19
DenseNect00
ResNet18
80.0
81.2
82.7
81.9
the best results. Accordingly, the auxiliary network of DRL can adapt to different network architectures, ie., the networks for exploring strongly and weakly label-related information could have different network architectures. Furthermore, the detection performance depends on the power of the adopted network archi- tecture.
4.5 Ablation Study
We run a set of ablation study experiments to verify that label- and distribution--iierininiii representa- tions are indispensable for improving OOD detection performance. Recall that DRL contains a pretrained network and an auxiliary network where the two networks learn label-diseriminative representations (D) and cnnerbuutoon-iicrmmnnatiii representations (C), respoctively, and apply the combination (D + C) of the two represental by Eq. 11 to detect OOD samples. Accordingly, we compare the combination with the two different representations in terms of AUROC, Accuracy and ECE.
The results are presented in Figure 0 and Figure 7 They show that exploiting only the label-disrrmmnaaiiee represenlations. representations achieve slightly better performance in detecting
representations has similar performance to exploiting only the instribbutionddiciiiiin Specifically, the instribution-disciimnnn OOD samples but slightly worse performance classifying ID samples than the aabel-iisrimmnntive repre- sentations. The main reason is that the cestributiondisccimnaattt information that is weakly related to its labeling, and the weakly label-related information is more sensitive to OOD samples than the strongly-related information in the label-discriminative representations. From Figure 7 we observe that the pretrained and auxiliary networks are poorly-calibrated generating highly over-confident predictions while DRL is nearly perfectly calibrated. Overall, considering both label- and distibbutiondiisciiinaatvee representations, DRL obtains better performance on all metrics than any of the two components. Therefore, DRL can lake advantage of both components, which indicates that label- and representations are complementary to each other.
representation of an ID sample contains
4.6 Visualization
We apply Grad-Cam /elvaraju (2020) produce coarse localization maps that highlight the essential regions for learning label- and distributionn-diirimnaati representations. Figure X visually shows the heat maps of different classes of input samples on pretrained and auxiliary networks. We can observe that the two networks focus on different regions for the same imput sample. For example, for the malamute, the pretrained network focuses on the head, while the auxiliary network focuses on the body. According to the design principles of the two networks, the pretrained network trained without any constraint on labeldiisrimmintive representations, while the auxiliary network is trained with an implicit constraint to ensure that the learned distributior-diieriinnnai representations differ from the corresponding label- discriminative representations. Therefore, know that the pretrained network tends to extract strongly label-related information from ID sample to learn label-diseriminnttie representation which is the head information in the malamute. Conversely, the auxiliary network tends to extract the weakly label- related information to learn a distributtonnddicciinnnaa represenlation which is the body information in the malamute. According to the presented experimental results and the properties of the two networks, the pretrained and auxiliary networks extract strongly and weakly label-related information from inputs learn representations, respectively. For an ID sample, the label- and distrbaationdddeerinnaaee that correspond to the same label are complementary.
discriminative representations
17

---

Published in Transactions on Machine Learning Research (08/2023)
5 Conclusion and Future Work
In this paper, we propose Dual Representation Learning (DRL) method combining both label- and disiiibttion.iiiciiinaaive representations to improve the OOD sensitivity. From the modeling perspec- tive, a pretrained network learns label-discriminative representations that are strongly related to labeling, while an auxiliary network learns complementary related to labeling. From the data perspective, the label- and distrrbutioo-ddiserinnnaiis representations are complementary in labeling for an ID sample and correspond to different labels for an OOD sample. Based on the different informativenoss properties of ID and OOD samples, DRL distinguishes ID and OOD samples according to the OOD scores estimated by integrating the two representations. We empirically demonstrate that DRL more effectively detects OOD samples than the state-of-the-art post-hoc, confidence enhancement and ensemble methods across various datasets. This work makes the first attempt to improve the OOD sen- sitivity by learning both label and distributionnsensitivee representations from training ID samples. However, it pays the price to learn an extra network built. on a pretrained work. A riveting follow-up is to explore the OOD-sensitive information by augmenting ID samples and then finetuning pretrained network with the augmented ID samples.
reprosentations that are weakly
Acknowledgments
This work was supported in part by the Australian Research Council Discovery under Grant DP190101079 and in part by the Future Fellowship under Grant FT190100734.
References
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, pp. 1-19, 2017.
Nima Anari, Shayan Oveis Gharan, and Alireza Rezaci. Monte carlo markov chain algorithms for sampling strongly rayleigh distributions and determinantal point processes. In Proceedings of the 29th Conference on Learning Theory, number 103 115, 2016.
Randy Ardywibowo, Shahin Boluki, Zhangvang Wang, Bobak J. Mortazavi, Shuai Huang, and Xiaoning Qian. VFDS: variational foresight dynamic selection in bayesian neural networks for efficient human activity recognition. In International Conference on Artificial Intelligence and Statistics, pp. 1359-1379, 2022.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, pp. 530-539, 2018.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: review and new per- spectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798 1828, 2013.
Christopher M. Bishop. Pattern Recognation and Machine Learning. springer, 2006.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pallern Recognition, number 3606-300 0013,633 2014.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23th International Conference on Machine Learning, pp. 233 233-240, 2006.
Jia Deng, Wei Dong, Richard Socher, Ti-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognétion, pp. 248 255, 2009.
Marco Foderici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representa- tions via multi-view information bottlenock. In 8th International Conference on Learning Representations, pp. 1-26, 2020.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, pp. 1050-1059, 2016.
Maoguo Gong, Jia Liu, Hao Li, Yu Xie, and Zedong Tang. Disentangled representation learning for multiple altribules preserving face deidentification. IEEE Trans. Neural Networks Learn. Syst., 33(1):244 256, 2022.
Jorge Gonzalez-Lopez, Scbastián Ventura, and Alberto Cano. Distributed selection of continuous features multilabel classification using mutual information. IEEE Trans. Neural Networks Learn. Syst. 31(7): 2280 2293, 2020.
lan Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pp. 2672-266222 2680, 2014.
Tan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference Learning Representations, pp. 1- 11, 2015.
Greg Griffin, Alex Holub, and Pietro Perona. The Caltech 256. Technical report, Caltech Technical Report, 2006.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Tn Proceedings of the 34th International Conference on Machine Learning, pp. 1321 1330, 2017.
Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garneut Simple and scalable predictive uncertainty estimation using deep ensembles. Tn Advances in Newral Information Processing Systems 30, pp. 6402-6413, 6413, 2017.
Jarrod Haas, William Yolland, and Bernhard Rabus. Linking neural collapse and 12 normalization with im- proved out-of-distribution delection in deep neural networks. Transactions on Muchine Learning Research, pp. 1- 19, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Paltern Recognition, pp. 770 778, 2016.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and disttibbiitiiito examples in neural networks. In 5th International Conference on Learning Representations, pp. 1-12, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Proceedings of the 36th International Conference on Machine Learning, number 2712-2721, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In Advances in Neural Information Processing Systems 32, pp. 15637-15648, 2019b.
Yen- Ven-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN detecting dut-of-distribution image without learning from out-of-distribbtiio data. In IEEE Conference on Computer Vision and Pallern Recognition, pp. 10948 10957, 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convo- lutional networks. In IEEE Conference on Computer Vision and Pottern Recográtion, pp. 2261-2261 2017.
Rui Huang and Yixuan Li. MOS: towards scaling out-of-distribution detection for large semantic space, IEFE Conference Computer Vision and Pattern Recognation, pp. 8710 8710-8719, 2021.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In Proc. CVPR Workshop on Fine- Graned Visual Calegorizalion, number 1 1-2, 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Iya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advonces in Neural Information Processing Systems 25, pp. 1106 1114, 2012.
Alex Kulesza and Ben Taskar. Learning determinantal point processes. In Proceedings of the Twenty- Seventh Conference on Uncertainty Artificial Intelligence, pp. 419-427, 2011.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Found. Trends Mach. Learn., 5(2-3):123 286, 2012.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--alibratee classifiers for detecting out-of-distribution samples. In 6th International Conference on Learning Representations, pp. 1 16, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems 31, pp. 7167 7177, 2018b.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of distofiiisti image detection in neural networks. In 6th International Conference on Learning Regresentations, pp. 1 1-27, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, volume 8693, pp. 740-755, 2014.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based based distrribittin detection. In Advances in Newral Information Processing Systems 39, number 1-12, 2020.
Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems 31, pp. 7047 7058, 2018
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out- of-distribution detection? In 11th International Conference on Learning Representations, pp. 1 19, 2023.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regular- ization method for supervised and semi-supervised lcarning. IEEE Trans. Pattern Anal. Mach. Intell., 41 (8):1979-1199) 1993, 2019.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In Thirty-Sinth AAAT Conference on Artificial Intelligence, pp- 7831-7840) 2022.
Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Com- puter Society Conference on Computer Vision and Pattern, number 1447-1454, 2006.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, number 3498-22 1998-3505,4 2012.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Takshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems 32, pp. 14680 14691, 2019.
Chandramouli Shama Sastry and Sageev Oore. Detecting aut-of-distriuutin examples with gram matrices. In Proceedings of the 37th International Conference Machine Learning, number 8491-8501, 2020.
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracoy, and David D. Cox. On the information bottleneck theory of deep learning. In 6th International Conference on Learning Representations, pp. 1-27, 2018.
20

---

Published in Transactions on Machine Learning Research (08/2023)
Robert Schmier, UlIrich Koethe, and Christoph-Vikolat Strachle. Positive difference distribution for im- age outlier detection using normalizing flows and contrastive data. Transactions on Machine Learning Research, pp. 1 34, 2023.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization, Int. J. Compul. Vis., 128(2):336 359, 2020.
Gabi Shalev, Yossi Adi, and Joseph Keshet. representations. In Advances in Neural Informalion Processing Systems 31, pp- 7386-778 73966,736--- 2018.
detection using multiple semantic label
Shai Shalev-Shwartz and Shai Ben-David. Underslanding Machine Learning From Theory to Algorithms.
Cambridge University Press, 2014.
Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: Out-of-distributioo detection with rectified activations. In Advances in Newral Information Processing Systems 34, pp. 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. In International Conference Machine Learning, pp. 20827-20840, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty delection via contrastive learning on distributionally shifted instances. Number 1-14, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advonces in Neural Information Processing Systems 30, pp. 5998 6008, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD birds-200-2011 datasct. Technical report, California Institute of Technology, 2011.
Fan Wang, Hao Tian, Haoyi Xiong, Hua Wu, Jie Fu, Yang Cao, Kang Yu, and Haifeng Wang. Evolving de- composed plasticity rules for information-ootlenneee metalleannigg Tronsactions on Machine Learning Research, pp. 1-23, 2022.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, A. Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved aut-of-distribution detection. CoRR, 2020.
Tao Wu and David F. Gleich. Multiway monte carlo method for linear systems. SIAM J. Sci. Comput., 41 (6):3449 3475, 2019.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Proceedings of the European Conference on Computer Vision, pp. 102 117, 2020.
Zhilin Zhao, Longbing Cao, and Kun-Yu Yu Lin. Revealing the distributional vulnerability of discriminators by implicit generators. IEEE Trans. Pattern Anal. Mach. Intell,, pp. 1-13, 2022.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae-ki Cho, and Haifeng Chen. Deep autoencoding Gaussian mixture model for unsupervised anomaly detection. In 6th International Conference on Learning Representations, pp. 1-19, 2018.
21


Reviews:
[{"review_id": "S67SfQjAE9", "reviewer": "Reviewer_5W2A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper addresses an important issue in the field of OOD detection, offering an innovative approach that builds upon the exploration of both strongly and weakly label-related information.\n- The concept of DRL is unique and appears to be effective based on the experimental results provided.\n- The authors have done a commendable job in providing a clear and detailed explanation of the methodology and the corresponding results.\n- The research seems to be well-motivated and theoretically grounded, offering an interesting extension to the current body of knowledge in the area of OOD detection.\n\nWeaknesses:\n- While the experimental results indicate that DRL outperforms the state-of-the-art methods, a more comprehensive comparison and deeper analysis might strengthen this claim. For example, detailed discussions about why DRL works better and under what conditions it might not perform as well could be valuable.\n- There could be a more theoretical discussion about the information bottleneck principle, including its limitations, as it forms a crucial part of the research foundation.\n- Do you have conducted a sanity check of the Distribution-discriminative Representation? Specifically, whether the distribution-discriminative representations are still useful for discriminating the labels of input data? Is it complementary to label-discriminative representations in the sense of ordinary classification? Based on the motivation of the paper, the distribution-discriminative representations would be useless in classification.\n\nOverall, this paper represents a valuable contribution to the field of OOD detection, introducing a novel methodology that seems to be effective based on the experimental results. With a few improvements, particularly in terms of clarity and depth of explanation, the paper could be even stronger.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method, Dual Representation Learning (DRL), for out-of-distribution (OOD) detection. The main idea is to make OOD samples more challenging to be assigned with high-confidence predictions by learning from both strongly and weakly label-related information. DRL trains an auxiliary network to explore weakly label-related information and learn distribution-discriminative representations. The paper demonstrates that DRL outperforms state-of-the-art methods in OOD detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Already discussed with suggestions in the section above.\n\n", "broader_impact_concerns": "N/A\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rr5qLjYoA4", "reviewer": "Reviewer_ZE3v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Clear and reasonable motivation for improving OOD performance of pretrained models.\n2. The idea of learning complementary representation based on a learned representation sounds interesting. \n3. The method seems novel.\n\nWeaknesses\n1. I'm convinced that label-discriminative representations are insufficient for OOD tasks since they lose information on original data. But I'm not convinced that the complementary representations should be learned with the same objective ((1) and (6)). I think the author should be more specific to this argument.\n2. Equation (14) indicates the use of mean and covariate. But there is no mechanism to ensure z_\\theta(x) to be (or learn to be) the mean. So I'm not sure if the deduction of (10)-(13) is necessary and correct.\n3. Is label-discriminative representation necessary for OOD tasks since we only need to identify whether a sample is in distribution?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First, this paper argues that pretrained networks learn label-discriminative representations and overlook distribution-discriminative representations, which is a reasonable explanation for their suboptimal performance. Second,  it proposes a novel learning principle of distribution-discriminative representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. On page 8, \"Accordingly, the correlation between $f_\\theta$ and $z_\\theta$ is\", \"correlation\" is a term in statistics. I recommend changing a word to avoid confusion", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uAKgKdbb1q", "reviewer": "Reviewer_Z9uQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear, and the idea of DRL is interesting and novel.\n2. The reported results are superior to the compared baselines and the ablation studies as well as parameter analysis are sufficient.\n\nWeaknesses:\n1. The compared baselines need to be enriched. There are some more recent methods published in 2022 and 2023 that should be introduced and compared, such as \"Out-of-Distribution Detection with Deep Nearest Neighbors\", \"Provable guarantees for understanding out-of-distribution detection\" \"How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?\".\n2. The theoretical analysis and support is weak. I am interested to see if there is a correlation between the similarity of different datasets and the final performance, such as DTD47 vs. CIFAR10 and CUB200vs. CIFAR10. Further analysis on such results is also insufficient.\n3. For visualization, is it possible to show some results on ImageNet? Because CIFAR 10 is relatively simple, the heat maps are easier.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper works on out-of-distribution detection and proposes Dual Representation Learning (DRL) to distinguish in- and out-of-distribution samples. The motivation is to make out-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network with label-discriminative representations, an auxiliary network is trained to explore the remaining weakly label-related information to learn distribution-discriminative representations. That is, for a label-discriminative representation, its complementary distribution-discriminative representation is constructed by integrating diverse representations. The goal is to combine label- and distribution-discriminative representations to detect out-of-distribution samples. Experiments are conducted on popular benchmark datasets to demonstrate the effectiveness of the proposed DRL for out-of-distribution detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were unanimous that the revised submission meets the criteria for acceptance and addressed open concerns from the initial submission.
Decision: Accept (poster)