Revisiting Deep Hybrid Models for Out-of-Distribution Detection
Authors: Paul-Ruben Schlumbom
Eibe Frank
Venue: Accepted by TMLR ()

Abstract: Deep hybrid models (DHMs) for out-of-distribution (OOD) detection, jointly training a deep feature extractor with a classification head and a density estimation head based on a normalising flow, provide a conceptually appealing approach to visual OOD detection. The paper that introduced this approach reported 100% AuROC in experiments on two standard benchmarks, including one based on the CIFAR-10 data. As there are no implementations available, we set out to reproduce the approach by carefully filling in gaps in the description of the algorithm. Although we were unable to attain 100% OOD detection rates, and our results indicate that such performance is impossible on the CIFAR-10 benchmark, we achieved good OOD performance. We provide a detailed analysis of when the architecture fails and argue that it introduces an adversarial relationship between the classification component and the density estimator, rendering it highly sensitive to the balance of these two components and yielding a collapsed feature space without careful fine-tuning. Our implementation of DHMs is publicly available.

Published in Transactions on Machine Learning Research (04/2025)
Revisiting Deep Hybrid Models for Out-of-Distribution Detection
Paul-Ruben Schlumbom
pschlumb@uuakato.coo
Deportment of Computer Science University of Waikalo
Eibe Frank
eibe. abb..frankk@huakaatt..ccm ac.
Department Computer Science University of Waikato
Reviewed on OpenReview: openrevveee........... net/ /ortm? id= IeInhuu4
Abstract
Deep hybrid models (DHMs) for out-of-distribution (OOD) detection, jointly training deep feature extractor with a classification head and density estimation head based on normalising flow, provide conceptually appealing approach to visual OOD detection. The paper that introduced this approach reported 100% AuROC in experiments on two standard benchmarks, including one based on the CIFAR-10 10 data. As there are no implementations available, we out to reproduce the approach by carefully filling in gaps in the descrip- tion of the algorithm. Although we were unable to attain 100% OOD detection rates, and results indicate that such performance is impossible on the CIFAR-10 benchmark, achieved good OOD performance. We provide detailed analysis of when the architec- ture fails and argue that it introduces an adversarial relationship between the classification component and the density estimator, rendering it highly sensitive to the balance of these two components and yiclding a collapsed feature space without careful fine-tuning. Our implementation of DHMs publicly available
1 Introduction
Out-of-distribution (OOD) detection is crucial for reliable Al systems in safety-critical domains. Deep learning systems are known to exhibit "silent failures" yielding highly confident incorrect predictions (Guo ltt al. 2017], prompting the need for methods that recognise inputs outside the distribution underlying the training data. One such approach is to use density estimators to model this distribution. Normalising flows, which perform density estimation by establishing diffeomorphism between the distribution of the training data and a well-defined base distribution, are flexible density estimators that seem very suitable for this. However, Nalisnick et al. (2018) observed that they often assign higher density to OOD obscrvations than training data. Kirichenko et al. (2020) argued this is because they model the distribution in a low-level feature space when applied to pixels directly, indicating the need for approaches based on high-level semantic features. The deep hybrid model (DHM) for OOD detection (Cao & Zhang 2022) follows this approach and reportedly achieves 100% OOD detection rates on common OOD detection benchmarks, a result that is cited as state-of-the-art (see, e.g., Humblot-Renaux et. al. (2023)
As software implementing DHMs for OOD detection is unavailable, we set out to replicate the approach. We obtain results comparable to other state-of-the-art methods but short of 100% AuROC. Indeed, similarly to Fort et al. (2021), we identify overlap between CIFAR-10 and CIFAR-100, rendering such result ques- tionable. We argue that the adversarial relationship between the classification component and the density estimator renders the DHM extremely sensitive to the balance the loss signals from these two components.

---

Published in Transactions on Machine Learning Research (04/2025)
This is a commonly recognised dynamic in the generative modelling literature; see, for example, (Xiao al, 2019h. We also argue that the proposed mechanism by which the DHM's feature extractor is supposed to preserve input volume in the feature space is problematic in practice, contributing to the DHM's brittleness.
Our contributions in this work are as follows:
1. We replicate the DHM and provide its code.
We find that performance is competitive with SOTA, but not better, and give reasons why 100% AUROC does not seem achievable.
3. We argue that considering the properties of the model, this is explained by an adversarial relationship between the classifier and the density estimator, and by a failure of the feature encoder to uphold the polume-preservee assumption posited by Cao & Zhang 2022)
2 Background and Related Work
Recent work in OOD detection in the context of classification, motivated by the arguments in Ahined & Courville 22000), focuses on methods of detecting new classes of data exhibiting similar distributions of low-level features as the classes in the training data. OOD performance typically measured with an OOD testing set containing different classes than the in-distribution (ID) training set. The CIFAR-10 and CIFAR- 100 datasets (Krizhevsky et al., 2009), both sampled from the "80 million tiny images" dataset (Torralba) et al. 2008), and exhibiting mutually exclusive classes, a popular benchmark. Similarly, the SVHN dataset (Netzer et al. 20011) is often employed as benchmark OOD data for CIFAR-10. Normalising flows trained on CIFAR-10 often assign higher density to SVHN samples (Nalisnick et al. 2018).
Several approaches have been proposed to model the density in a higher-level feature space provided by a feature extractor. Loe et al. (2018) fit Gaussian distributions the fcature vectors of a trained classifier, while Feinman et al. (2017) applied a kernel density estimator, Liu et al. (2020) and yan Amersfoort et al. (2020) advance this by applying spectral normalisation to the feature extractor's weights. Spectral normalisation, introduced in (Yohhida & Miyato 2017),, imposes an upper bound on network's Lipschitz constant. In residual neural networks, setting this upper bound for residual connections in residual blocks will also set the lower bound for the Lipschitz constant of the block, a property known as bi-Lipschitz continuity {Liu ct al. 2020). This is decmed particularly suitable for OOD detection as it yields sensitivity in the feature ractor, meaning that change the input will yield a change the corresponding feature vector, and smoothness, meaning the feature space reflects distances in the input space. vaa Amersfoort et al.] (2020) and Postels et al. (2020) pointed out the tendency for "fcature collapse" in extractors obtained using purely supervised training, where OOD samples share features with ID classes, rendering them indistinguishable in feature space. The latter argue that this can be mitigated by imposing sensitivity. They also introduce the concept of smoothness; Liu et al. (2020) refer to it as distance awareness.
Normalising flows provide a flexible approach to density estimation, making them attractive for OOD de- tection in classification models. This has been investigated using deep hybrid models, e.g., the architecture described by [hhan et al. (2020), which uses a feature extractor o + to generate features h that are passed to classifier head C h y for classification and estimation, see Figure [ Cao & Zhang (2022) adopted this approach, obtaining the method we refer to DHM in this paper, by applying spectral normalisation to the feature extractor, similarly to van Amersfoort et al. 2020) and Liu et al. (2020). Use of spectral normalisation is motivaled by the argument that a tight Lipschitz bound will approximately preserve volume in the transformation from x to h. Sensitivity and smoothness are not considered. Notably, Cao & Zhang (2022) eventually loosen the Lipschitz bounds so as not to restrict the classifier's performance. Indeed, our results show that strong Lipschitz constraints usually disrupt classification performance in DHMs, even for simple synthetic datasets.
model F h + for density
2

---

Published in Transactions on Machine Learning Research (04/2025)
normalising flow
DNN feature encoder
h
classifier head C
y
Figure 1: Diagram of the deep hybrid model architecture for out-of cif-off-dstrbbutinn detection.
3 Implementing DHMs for OOD detection
The DHM's loss function combines the classifier (i.c., Coo) loss, Lana. and the normalising flow loss, Lut. The classifier loss Ldnn is the standard categorical cross-entropy loss. The normalising flow loss Luf is designed to maximise the likelihood the training samples and is defined as:
(0)
Σ log Dg (h4),
(1)
N
where log (h) (h) the log density assigned by the normalising flow to the sample h o(x). Note that Cao & Zhang (2022) argue that spectral normalisation in the feature encoder will ensure that p(h) p(z), and that estimating the density of h therefore also provides the density for æ. We question this claim in Section 4.4 For a normalising flow f defined in the normalising direction (i.e., f(h) = z), and composed of M diffcomorphic flow components, pi (h) is defined as
II det @fi(h+-1)
p2())>p(((())
(2)
Oh
where h; is the intermediate value f ft-1 o... o fi h with ho = h and he = 2, and pz some well-defined base density typically standard normal distribution. This is commonly stated
ph (h) = p2(((h)) |
(3)
Put together, the loss function for the DHM, LDHM is defined as
CDHM Ldon XLnf-
(4)
Since Ent can reach far greater magnitudes than Lanns and may overshadow it, is scaled by the hyperpa- rameter X. The DHM proves to be highly sensitive to X, as we explore further in Section 4.2
Following the implementation described by Cao & Zhang (2022), we a wide residual network (WRN) (Zagoruyko & Komodakis 2016) as the feature extractor; specifically, the variant with 28 convolutional layers and a widening factor of 10, WRN-28-10.
Spectral normalisation is applied to the weight matrices of the encoder network. We follow the method described by Gouk et al. (2021), where the weights are left unnormalised their spectral norm below some coefficient c, a hyperparameter to be tuned. Cao & Zhang (2022) recommend using 6.0, although similarly to van Amersfoort (2021), we find no degradation performance even for 3.0 (sce Figure
3

---

Published in Transactions on Machine Learning Research (04/2025)
We also apply Ls normalisation to the feature vector. This step is not specified by Cao & Zhang 2022 but we found it to be vital for achieving good performance. We discuss why this is in Section 4.5
For the normalising flow component, following Cao and Zhang, we use a residual normalising flow Behrraan et al. 2019). We employ the implementation by Chen et al. (2019), applying it with 10 residual blocks (i.e., M 110,, each with a hidden dimension of 640. We leave out activation normalisation, as this yields better performance. The original work on deep hybrid models by Zhang et al. (2020) also describes preprocessing the features with a logit transform before passing them to the normalising flow. Logit transforms were introduced by Dinh et al. 20166) to preprocess images for their ReaINVP normalising flow and are defined
as:
y = logit((a + (1 20x)
255 )
where the logit function is defined logit(p) log(p) - log(1 p), and is some small positive value that prevents the from being exactly zero or exactly one. The purpose of the logit transformation is to map values from (0, 1) to (-o0, oo). In the case of modeling images, this is beneficial as it represents the bounded distribution of pixel values as an unbounded distribution, i.c., it prevents the normalising flow from producing impossible pixel values in the generalive direction. However, since the normalising flow in the DHM is receiving the (theoretically unbounded) features of an encoder model rather than the image itself, the transformation is not appropriate, and we do not apply it. Indeed, koeping the logit transformation results in considerably worse performance.
We train the DHM components jointly, but with different algorithms for the weight updates. The feature extractor and classifier head are trained with SGD, using Nestcrov momentum with a momentum of 0.9, an initial learning rate of 0.05, and a weight decay rate of 5e-4. The learning rate is scaled by 0.2 at 60, 120, and 160 epochs, and the model (unless specified otherwise) is trained for 200 epochs with a batch size of 256. The spectral normalisation coefficient e is set to 3.0. The residual flow component is trained using Adam optimisation with learning rate of le-4 and a weight decay rate of 16e-4. We chose different methods for the weight updates of the two components as we found them to respond better to these regimes when trained in isolation. Morcover, the DHM also scems to attain the best performance when the two different algorithrms are used for the weight updates. When training for 200 epochs on an NVIDIA RTX 2080 Ti, this DHM configuration could be trained in about 17 and a half hours.
Additionally, uniform noise u ~ U 1) is added to the training images, method introduced by Uria et al. (2013) for training normalising flows on images to prevent the model from assigning arbitrarily high densily to discrete input pixel values. Although in DHMs, the normalising flow operates extracted features, these are deterministic transformations of the original images, and we empirically obscrved that introducing noise
is useful.
4 Experiments
We present results on the CIFAR and SVHN datasots for our implementation of DHMs. In Section 1. we cover our attempts at replicating the results from Cao & Zhang 2022). ) We investigate the effect of the
hyperparameter À in Section 4.2 In Section 4.4, we study the polume-preservee characteristics of the DHM, and in Section 4.5
we explore the role of sensitivity in DHM performance.
4.1 Replication of DHM Results
Following the training regime described by Cao & Zhang 2022). we train the DHM model on CIFAR- 10 training data and then evaluate the OOD detection performance CIFAR-100 and SVHN test sets compared to the CIFAR-10 test set. This is done by computing the density assigned to cach image by the DHM's normalising flow and then computing the area under the receiver operating curve (AuROC) and the area under the precision-recal curve (AuPR-in), where we treat ID samples as the positive class and OOD samples as the nogative class. This is a standard evaluation procedure the OOD detection literature, see Hendrycks & Gimpel (2016). Additionally, Humblot-Renan et al. (2023) argue that AuROC and AuPR- in scores fail to capture the level of separation between the positive and negative sample populations and
4

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset Log Densities
Validation Accuracy and AUROC vs. À
1.0
50
CIFAR10
AUROC Valid. Acc.
2
40
CIFAR100
0.8
0
SVHN
30
0.6
-2
0
validation accuracy
20
CIFAR100 AUROC
SVHN AUROC
10
0.2
validation loss
-8
0
640
-630
-610
-600
0.0
0.000
0.001
0.002
0.003
Log Density
X
Figure 2: Histogram of estimated log densities for
Figure 3: plot of validation accuracy, validation loss, CIFAR-100 AuROC, and SVHN AuROC scores against, A, showing an immediate collapse of perfor- mance above certain threshold.
the DHM trained on CIFAR-10.
AUROC and AUPR-in vs. C
1.0
AUROC or AUPR-in in
0.6
CIFAR100 AUROC
CIFAR10D AUPR-in
SVHN AUROC
0.2
SVHN AUPR-in
0.0
0
3
4
6
7
C
Figure 4: A plot of CIFAR-100 100 and SVHN AuROC and AuPR-in scores against the spectral normalisation coefficient for the DHM trained on CIFAR-10.
introduce the area under the threshold curve (AuTC) measure. The AuTC score becomes smaller as the separation of the populations becomes greater, and we report all three scores for our DHM implementation.
We report results for AuROC, AuPR-in, and AuTC in Table [ For comparison, we also provide the results reported by Liu et al. (2020) for their SNGP architecture, which represents the state-of-the-art and is structurally similar to the DHM. We show the results for SNGP implemented with WRN-28-10 10 feature extractor, is used in our DHM implementation, to enable a direct comparison. We also include performance scores calculated with the classifier component's 's softmax scores a simple baseline.
Our implementation of the DHM yields performance similar to the state-of-the-art in the field but is clearly not perfect. As is clear Figure the DHM is far from perfect separating the ID sample densities from the OOD sample densities. Indeed, it questionable whether a 100% OOD detection score possible at all in the CIFAR-10/ / benchmark: Fort et al. (2021) noted, there are several images in the CIFAR- 10 dataset that in fact depict CIFAR-100 classes (see Figure 5), well as ambiguous images such as vans that can be found in both the CIFAR-10 "car" class and the CIFAR-100 "bus" class. In performing nearest neighbour search over the CIFAR-10 and CIFAR-100 images using the features of our DHM model, we also found the case of an identical image present in both datasets, and CIFAR-100 images that arguably depict CIFAR-IK 10 categories (see Figure 6 Given the overlap between the CIFAR-10 and CIFAR-100 datasets, one would expect even an idcal model-with features that optimally exclude true OOD classes- include OOD samples and exclude ID samples in the actual datasets used, and thus be unable to attain 100% OOD detection rates.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Our DHM compared with results reported by Cao Zhang 2022 the softmax OOD detection baseline (results averaged over independent sceds). is used as the in-distribution dataset.
SNGP
(Liu et al. 2020 and The CIFAR 10 datasct
MODEL
METRIC
CIFAR-100
SVHN
DHM (ours)
AuROC t
0.897 + 0.008
0.96 + 0.01
AuPR-in f
0.908 + 0.007
0.966 0.007
AuTC 4
0.344 0.006
0.27 0.01
DHM (original)
AuROC
1.000 + 0.00
1.000 + 0.00
AuPR-in
1.000 0.00
1.000 + 0.00
SNGP
AuROC
0.960
0.902 0.01 0.990 = 0.01
AuPR-in
0.905 + 0.01
Softmax Baseline
AuROC
0.872 + 0.009
0.95 - 0.01
AuPR-in
0.80 = 0.02
0.93 - 0.02
AuTC L
0.427 + 0.006
0.36 = 0.02
(a) "car"
(b) "truck"
(a) "cat"
(b) "deer"
(c)
"truck"
(c) "bus"
(d) "pickup"
Figure Images in the CIFAR-10 dataset that do not match their class.
Figure 6: Nearest neighbours between CIFAR-10
(lop row) and CIFAR-100 (bottom row) samples.
4.2 Sensitivity to the Hyperparameter A
The DHM proves to be highly sensitive to the À parameter in the hybrid loss function (see Equation Since the log density calculated for the normalising flow is dependent on the determinant the flow 's Jacobian (see Equation 3 which can be arbitrarily large depending on the change density between æ and 2, the normalising flow loss can be considerably larger than the classificr loss. This can result in gradients obtained from that component of the loss overpowering those from the classification loss, causing the feature encoder to optimise for maximising pz(f (f()) by simply collapsing the feature space maximise density of the observations in this space.
This can be seen clearly when measuring performance over À values (see Figure 3 where OOD detection performance immediately collapses above a certain threshold. Validation accuracy also begins to degrade above this threshold, albeit more slowly, so discrimination still possible in the partially collapsed feature space. An inspection of the value of the final log determinant of the DHMs trained above and below the X threshold indicate that the feature space indeed collapses: moving from X 3.594e 3 to X 3.672e 3.

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
(b)
Figure 7: Examples of the "stripes" dataset, with horizontal/verre vertical in-distribution samples and diagonal out-of-distribution samples.
the value of the normalising flow's log determinant jumps from 0.1209 to 1113. This shows that, after the threshold is exceeded, the change in volume is very large when moving between & and æ: the density for must be scaled by large value conserve total probability when transformed to the space in which a resides because the feature values in æ are concentrated into very narrow region.
4.3 Feature Collapse in Synthetic Data
We demonstrate the DHM's feature collapse directly with a synthetic dataset 9x9 striped images, where vertical/horizontal stripes serve as the ID classes and diagonal stripes serve as the OOD samples. More specifically, randomly generate images by creating stripes between 1-4 pixels wide and assigning to each stripe value sampled uniformly from [0. 1]. For the OOD samples, the angle of the stripes for given image is sampled from between 15 and 75" (i.e., 45 + 15"), with the gradient flipped half the time to ensure diagonal stripes in both directions. See Figure 7 for examples.
This synthetic datasct has several benefits for analysing the DHM. First, the distributions of low-level features (i.e., individual pixel values) are uncorrelated with the target classes: the targel classes are only correlated with local patterns of pixel values (i.e., the gradient of pixel values along different axes), in other words, with higher-level features. This imitates the property wish to capture in more complex natural datascts.
This also highlights the second benefit, that the "high-level" features are known, and can therefore be captured with a hand-crafted feature encoder. We use a simple 2-channel convolutional layer with horizontal and vertical Prewitt filters (Prewitt et al. 1970) and an absolute value activation function, followed by a maxpooling layer to generate a dimensional output. Our DHM implementation uses the same architecture, leaving the convolutional parameters trainable. This allows us to compare the DHM's parameters after training known solution.
Finally, the features two-dimensional, they also enable direct visualisation for the benefit of inspection. The stripes dataset thus captures the relevant propertics of natural datasets that OOD detectors are intended to capture, while doing so at an interpretable scale.
To create the DHM, we apply a softmax operation to the features to serve as the "classifier head" and attach 50-layer residual normalising flow with a hidden dimension of to serve as the density estimator. Note that while significantly smaller normalising flows could still reach the performance of the 50 layer model, they took longer to achieve the same performance in our experiments.

---

Published in Transactions on Machine Learning Research (04/2025)
50
DHM Density Hcatmap
DHM Density Heatmap
0.5
0.5
Feature Density
Featura Log Denaty
0.4
0.4
Festure2
Feature
155
15
0.1
0.1
0.0
-0100
Feature
(a) Result for 0.01
(b) Result for
Figure 8: DHM density heatmaps for the stripes dataset, with datapoint embeddings overlaid, for low and high X values. Blue and green points the horizontal and vertical ID samples, respectively, while orange points represent the diagonal OOD samples. This shows the collapse of features for high A.
Upon conducting a search over À values, we see a repeat of the behaviour observed in Section 4.2 with OOD performance dropping to random guessing above a certain threshold. Visualising the features at two different À values (see Figure S we observe the features collapse into a single region that eliminates the distinction both between the ID and OOD samples and even the ID class samples.
4.4 Non-Volume Preserving Feature Spaces
For normalising flows applied to raw image data, feature collapse is not issue the datapoints are fixed; however, the DHM architecture allows h to be adjusted to maximise pz(z): the parameters of the feature extractor are free to be adjusted during training. To demonstrate this, we consider DHM architectures trained on small synthetic datasets with features for ease of inspection. Since the purpose the normalising flow is establish bijective mapping between the known density function pz and the unknown density We
PH, the density in the feature space preserves the volume of the base distribution and integrates are interested in the volume of the density function integrated over the input space. In the case of the DHM with 2D data, can pass the model grid of points covering the input space and measure the density at each point to approximate the overall volume assigned by the DHM to the input space via the trapezoid.
We set up a simple DHM architecture where the feature encoder consists dimension 128, and the normalising flow consists of a residual flow with residual blocks and hidden dimension of 128. By default, we use = 0.9 as the value for the spectral normalisation coefficient as this empirically yields high accuracy. We train the model the common " "moons" dataset, with classes, and slightly more complex dataset of 5 concentric rings, where each ring is separate class (see Figure g.
6-layer Resnet with a hidden
We find the DHM assigns an approximate volume under the density of 0.4 across the input space on average and does not preserve volume. Indeed, Cao & Zhang (2022) explain that this will only be the case when the network's Lipschitz constant is close to 1, which can be enforced by setting c & for a residual network (sce Section p. Unfortunately, in the degenerate case, this can result in a simple identity mapping. For this example problem, setting 0.2 yields an approximate volume of 0.995 for the input space but classification accuracy is reduced to 0.91 the test set, from 1.0. We note that the same pattern arises when training
8

---

Published in Transactions on Machine Learning Research (04/2025)
Val. Acc. & Approximate Volume vs C
1.0
0.8
15
Accuracy
0.6
Validation Accuracy
Volume
Appreximate Probability Volume
10
0.4
Reference Volume=1
5
0.2
0.0
0.8
Figure DHM densities assigned to moons (left) and rings (right) datasets, with the original data- points overlaid.
Figure 10: Testing accuracy and approximate vol- under the density on the synthetic rings dataset against the spectral normalisation coefficient. The horizontal dotted line denotes the expected volume of 1.
Feature Space Distance Input Space Distance
No Norm
0.30
Norm Regression
h Norm
Feature Space Distance
0.25
l Norm Regression
b Norm
0.20
I2 Norm Regression
Norm
I Norm Regression
0.05
0.00
0.0
0.5
1.0
1.5
2.0
2.5
Input Space Distance
Figure 11: Deviations in feature space against deviations in input space, for the full DHM trained on CIFAR- 10, showing that the Lo normalised DHM is the most sensitive configuration.
the full DHM on the CIFAR-10 dataset, where OOD performance rapidly degrades for c < 2.5 (see Figure
The phenomenon is more extreme on the rings dataset. Training the DHM with c = 0.9 yields a testing accuracy of 0.94, but the modelled density is concentrated on the central ring and has an approximate volume greater than 15. Constraining e ensures more evenly distributed probability densitics and an input space volume of 1, but in exchange for considerable drop in lesting accuracy 0.2 (see Figure 10
4.5 Sensitivity and Normalisation in Feature Space
Our results indicate that rather than being dependent on the volume-preserving characteristics of the fea- ture encoder, which was put forward as the motivation for spectral normalisation in Cao & Zhang (2022)
the DHM's performance depends on feature spaces exhibiting smoothness and sensitivity, the properties considered by van Amersfoort et al. (220 and Liu et al. (2020) (see Section 2).
It is instructive consider the effect of different forms of normalisation of the feature vectors on sensitivity, By applying random deviations to test input and measuring the resulting (Fuclidean distance) deviations

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Summary the normalisation schemes tested and the corresponding gradient and r-values. l2 and Io normalisation shows the largest r-values (i.e., the strongest correlation between input and feature space deviations) and the greatest rate of change in feature space with respect to input space.
NORM GRADIENT
R-VALUE
None
0.00478
0.847
h
0.00176
0.693
12 0.01879
0.930
1x
0.08966
0.921
in feature space, we can mcasure the sensitivity of a model directly. This reveals that the Ls normalised configuration of the DHM (as described in Section 3 yields the greatest change feature space for a given change in imput space on average, and therefore the most sensitive (see Figure II Furthermore, considering the correlation between feature space and input space distances (see Table 2 which shows the slopes of the lines in Figure 11 as well the R-values between distances), the strongest correlation. This is evidence of 'smoothness'; that is, the feature space distances most strongly correlate with input space distances with these configurations.
that l2 and &oo norm yield
This provides support for the use of ls normalisation in our experiments: it is likely to yield better OOD detection performance because renders the method more sensitive changes in the input and offers smoother feature spaces. This increases the separation of OOD samples from ID samples, making their identification simpler problem, while ensuring these distances meaningfully correspond to changes in the input space.
5 Conclusion
In this paper, revisited the deep hybrid model (DHM) architecture for OOD detection proposed by (Cao & Zhang] 2022). With our implementation of the algorithm, reached performance comparable to the state-of-the-an but below the 100% OOD detection rates reported by (Cao & Zhang 2022). This is consistent with obscrvations made by Fort et al. (2021) and our own findings, namcly that overlaps appear to make it infeasible to achieve perfect OOD detection rates on the datasets used in the experiments.
Our results show that the adversarial relationship between the classification head and the normalising flow under the joint training framework necessitates careful finc-tuning of the hyperparameter À to balance the contributions of the two components in the hybrid loss. We also found that the DHM's spectral normalization does not reliably preserve the volume of the input space, which calls into question the motivation for applying it put forward in (Cao & Zhang, 2022). We argue that the DHM's behaviour is better understood using the criteria considered by van Amersfoort et al. (2020) and Liu et al. (0200] that is, in terms the feature encoder's sensitinity and smootlness.
While the DHM offers some promising results, also proves to be fairly brittle. The fundamental issue it must contend with is the adversarial relationship between the classifier and the normalising flow, which encourages the collapse of features without careful fine-tuning. This suggests future work alternative structural approaches that avoid this unstable behaviour would be beneficial. We note that there is some overlap with the unsupervised learning literature, where there is considerable interest in developing non- contrastive feature learning methods that do not degenerate to a trivial collapsed feature representation. For example, there has been some recent success with self-distillation methods (Grill et al. 2020 Caron et al. 2021). Future work may look adopting some of these concepts develop more robust density- based OOD detection system.
10

---

Published in Transactions on Machine Learning Research (04/2025)
References
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3154-3162, 2020.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International conference on machine learning, pp. 573-582 PMLR, 2019.
Senqi Cao and Zhongfei Zhang. Deep hybrid models for out-of-distribution detection. In Proceedings of the IEFE/CVF Conference on Compuler Vision and Paltern Recognition, pp. 4733-4777777 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-superviedd vision transformers. In Proceedings of the IEEE/CVP international conference computer vision, pp. 9650 9660, 2021.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Densily estimation using real nvp. ar preprinl arXiv:1605.08e: 2016.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXio:1703.000 100:0000000 10, 2017.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distributtin detection. Advances in Newral Information Processing Systems, 34:7068-7081, 2021.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393 416, 2021.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Picrre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et. al. Bootstrap your own latent-a new approach to sele-supervised learning. Advances in neural information processing systems, 33:21271-2128. 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributiin examples in neural networks. arXie preprinl
2016.
Galadrielle Humblot-Renatee Sergio Escalera, and Thomas B Moeslund. Beyond auroc co. for evaluating out-of-distribution detoction performance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3880 3889, 2023.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data. Advances in neural information processing systems, 33:20578 20589, 2020.
Alex Krichevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep lcarning via distance awareness. Advances in neural informalion processing systems, 33:7498 7512, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? aixiv preprint 1xXi::110.091930 2018.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NTPS workshop deep learning and unsupervised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
Janis Postels, Hermann Blum, Cesar Cadena, Roland Siegwart, Lue Van Gool, and Federico Tombari. Quantifying alcatoric and epistemic uncertainty using density cstimation in latent space. orXiv preprint 0xXx:2012.03082, 1, 2020.
Judith MS Prewitt et Object enhancement and extraction. Picture processing and Psychopictorics, (1):15-19, 1970.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for non- parametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958-197) 2008.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density- estimator. Advances in Neural Information Processing Systems, 26, 2013.
Joost Amersfoort, Lewis Smith, Yee Whye Tch, and Yarin Gal. Uncertainty ostimation using single deep deterministic neural network. In International conference machine learning, pp. 9690 9700. PMLR, 2020.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. February 2021. URL http://arxiv.orr org/ abs/
2102.11409 11409
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv preprint arXiv:1905.14485, 2019.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arxio preprint arXiv: 1eei1705.19944 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks Xr preprint arXic:1605.0711 2016.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pp. 102-117 117. Springer, 2020.
12


Reviews:
[{"review_id": "Qa0vIxpdsr", "reviewer": "Reviewer_xiyb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses a method that made strong claims, but for which there is no known implementation. This is valuable info for the OoD community.\n- The claims are supported by experiments.\n\nWeaknesses:\n\n- There are some theoretical angles and experiments that could strengthen the paper, I comment on these below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors attempt to replicate the DHM results found in Cao & Zhang 2022. This is a relevant and interesting study because that paper claimed perfect scores on a standard OoD performance benchmark.\n\n- The authors quickly note three pathologies with the Cao & Zhang work: \n    - L_inf normalization, which was not used in the initial study, is critical for good performance\n    - Logit transforms, which were used in the original study, result in significant performance decreases\n    - CIFAR10/100 have duplications and errors that preclude 100% separation of these datasets in an OoD task\n\n- Additionally:\n    - classification performance is inversely related to input-space volume preservation under this method, illustrating that it has a fundamental limitation and further undermining the original papers claims (and also raising questions about volume preserving motivations for spectral normalization)\n    - The authors argue that discriminating samples via feature space relies on properties of sensitivity and smoothness over that feature space (previously proposed by van Amersfoort et al. (2020) and Lie et al. (2020)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Not critical to acceptance but please:\n\n1) It would be nice to have a plot of AUROC / AUPR changes with respect to C, the spectral norm coefficient. \n2) Normalizing flows can be quite slow to train. Please comment on the training resources used and how long it took, as that would further illuminate the utility (or lack thereof) of the proposed method\n3) a couple of typos: \"cleary\" should be \"clearly\" in last paragraph of 4.1, \"content\" should be \"contend\" in last paragraph of conclusion\n\nI would appreciate your comments on the below, as I think these ideas would strengthen the paper, but they are not necessary for acceptance:\n\n- Neural collapse metrics would be useful. Papyan et al. and others have studied the relationship between CE loss and the resultant simplex ETF geometry of feature space. There is a “compressive” effect here, and it is also known that contrastive loss approaches simplex ETF geometry, but mitigates intraclass variability (Fang and He et al, 2021), and this is probably why contrastive learning can produce discriminative features well suited for OoD and classification purposes. \n- It would be interesting to study what is happening to the simplex ETF geometry that CE loss will pursue while also trying to maximize P(z). Is flow loss “breaking” the simplex ETF structure significantly?\n- Flows are universal density approximators in theory, but employ a litany of shortcuts and approximations to manage the O(d^3) complexity due to the Jacobian. In my experience, a single flow trying to measure density over a simplex ETF with 10 modes would be tricky for most if not all implementations, let alone what ends up happening with high-dimensional issues (Nalisnick et al. 2018), but I’m open to push back on this as I haven’t tried to solve this extensively. I would be interested in knowing how OoD performance and the adversarial nature of flow/CE loss is tied to the residual flow implementation. This is especially interesting, I think, because L2 normalization of feature space will, of course, constrain to a hypersphere, and max-norm to a hypercube, and certain flow implementations exist to deal with the former, if not the latter as well. \n- In summary:\n    - I’m curious whether the adversarial relationship is less due to a trivial solution for flows (although I’m sure this is still in there) and more to do with a regularizing effect due to the flow implementation, and how is this related to the simplex ETF optimum of CE loss?\n    - I too like the “sensitivity and smoothness” way of conceptualizing feature space in this domain. I think Neural Collapse may shed some light on the shape of the manifold we end up dealing with under CE loss and maybe even suggest ways to characterize things.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LPWrv3q4Mj", "reviewer": "Reviewer_TFSm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy-to-read.\n2. The background provides good motivation for the analysis.\n3. The careful reporting of experimental details is much appreciated.\n\nWeaknesses:\n1. The claims could be better substantiated. For instance, a main contribution claimed by the paper is identifying an adversarial relationship between classification and density estimation, but specifically in individual use cases mentioned in the text rather than more systematically. Moreover, given Cao & Zhang 2022 present plots which do not show an obvious inverse correlation (e.g., Figs 4 and 5), a more substantiated claim would ideally explain how their previous results fit into the picture presented in this work.\n2. It's hard to pinpoint the precise contributions of the analysis beyond observations on a specific dataset / setting. While there seem to be various details in this paper not mentioned in the original, their presentation is rather diffuse and specific to a particular example, making it hard to appreciate whether they are unique to the specific setting run or more systematic. For instance, while many of the figures are interesting, they seem more illustrative (e.g., $\\ell_\\infty$ normalization performs better than others on a dataset) than a comprehensive experimental contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reproduces and analyzes an existing out-of-distribution detection algorithm, namely the Deep Hybrid Model (DHM) of (Cao & Zhang, 2022), of which there exist no public implementations. In their analysis, the authors notice an inverse relationship between classification and density estimation. The paper also presents other details for optimizing performance, e.g. tuning the loss coefficient hyperparameter, normalization of the feature vectors, etc.\n\nQuestions:\n\nCould the authors please enumerate the contributions of this work, e.g. in the intro, abstract, or conclusion, and emphasize what they see to be the primary contributions?", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could the authors please enumerate the contributions of this work, e.g. in the intro, abstract, or conclusion, and emphasize what they see to be the primary contributions?\n2. For claims made (e.g., identifying an adversarial relationship between classification and density estimation), could the authors ensure sufficiently supporting evidence is provided, e.g., experiments over multiple datasets and tasks?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JL9uzss0Gs", "reviewer": "Reviewer_gD8q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well written.\n2. The paper is quite convincing is showing that the results in the original DHM paper are likely not reproducible.\n3. The results in the paper are of interest to people working in OOD detection.\n\n## Weaknesses\n\n1. Reading the paper seems to convey the \"adversarial relationship\" between the feature extractor and the flow as a novel finding. It is well known in the generative modelling literature that simultaneously training autoencoders along with generative models in the corresponding latent space will typically exhibit these \"adversarial relationship\", see e.g. the discussion at the end of section 5.3 in [1] or section 3.3 in [2]. This is essentially the same phenomenon, and so exploring it in depth in the toy example seems slightly superfluous and should be better motivated, and wording which makes it sound like a novel finding should be toned down.\n2. Although the paper is mostly well written, I do think there a few improvements that could be made. I think the biggest thing is that the description of the flow in section 3 could be cleaner: please be more specific about what $f$ is in terms of the notation that you have already introduced. I think it should be $f=F \\circ \\phi$, which would then imply that $h$ has the same dimension as $x$ but as someone who was not familiar with DHMs, it was unclear if $h$ was lower-dimensional and ignored in the computation of the log determinant; it was also unclear to me how the log determinant is computed, since it has to be computed not only for the flow but also for the encoder. Lastly, small details:\n- Figure 1: please add function names to blocks ($\\phi$ to the encoder, $F$ to the flow, and $C$ to the classifier). Also, please make the box for $h$ the same size as the box for $x$ to convey they are of the same dimensionality.\n- Please use $\\partial$ instead of $\\delta$ in eq 2, it's much more standard notation.\n- Please be more precise when appropriate in figure and table captions as sometimes the results being shown can only be understood when reading the main text. For example, Table 1 does not mention the training set being used, figure 10 does not explain if it is showing results on toy or image data, table 2 does not even say what metric is being shown, etc.\n- The captions in figure 1 and table 2 are missing periods at the end.\n- The equations in the legend of figure 10 are highly uninformative.\n- Please use \\citet and \\citep appropriately in LaTeX, I found a few incorrect uses.\n- \"< <\" should be replaced by \"\\ll\" in LaTeX\n\n\n[1] Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections, Loaiza-Ganem et al., TMLR 2024\n\n[2] Generative Latent Flow, Xiao et al., arxiv 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Deep hybrid models (DHMs) are an approach to out-of-distribution (OOD) detection which simultaneously trains a feature encoder, along with a normalizing flow and a classification head on top of these features; the latter two can then be used for OOD detection. The original DHM paper reported perfect performance (AUC=1) at many common OOD tasks but provided no code. This paper deconstructs the original DHM paper: First, the authors attempt to reproduce results, and are unable to do so. Although the authors show that DHMs obtain highly competitive performance at OOD detection, they did not manage to obtain perfect AUC scores. The authors then create a toy dataset/OOD detection task to probe into the inner workings of DHMs, finding that the reason behind their good performance is likely not what was suggested in the original DHM paper. Along the way, the authors find that the performance of DHMs can be very sensitive to hyperparameter choices.\n\nOverall I think this is a good submission which fits the acceptance criteria for TMLR: the paper will be of interest to the OOD detection community, and the claims made in the paper are backed by evidence since the experiments are well executed and accompanying code is provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper revisits the Deep Hybrid Models (DHMs) approach for out-of-distribution (OOD) detection, originally introduced by Cao & Zhang (2022), by carefully reproducing and analyzing its claims.  Through careful reproduction and additional analysis, this work identifies critical implementation details that significantly affect model performance, providing valuable insights and public implementations beneficial to the OOD detection community. The reviewers broadly agree on the value of the careful experiments and analysis, despite minor concerns about the breadth of datasets used and the clarity.

For the final version, please make sure to update the link to the public code.
Decision: Accept (poster)