Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval
Authors: Frederik Rahbæk Warburg
Marco Miani
Silas Brack
Søren Hauberg
Venue: NeurIPS 2023 poster (2023)
Keywords: Laplace approximation
metric learning
uncertainty quantification
weight posterior
bayesian
TL;DR: Laplace approximation for metric learning

Abstract: We propose a Bayesian encoder for metric learning. Rather than relying on neural amortization as done in prior works, we learn a distribution over the network weights with the Laplace Approximation. We first prove that the contrastive loss is a negative log-likelihood on the spherical space. We propose three methods that ensure a positive definite covariance matrix. Lastly, we present a novel decomposition of the Generalized Gauss-Newton approximation. Empirically, we show that our Laplacian Metric Learner (LAM) yields well-calibrated uncertainties, reliably detects out-of-distribution examples, and has state-of-the-art predictive performance.

Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval
Frederik Warburg"
Marco Miani"
Technical University of Denmark frwaddtu.dd dk
Technical University of Denmark mmi amia@dtu.de
Silas Brack
Søren Hauberg
Technical University of Denmark si silasbrack@gmail. com
Technical University of Denmark sohau@dtu.dd
Abstract
We propose Bayesian encoder for metrie learning. Rather than relying neural amortization as done prior works. we learn a distribution over the network weights with the Laplace Approximation. We first prove that the contrastive loss is a negative log-likelihood on the spherical space. We propose three methods that en- sure positive definite covariance matrix. Lastly, present novel decomposition the Generalized Causs-Newton approximation. Empirically, we show that our Laplacian Metric Learner (LAM) yields well-calibrated uncertainties, reliably de- tects out-of-distribution examples, and has state-of-the-art predictive performance.
Introduction
Metric learning seeks data representations where similar observations are near and dis- similar ones are far. This elegantly allows for building retrieval systems with simple wearest-neighbor search. Such systems cas- ily cope with a large number of classes, and new classes organically be added with- out retraining. While these retrieval sys- tems show impressive performance, they
Uncertaitt
PFE
05
LAM
0.5
quickly, and with no raised alarms, dete-
riorate with out-of-distribution data [38]. In particular, in safety-critical applications, the lack of uncertainty estimation a con-
00
Uncoraaee
cern as retrieval eттотs may propagate un-
0.5
noticed through the system, resulting roneous and possibly dangerous decisions.
Figure 1: Reliable stochastic embeddings. LAM (ours) estimates reliable uncertainties of latent image embeddings that reflect the amount of blur, noise, or occlusion in the input image.
We present the Laplacian Metric Learner (LAM) to estimate reliable uncertainties of image embeddings as demonstrated in Fig. I. We learn distribution over the network weights (weight posterior) from
"qqual contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

which we obtain stochastic representation by embedding an image through sampled neural networks. This Bayesian formulation has multiple benefits, namely (1) robustness -ut-of-diitribution examples, (2) calibrated in-distribution uncertainties, and (3) slight improvement in predictive performance.
More specifically, our method extends the Laplace Approximation [27] for metric learning. We present a probabilistic interpretation of the contrastive loss [15] which justifies that it can be interpreted as an unnormalized negative log-likelihood. We propose three solutions to ensure a positive semidefinite Hessian for the contrastive loss and present two approaches to compute the Generalized Gauss- Newton [12] approximation for lg fg-normalized networks. Finally, we boost our method with the online training procedure from [29] to achieve state-of sate--of-thearre performance.
We are not the first to consider uncertainty quantification in image retrieval. Seminal works [38, 33] have addressed the lack of uncertainties in retrieval with amortized inference [14), where neural network predicts a stochastic embedding. The issues with this approach are that (1) it requires strong assumptions on the distribution of the embedding, (2) the networks are often brittle and difficult to optimize, and (3) out-of-distribution detection relies on the network's 's capacity to extrapolate uncertainties. As neural networks extrapolate poorly [50], the resulting predicted uncertainties are unreliable for out-of-distributio data [10] and are thus, in practice, of limited value.
In contrast, method does not assume any distribution on the stochastic embeddings, is simple to optimize, and does not rely on a neural network to extrapolate uncertainties. Instead, our weight posterior is derived from the curvature of the loss landscape and the uncertainties of the latent embed- dings deduced (rather than learned) with sampling. Empirically, we show that this leads to reliable diitii--ittiittioonn performance and calibrated uncertainties in both controlled toy experiments and challenging real-world applications such as bird, face, and place recognition.
2
Related Work
Metric learning attempts map data an embedding space, where similar data are close together and dissimilar data are far apart. This is especially useful for retrieval tasks with many classes and few observations per class such as place recognition [46] and face recognition [37] or for tasks where classes are not well-defined, such as food tastes [48] or narratives in online discussions [4].
There exist many metric losses that optimize for well-behaved embedding space. We refer to the excellent survey by [30] for an overview. We here focus on the contrastive loss [15]
Coon () = 2 [1f((x)) - fe(z,)) + 2 max m
-0 [ee( - fo(z))11))
(I)
which has shown ofate-of-the-art performance [30] and is one of the most commonly used metric losses. Here, So a neural network parametrized by which maps from the observation space to the embedding space. The loss consists of two terms, that attracts observations from the same class (anchor Va and positive zp), and one that repels observations from diffcrent classes (anchor To and negative x.)) The margin 2 0 ensures that negatives are repelled sufficiently far. We will later present a probabilistic extension of the contrastive loss that allows us to learn stochastic, rather than deterministic, features in the embedding space.
Uncertainty in deep learning studied across many domains to mitigate fatal accidents and allow for human intervention when neural networks make erroneous predictions. Current methods can be divided into methods that apply amortized optimization to train a neural network to predict the parameters of the output distribution, and methods that do not. The amortized methods, best known from the variational autoencoder (VAE) [19, 36), seem attractive at first as they can directly estimate the output distribution (without requiring sampling), but they suffer from mode collapse and are sensitive to out-of-distribution data due to the poor extrapolation capabilities of neural networks [31, 10]. 'Bayes by Backprop' [2] learns a distribution over parameters variationally but is often deemed too brittle for practical applications. Alternatives to amortized methods includes deep ensembles [23], stochastic weight averaging (SWAG) [28], Monte-Carle dropout [13] and Laplace Approximation (LA) [24, 27] which all approximate the generally intractable weight posterior p(0 D) of neural network. We propose extend LA metric learning.
Laplace approximations (LA) can be applied for every loss function L that can be interpreted as an unnormalized posterior by performing second-order Taylor expansion around chosen weight

---

vector 0* such that
=(0) +0 (0-) (0-00) + +2(0 -0-00) (00-0) 07);
where L* is the loss evaluated in 0". Imposing the unnormalized log-posterior be second-order polynomial equivalent to assuming the posterior to be Gaussian. If #" is a MAP estimate, the first-order term vanishes, and the second-order term can be interpreted inverse of the covariance. Assuming 0* is a MAP estimate, this second-order term is negative semi- definite for common (convex) supervised losses, such as the mean-squared error and cross-entropy. Recently, Daxberger et al. [8] demonstrated that post-hoc LA is scalable and produces well-behaved uncertainties for classification and regression. The Laplacian Autoencoder (LAE) [29] improves on the post-hoe LA with an online Monte Carlo EM training procedure to learn a well behaved posterior. demonstrates state-of-the-an uncertainty quantification for unsupervised representation learning.
precision matrix, the
Uncertainty in metric learning is not new [44), but the majority of recent methods apply amortized inference predict distributions in the embedding space [47, 3, 5, 33, 38, 39, 41], making them sen- sitive to mode collapse and out-of-distribution data. Alternatives like deep ensembles [42] and Monte- Carlo dropout [43] suffer from increased training time, poor empirical performance, and limited Bayesian interpretation [8]. We explore LA in metric learning and attain state-of-the performance.
3 Laplacian Metric Learning
To perform Bayesian retrieval, we estimate the weight posterior of the embedding network fe such that we can sample data embeddings to propagate uncertainty through the decision process. The embedding net- work is parametrized by A E θ and trained with the contrastive loss. The network maps an image E X' := R #WC to an embedding 2 E 2, which restricted be on Z-dimensional unit sphere 2 == S2, This spherical normalization is often done retrieval obtain faster retrieval and slight per- formance boost [1, 35]. Fig. 2 illustrate our Bayesian mapping from image to latent space.
fo
0 N(Ha, og)
Figure 2: Model overview. We learn a dis- tribution over parameters, such that we em- bed an image through sampled encoders fo to points z; (red dots) latent space Z. We re- duce these latent samples to a single measure of uncertainty by estimating the parameters of a von Fiises-Fisher distribution.
Laplace Approximation (LA) recap. We apply LA to obtain the posterior over the weights 0. LA comes two flavors: (1) The post-hoc LA is found by first training standard deterministic network through gradient steps with the contrastive loss to find the maximum posteriori (MAP) parameters 8" Since we are in a local optimum, the first-order term in the second-order Taylor expansion (Eq. 2) vanishes, and we can define the parameter distribution as
p(0ID) = NO00NNO0// (0for, (VoLom (0"; + Ppiar
The advantage of post-hoc LA is that the training procedure does not change, and already trained neural networks can be made Bayesian. In practice, however, we empirically observe post-hoc LA to be unstable. We refer to the large standard deviation in Table 1, Table and Table over random seeds. The instability stems from curvature differences between the local minimas through stochastic optimization. (2) Online LA [29] improves this instability by marginalizing the LA during training with Monte Carlo FMM. This helps the training recover a solution 0* where the Hessian reflects the loss landscape. Specifically, each step during training, we keep in memory Gaussian distribution on the parameters g (0) NO (0000 (r, H, The parameters are updated through an expected gradient step
Oc+1 = 02 + >B (VeCoon (0; D)]
(4)
and discounted Laplace update
Ha = (1 aa) + Vo.Con
(5)
where a describes an exponential moving average, similar to momentum-like training. initialization follows the isotropic prior q = (010, o 1).
The
3

---

The probabilistic contrastive likelihood. LA expects the loss function to be a log-posterior, that is log- loglikelihood plus log-priot, A simple choice of the log-rrior ||0| (weight decay). Further, we find that the contrastive loss Eq. 1 has a probabilistic interpretation and is a negative log-likelihood on the spherical space. We define the attractive term P- (z|r,, ()) ~ N/ (z) k) and the repelling term P=-(2lar 0) ~ N frrr k) as Von Mises-Fisher distributions on the latent space spherical space, where the concentration parameter x 2 0. The product of these two likelihoods yields contrastive likelihood, which is valid on the spherical space, and its negative log-likelihood is equivalent to the contrastive loss (proof and details in ??).
The intuitionn helped by electrostatics, is that the log probability density of these Von Mises-Fisher distributions (i.e. spherical Gaussians) plays the role of potential energy. The precision parameter controls the concentration of the potential energy and, consequently, the strength of the associated force. If K = 0 the distribution is uniform, the potential is constant and the associated force is zero. On the spherical space, moreover, repuisive term from some point is equivalent to an attractive term from the antipodal point, which in turn is mathematically equivalent (see ??) to a negative precision K. This is the fundamental reason behind the non positive-definitenss of the hessian.
Hessian of the contrastive loss. Both post-hoc and online LA require the Hessian of the contrastive loss Loon (0; The Hessian is commonly approximated with the Generalized Gauss-Newtor
(GGN) approximation [12. 8. 6, 11]. The GGN decomposes the loss into L g o f. where g is usually chosen as the loss function and f the model function, and only f is linearized [22].
However, in our case, this decomposition is non-trivial. Recall that the last layer of our network is an f2 normalization layer, which projects embeddings onto a hyper-sphere. This normalization layer can either be viewed as part of the model (linearized normalization layer) or part of the loss g (non-linearized normalization layer). The former can be interpreted as using the Euclidean distance and the latter as using the Arccos distance for the contrastive loss (see ?? and ??). These two share the zero- and first-order terms for normalized embeddings but not the second-order derivatives due to the GGN linearization. The Euclidean interpretation leads to simpler derivatives and interpretations, and we will therefore use it for our derivations. We emphasize that the Arccos is theoretically more accurate approximation, because the l2-layer is not linearized, and we provide derivations in ??.
The GGN matrix for contrastive loss with the Euclidean interpretation is given by
ViLcon (0:I) = -- HH Σ
H
H
ijEI
ijEI,
inEIn
(6)
GGN
Σ
G Mn
NEI
:= Hp
where = [Jnfala
with Jg is the Jacobian wrt. the parameters and H, and H, are the Hessians of the contrastive loss wrt. the model output for positive and negative pairs. The first sum runs over positive pairs and the second sum runs over negative pairs within the margin. Negative pairs outside the margin do not contribute to the Hessian, and can therefore be ignored to reduce the computational load (??).
The eigenvalues of the Hessian wrt. to the output are (0, and (-2, for the positive Hp and negative Hn terms, so we are not guaranteed have a positive semidefinite Hessian, Hg. To avoid covariances with negative eigenvalues, we propose three solutions to ensure a positive semidefinite Hessian. Proofs are ??.
Ensuring positive definiteness of the covariance matrix. We do not want to be restricted in the choice of the prior except have non-zero precision, so we must ensure that V Leonn000 DD) is positive semidefinite. Differently from the standard convex losses, this is not ensured by the GGN approximation [18]. Our main insight is that we can ensure positive semidefinite Hessian He by only manipulating the Hessians Hy and H, in Eq. 6.
1. Positive: The repelling term is ignored, such that only positive pairs contribute to the Hessian.
Hp (-1 -
0
00
0
H =
0
0

---

2. Fixed: The cross derivatives are ignored.
Hp ( 0 1
Hn = (-1 -i!
(8)
3. Full: Positive semidefiniteness is ensured with ReLU, max(0, V (0; D)), on the Hessian of the loss wrt. the parameters.
Hp - (
(1
Hn -
The positive approximation is inspired by [38], which only uses positive pairs train an uncertainty module. The gradient arrows in Fig. 3a illustrate that negative pairs are neglected when computing the Hessian of the contrastive loss. The fixed approximation considers one data point at a time, assuming the other one is fixed. Thus, given pair of data points, this can be interpreted as first moving one data point, and then the second (rather than both the same time). We formalize this in ??. Fig. 3b illustrate this idea when all points except a are fixed. Lastly, we propose the full Hessian of the contrastive loss (Fig. 3c) and ensure positive semidefiniteness by computing the ReI.U of the diagonal Hessian. This approximation can equivalently be interpreted matrixes. In practice, the Hessian scales quadratically in memory wrt. the number of parameters. To mitigate this, approximate this Hessian by its diagonal and only apply the LA on the last layer [25, 9]. We experimentally find that the fxxed approximation yields the best performance (Section 4).
projection into the space of psd
Hard negative mining. Most pairs, namely the negatives outside the margin, do not contribute to the Hessian, so it is wasteful compute their Hessian. There- fore, we use hard negative mining [30] to only compute the Hessian of pairs that have non-zero Hessian, i.e. the negative sample lie within the margin (illustrated with the dotted line in Fig. 3).
(e) posiiios
(b)
fized
(c) foll
Figure 3: Hessian approximations. To ensure pos- itive semidefinite Hessian approximation we propose three approximations.
Von Mises-Fisher distribution. To obtain a single measure of uncertainty from our sampled image embeddings (red dots Fig. 4), we fit von Mises-Fished distribu- tion. The von Mises-Fisher distribution describes normal distribution where all probability mass lies Z-dimensional hyper-sphere. It is parametrized with directional mean and scalar con- centration parameter K, which can be interpreted as the inverse of an isotropic covariance k = 1/o². i.e., small means high uncertainty and large means low uncertainty. There exist several methods to estimate K. We opt for the simplest and most computationally efficient [40] (see ??).
(a) only the positives p con- tribute to the Hessian as the negatives n are ignored. In (b) we consider one point at time, e.g. only the anchor a contributes. In (c) we consider all interactions.
4 Experiments
We benchmark our method against strong probabilistic retrieval models. Probabilistic Face Embed- dings (PFE) [38] and Hedge Image Embedding (HIB) [33] perform amortized inference and thus estimate the mean and variance of latent observation. We also compare against MC Dropout [13] and Deep Ensemble [23], two approximate Bayesian methods, which have successfully been applied in image retrieval [43, 42).
We compare the models' predictive performance with the recall (recall k) and mean average precision (mAP@ k) among the nearest neighbors [47, 30, 1]. We evaluate the models' abilities to interpolate and extrapolate uncertainties by measuring the Area Under the Sparsification Curve (AUSC), Expected Calibration Error (ECE) on in-distribution (ID) data, the Area Under Receiver Operator Curve (AUROC), and Area Under Precision Recaal Curve (AUPRC) ant-of-distribution (OoD) data. We provide more details on these metrics ??.
We extend StochMan [11] with the Hessian backpropagation for the contrastive loss, and the PyTorch [34] code publicly available2. ?? details the experimental setup.
"eee https: com http://www...ooo.eeeeeeettt/aaaa Learningg

---

CFSM
LAM Jan
Deep Cteamids
LAM koot
LAM jariines
LAM josohvad
po
MC arspedt
LAW Ooog
LAM gait
Delemasit
Dalom
HiC
MG Orig
Doep Groant
Loog Raaoontt
Coep Crosmclo
60
0.5
1.0
0.5
0.0
0.00
0.
0.22
AUNOC
AUTOO
AUTICO
TAng2
maigis
mAngo
(a) Out-of-distribution detection.
(b) Predictive performance.
Figure 4: Summary of experimental results. LAM consistently outperforms existing methods on OoD detection, measured by AUROC, and matches or surpasses in predictive performance measured by mAP@k. k. Error bars show one std across five runs.
Experimental Summary. We first summarize experimental results. Across five datasets, three network architectures, and three different sizes of the latent space (ranging from 3 to 2048), we find that LAM has well-calibrated uncertainties, reliably detects OoD examples, and achieves state-of-the- art predictive performance. Fig. 4a shows that the uncertainties from online LAM reliably identify OoD examples. Online LAM outperforms other Bayesian methods, such as post-hoc LAM and MC dropout, on this task, which in turn clearly improves upon amortized methods that rely on a neural network to extrapolate uncertainties. Fig. 4b shows that LAM consistently matches or outperforms existing image retrieval methods in terms of predictive performance. We find that the fixed Hessian approximation with the Arccos distance performs the best, especially on higher dimensional data.
Ablation: Positive definiteness co- variance matrix. We experimentally study which method to ensure a pos- itive semidefinite Hessian has the best performance measured in both pre- dictive performance (mAP@5) and uncertainty quantification (AUROC,
Table I: Ablation Hessian approximation and GGN decomposition. Online LA with the fixed approximation and Arccos distance performs best. Error bars show one std. across five runs.
mAP@5 f
AUROC
AUSC t
Euclidean fix
0.70 0.0
0.57 0.25
0.44 1 0.01
AUSC). We found that all methods perform similarly on simple datasets and low dimensional hyper-spheres, but the fixed approximation with Are- cos distance performs better on more challenging datasets and higher di- mensional hyper-spheres. We present
Post-hoc
Euclidean pos
0.70 0.0
0.52:
0.45 = 0.01
Euclidean full
0.70 0.0
0.26
0.44 0.01
Arccos fix
0.69 0.0
0.53
0.46 + 0.02
Arecos pos
0.70 0.0
0.11
0.48 = 0.01
Arecos full
0.0
0.55
0.45 0.01
Euclidean lix
0.00
0.07
0.31 = 0.02
Euclidean pos
0,70
0.38
0.47 - 0.01
Online
Euclidean full
0.67 1.01
0.59
0.42 = 0.01
results on one of these more challeng- ing datasets, namely the LFW [17]
Arccos fix
0.71 0.0
0.78 + 0.18
0.50 - 0.03
Arccos pos
0.70 +
0.23 0.03
0.46 - 0.00
face recognition dataset with the CUB200 [45] bird dataset as an OoD dataset. We use ResNet50 [16] with a GeM pooling layer [35] and 2048 dimensional embedding and diagonal, last-layer LA [8].
Arccos full
0.71 1 0.0
0.70 1 0.12
0.51 = 0.01
g
gos
0.5
0.0
1.0
1.0
PFE
(posthoc)
0.8
LAM (online)
Table 1 shows the performance for post-hoc and online LA with fixed, positive, or full Hessian approxima- tion using either Euclidean or Arccos distance. Across all metrics, the on- line LA with Arccos distance and the fixed Hessian approximation performs similarly or the best. We proceed to benchmark this method against sev- eral strong probabilistic baselines on closed-set set retrieval and a more challenging open-set retrieval.
Accurag
0.6
Best fit
0.6
0.2
0.2
0.0
0.0
0.0
0.4
1.0
Confidence
Confidence
(a) FashionMNIST
(b) CIFARIO
Figure Calibration Curves.
Closed-Set Retrieval. OoD capabilities are critical for identifying distributional shifts, outliers, and irregular user inputs, which might hinder the propagation of erroneous decisions an automated system. We evaluate OoD performance the commonly used benchmarks (3) 1],, where we use (1)

---

Table 2: Closed-set results. LAM matches outperforms existing methods in terms of predictive performance. It produces reliable uncertainties ID and OoD on two standard datasets FashionMNIST and CIFARIO. Error bars show one std. across five runs.
IMAGE RETRIBVAL mAP@5 1
OoD AUPRCT
CALIBRATION BCE
mAp@
11
AUROCT
AUSC
Deterministic
0.78 + 0.01
0.73 + 0.01
0.72 + 0.01
FushionMNIST
Deep Ensemble
0.69
0.62
0.59
0.41
0.46
0.61
0.04
PFE
0.78 0.00
0.74 174000000 0.00
0.72 0.00
0.53 : 0.03
0.46 0.01
0.65 = 0.01
0.26 0.02
HIB
0.69 0.08
0.63 0.09
0.61 0.09
0.60 0.12
0.60 0.11
0.65 0,08
0.54 0.08
MC dropout
0.76 + 0.03
0.71 + 0.03
0,70 +: 0.03
0.93 + 0.03
0.93 + 0.03
0.84 = 0.06
0.03
0.04
LAM (post-hoc)
0.78 + 0.00
0
0.72 0.00
0.96 1 0.02
0.96 + 0.02
0.86 0.01
0.03
0.00
LAM (online)
0.00
0.01
0.76
0.98
0.01
0.01
0.02
Deterministic
0.66 0.00
0.59 0.00
0.58
Deep Ensemble
0.66
0.61
0.59
0.42
0.67
0.72
0,02
CIFARIO
MC dropout
0.46 - 0.01
0.37 0.01
0.04
0.60 0.03
0.76 1 0.02
0.61 = 0.01
0.05 0.00
HIB
0.11 0.01
0.05 0.07 0.00
0.44
0.70
0.29 0.03
0.04 0.02
PFE
0.66 + 0.00
0.60 0.00
0.00
0.56 0.22
0.56 0.01
0.11 0.01
LAM (past-hoc)
0.66 +: 0.00
0.60 +
0.00
0.50
0.09
0.81 = 0.01
0.23 0.01
LAM (online)
0.66 0.01
1: 0.00
0.01
0.78 0.04
0.85 = 0.03
0.83 = 0.011
0.01 0.00
FashionMNIST [49] ID and MNIST [26] as OoD, and (2) CIFARIO [21] ID and SVHN [32] as OoD. We use, respectively, standard 2- or 3-layer relu convolutional network followed by single linear layer on which we compute LA with diagonal Hessian.
FashionMNIST (ID) vs MNIST (OoD). Table 2 shows that both PFE and post- hoc LAM have similar predictive perfor- mance to the deterministic model. This is not surprising. as both methods are initial- ized with the deterministic parameters, and then uncertainties are learned (PFE) or de- duced (post-hoc LAM) with frozen weights. The awareness of uncertainties during train- ing, grants the online LAM slightly higher predictive performance.
1.0
LAM (posthoc) lonline)
0.4
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
TPR
TPR
(a) FashionMNIST
(b) CIFARIO
Figure 6: Receiver Operator Curves. LAM assign high uncertainty to OoD observations.
PFE uses amortized inference to predict variances. This works reasonably within distribution but does not work well for OoD detection. This is because a neural network is trusted to extrapolate far away from the data distribution. Table shows that MC dropout, LAM (post-hoc), and LAM (online) assign high uncertainty to observations outside the training distribution. Fig. 5 shows that both post-hoe and online LAM are near perfectly calibrated, giving very low ECE measures (Table 2).
CIFARIO (ID) vs SVHN (OoD) is a slightly harder setting. Table 2 yields similar conclusions as before: Bayesian approaches such as MC dropout, LAM (post-hoc), and LAM (online) better detect OoD examples than neural amortized methods such as PFE. Online LAM has a similar predictive performance to ofte-of-the-ard while having better ID (lower ECE and higher AUSC) and OoD (higher AUROC and AUPRC) performance. Fig. 5 shows the calibration plot for CIFAR1O, where online LAM has near- perrect calibration. The CIFAR10 ROC curves (Fig. 6) show that online LAM is better at distinguishing ID and OOD examples.
Open-Set Retrieval. A key advantage of metric learning methods is that they easily cope with large number of classes and new classes can be added seamlessly. We therefore evaluate LAM's performance on challenging open-set retrieval, where none of the classes in the test set are available during training. We first test with CUB200 [45] as ID and CAR [20] as OoD similarly to Warburg et al. [47], and second, test with LFW [17] as ID and CUB200 as OOD. We use ResNet50 [16] with a GeM pooling layer [35] and a 2048 dimensional embedding and diagonal, last-layer LA [8].
CUB200 (ID) CARS196 (OoD). The CUB-200-2011 dataset [45] has 200 bird species captured from different perspectives and in different environments. We follow the zero-shot train/test split [30]. In this zero-shot setting, the trained models have not seen any of the bird species in the test set, and the learned features must generalize well across species. Table shows that LAM matches or surpasses the predictive performance of all other methods. LAM (post-hoc) achieves ofte-of-the-ar
7

---

Table 3: Open-set results. LAM matches outperforms existing methods in terms of predictive performance and produces state-of- of-the-art uncertainty quantification for challenging zero-shot metric learning datasets LFW and CUB200. Error bars show one std. across five runs.
IMAGE RETRIEVAL
OoD
ID AUSC
mAP@ 1
MAP@5
mAP@101
AUROC1
AUPRC1
Deterministic
0.62 0.01
0.48 0.01
0.42 0.01
Deep Ensemble
0.21
0.11
0.07
0.47
0.55
0.21
CUB200
0.62 0.01
0.01
0
0.16
0.5 0.08
0.61 + 0.02
0.33 0.04
0.19 = 0.02
0.14
0.12
0.61
0.31 + 0.07
MC dropout
0.61 0.00
0.48 + 0.00
0.42
0.73
0.08
0.68 0.07
0.63 + 0.01
LAM (post-hoc)
0.65 0.01
0.52 + 0.01
0.45 0.01
0.56
+.1.-0166
0.61 0.11
0.66 + 0.03
LAM (online)
0.61 0.00
0.48 + 0.00
0.42 + 0.00
0.03 0.80
0.75 0.03
0.63 + 0.01
Deterministic
0.44 0.00
0.68 + 0.00
0.65 0.00
Deep Ensemble
0.36
0.57
0.54
0.52
0.64
0.33
LFW
PFE
0.44 0.00
0.68 0.00
0.65 0.00
0.03 0.02
0.0
0.49 0.01
MC dropout
0.42 0.00
0.65 0.01
0.63 +63+0.00
0.01
+..
0.46 + 0.01
LAM (post-hoe)
0.44
0.68 0.01
0.65 0.00
0.65
0.12
0.45 + 0.03
LAM (online)
0.46 0.00
0.71 0.00
0.69 0.00
0.71 0.22
0.78
0.50 + 0.02
PFE
LAM (A) (p)
HE
Lower uncertainty
Higher uncertainty
Figure 7: Images with lowest and highest variance for PFE, post-hoc LAM, and online LAM across LFW (ID) and CUB200 (OoD) datasets. LAM associates high uncertainty to OoD examples, and vice versa for PFE. Shows the best-performing PFE and online LAM across five runs.
predictive performance, while LAM (online) matches the predictive performance of the deterministic trained model while achieving state-of-the-aat AUROC and AUPRC for OoD detection.
LFW (ID) vs CUB200 (OoD). Face recognition is another challenging metric learning task with many applications security and surveillance. The goal is retrieve images of the same person as in the query image. Table 3 shows that online LAM outperforms existing methods both in terms of predictive performance and uncertainty quantification. Fig. shows that PFE assigns higher uncertainty to images from the ID dataset (faces) than those from the OoD dataset (birds). In contrast, both online and post-hoc LAM better asso- ciate high variance to OoD examples, while PFE predicts high variance ID examples. Furthermore, online LAM seems to assign the highest variance to images in which the background complex and thus camouflages the birds.
0.8
PFE
MC Dropout
mAP@1
0.7
LAM Posthoc
LAM Online
Visual Place recognition is important for the long- term operation of autonomous robots [7], where the goal is to retrieve images taken within a radius of 25 meters from a query image. The high number of unique places and varying visual appearance of each lo- cation including weather, dynamic, structural, view- point, seasonal, and day/night changes makes sual place recognition a challenging metric learning
0
25
50
75
100
Filter Out Rate
Figure 8: Sparsification curve. Online and post-hoc LAM's sparsification curves monotonically increase, showing that they reliably associate higher uncertainty to harder observations.

---

Table 5: Results MSLS. LAM yields state-of-the-ar uncertainties and matches the predictive performance of deterministic trained models.
Validation Set
Challenge Set
R@11
R
R@:10
MG:5
M6
AUSC1
R@11
R@51
R 101
M@.5
M@:
AUSCT
Deterministic
0.77
0.88
0.90
0.61
0.56
0.58
0.74
0.78
0.45
0.43
MC. Dropout
0.75
0.87
0.87
0.59
0.54
0.77
0.55
0.76
0.43
0.41
0.57
PTE
0.77
0.88
0.90
0.61
0.56
0.73
0.58
0.74
0.78
0.45
0.44
0.57
LAM (post-boc)
0.76
0.86
0.89
0.60
0.55
0.74
0.58
0.74
0.78
0.45
0.44
0.59
LAM (onlinc)
0.76
0.90
0.57
0.74
0.78
LAM LAM (p) PrB
Lower uncertainty
Higher uncertainty
Figure 9: Images with lowest and highest variance for PFE, post-hoc LAM, and online LAM across MSLS validation set. LAM reliably associates high uncertainty to images that blurry, captured facing the pavement, or contain vegetation. These images do not contain features that are descriptive of specific place, making them especially challenging geographically locate.
problem. Reliable uncertainties and reliable out-of-distribution behavior are important avoid incorrect loop-closure, which can deteriorate the autonomous robots' location estimate. We evaluate on MSLS [46], which is the largest and most diverse place recognition dataset currently available comprised of 1.6. images from 30 cities spanning six continents. We use the standard
train/test split, training on 24 cities and testing on six other cities. open-set retrieval.
We use the same model in
Margin
Table 5 shows that online LAM yields Mate-of-the-aal uncertain- ties for visual place recognition measured with AUSC, while matching the predictive performance of the alternative probabilis- tic and deterministic methods on both the MSLS validation and the challenge set. Fig. shows the sparsification curves the challenge set. Both online and post-hoe LAM have monotonically increasing sparsification curves, implying that when we remove the most uncertain observations, the predictive performance in- crease. This illustrates that LAM produces reliable uncertainties for this challenging open-set retrieval task. Fig. 9 shows the queries associated with the highest and lowest uncertainty. LAM predicts high uncertainty to images with are blurry, captured facing into the pavement, or contain mostly vegetation. These images do not have features that are descriptive of specific place, making them hard to geographically locate.
mAP@5
AUROC
AUSC
0.1
0.71
1.00
0.50
0.71
0.57
0.49
0.70
0.16
0.48
0.7
0.70
0.09
0.48
Number of pairs per batch
mAP@5
AUROC
AUSC
0.72
0.99
0.49
0.72
0.98
0.50
0.71
1.00
0.49
0.72
1.00
0.49
Latent dimension
mAP@5
AUROC
AUSC
128
0.71
0.99
0.47
256
1.00
512
0.71
1.00
0.50
2048
0.72
1.00
0.52
Memory factor
Ablations. We conduct ablations on LFW for online and post-hoc LAM. Table shows that the predictive performance does not depend on the margin, however, OOD performance decreases significantly for larger margins. We find that LAM robust the number of pairs sampled per batch. We find that larger latent space results better-calibrated uncertainties. Online LAM is rather robust to the choice of memory factor a. However, choos- ing a memory factor in the 0.0001-0.001 range gives both good AUROC and AUSC. This memory factor a can be interpreted as a momentum-like parameter that relates to the learning rate and indicates how fast we should update the current Hessian [29].
mAP@5
AUROC
AUSC
0.1
0.70
1.00
0.41
0.01
0.71
0.46
0.001
0.72
1.00
0.48
0.0001
0.72
0.98
0.50
0.00001
0.72
0.87
0.51
Post-hoe tempering 3
mAP@5
AUROC
AUSC
100
0.67
0.10
0.48
0.69
0.68
0.46
0.69
0.86
0.44
0.1
0.69
0.87
0.43
0.01
0.001
0.69
0.86
0.43
Table 4: Ablation on LFW.

---

Lastly, we perform cold-posterior tempering of post-hoc LA by scaling the log- log-likelihood hessian with a factor 3 > 0, such that the precision in Eq. became B Vigcon (0"; D) + a I I. We do not see any benefits from changing to other values than one.
Limitations. Similar to other Bayesian methods, LAM relies n samples to obtain uncertainties. This makes inferences n times slower. Computing the Hessian at every step during online LAM also makes training time slower. To combat long training times and high memory usage, we use a last-layer LAM and thus only estimate and sample for weight posterior of the last layer. The last-layer LAM training time is 3 hours for online LAM vs 2.3 hours for deterministic contrastive loss on LFW, and 30 minutes vs 15 minutes loss on CUB200 on an NVIDIA RTX A5000.
5 Conclusion
In this paper, we have introduced a Bayesian encoder for metric learning, the Laplacian Metric Learner (LAM), which uses the Laplace approximation. We prove that the contrastive loss is unnormalized negative log-likelihood on the spherical space, and develop three Hessian approximations, which ensures positive definite covariance matrix. We propose novel decomposition of the Generalized Gauss-Newton approximation that improves Hessian approximations of f fz-normalized networks. Empirically. we demonstrate that LAM consistently produces well-calibrated uncertainties, reliably detects out-of of-ooirrriibution examples, and achieves stute-of-the-eee predictive performance on both closed-set and challenging open-set image retrieval tasks.
Acknowledgement. This work was supported by research grants (15334, 42062) from VILLUM
FONDEN. This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 757360). The work was partly funded by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF200C0062606),
References
[1] Relja Arandjelović, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5297-5307, 2016
[2] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty neural networks, 2015.
[3] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, pages 5710-5719, 2020.
[4] Peter Ebert Christensen, Frederik Warburg, Menglin Jia, and Serge Belongie. Searching for structure in unfalsifiable claims. arXiv preprint arXiv:2209.0044 2022.
[5] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8415 8424, 2021.
[6] Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into backprop. In International Conference on Learning Representutions (ICLR), 2020.
[7] Andrew J. Davison, Ian D. Reid, Nicholas D. Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE Transactions Pattern Analysis and Machine Intelligence, 29(6): 1052-1067, 2007. doi: 10. 00.1109//PAMM..000..000
[8] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-efforlees Bayesian deep learning. In NeurlPS, 2021.
[9] John Denker and Yann LeCun. Transforming neural-net output levels to probability distributions. Advances in Neural Information Processing Systems, 1990.
10

---

[10] Nicki S Detlefsen, Martin Jørgensen, and Søren Hauberg. Reliable training and estimation of variance networks. In Advances Neural Information Processing Systems, 2019.
[11] Nicki S. Detlefsen, Alison Pouplin, Cilie W. Feldager, Cong Geng, Dimitris Kalatzis, He- lene Hauschultz, Miguel González Duque, Frederik Warburg, Marco Miani, and Søren Hauberg. StochMan, 2021. URL wwt.h:ttp//// stochman/.
hattts://githubb. htt///gtttttt...teee htt.sat..cc///tiiiillammmmmnnneo
[12] Dan Foresee and Martin T Hagan. Gauss-Newton approximation to Bayesian learning. In Proceedings of International Conference Neural Networks (ICNN'97), volume 3, pages 1930-1935. IRHE, 1997.
[13] Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty deep learning. international conference on machine learning, pages 1050-1059. PMLR, 2016.
[14] Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science society, volume 36, 2014.
[15] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'06) 06), volume pages 1735-1742. IEEE, 2006.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[17] Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Leamed-Miller. Labeled faces in the wild: A database for studying face recognition unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
[18] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. In International Conference on Artificial Intelligence and Statistics, pages 703-711. PMLR, 2021.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 1311v 14, 2013.
[20] Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference Computer Vision Workshops, pages 554-561, 2013.
[21] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
[22] Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
[23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[24] Pierre Simon Laplace. Mémoire sur la probabilité des causes par les événements. Mémoire de I' l'Académie Royale des Sciences, 1774.
[25] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in Neural Information Processing Systems, 1989.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 96(11):227-2222 1998.
[27] David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, ((3):4484774 1992.

---

[28] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. simple baseline for Bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[29] Marco Miani, Frederik Warburg, Pablo Moreno-Muñoz, Nicke Skafte Detlefsen, and Søren Hauberg. Laplacian autoencoders for learning stochastic representations. In Advances in Neural Information Processing Systems, 2022.
[30] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision, pages 681-699. Springer. 2020.
[31] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv: 181009916.. 2018.
[32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning, 2011.
[33] Seong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. Modeling uncertainty with hedged instance embedding. arXiv preprint arXiv:1810.00319, 2018.
[34] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch In NIPS-W- W, 2017.
[35] Filip Radenović, Giorgos Tolias, and Ondfejj Chum. Fine-tuning CNN image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7): 1655-1668, 2018
[36] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models, In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[37] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In Proceedings the and Pattern Recognition, pages 823, 2015.
the IEEE Conference Computer Vision
[38] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6902-6911, 2019.
[39] Yale Song and Mohammad Soleymani. Polysemous -emantic embedding for cross- modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1979-1999999 1988, 2019.
[40] Suvrit Sra. A short note parameter approximation for von Mises-Fisher distributions: and fast implementation of i s (x). Computational Statistics, 27(1):177-190, 2012.
[41] Jennifer J Sun, Jiaping Zhao, Liang-Chich Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View-invariant probabilistic embedding for human pose. In European Conference on Computer Vision, pages 53-70. Springer, 2020.
[42] Ahmed Taha, Yi-Ting Chen, Teruhisa Misu, Abhinav Shrivastava, and Larry Davis. Unsuper- vised data uncertainty learning in visual retrieval systems. arXiv preprint arXiv: arXiv:1902. 1902.02800..8 2019.
[43] Ahmed Taha, Yi-Ting Chen, Xitong Yang, Teruhisa Misu, and Larry Davis. Exploring uncer- tainty in conditional multi-modal retrieval systems. arXiv preprint arXiv:1901.07703 2019.
[44] Luke Vilnis and Andrew McCallum. Word representations via Gaussian embedding. arXiv preprint arXiv: /4/ 2014.
[45] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie, Caltech- UCSD birds 200, 1.
12

---

[46] Frederik Warburg, Søren Hauberg, Manuel Lopez-Antequera. Pau Gargallo, Yubin Kuang, and Javier Civera. Mapillary street-level sequences: A dataset for lifelong place recognition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR),
June 2020.
[47] Frederik Warburg, Martin Jørgensen, Javier Civera, and Søren Hauberg. Bayesian triplet loss: Uncertainty quantification in image retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12158-12168, 2021.
[48] Michael Wilber, Iljung S Kwak, David Kriegman, and Serge Belongie. Learning concept embeddings with combined human-machine expertise. In Proceedings of the IEEE International Conference on Computer Vision, pages 981 981-989, 2015.
[49] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms, 2017.
[50] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks, 2021.
13


Reviews:
[{"review_id": "r47S8rD0zR", "reviewer": "Reviewer_SAQp", "summary": "They propose a Bayesian encoder for metric learning. They learn a distribution over the network weights with the Laplace approximation. They first prove that the contrastive loss is a negative log-likelihood on the spherical space. They propose three methods that ensure a positive definitive covariance matrix. They present a novel decomposition of the Generalized Gauss-Newton approximation. \nThe empirical results leads previous methods on OOD examples.", "questions": "1. Could you provide a systematic analysis of the computational efficiency?\n2. Proxy-anchor loss can be a good case to study with this uncertainty measurement.", "limitations": "yes, they have addressed the limitations of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well organized paper and good writing. Clearly presenting the idea. Code released.\n- The bayesian approach for uncertainty measurement in metric learning is interesting.\n- The method show improved results, especially on OOD examples, and ablation study is comprehensive.", "weaknesses": "- First the experimental results only achieved limited improvement.\n- As the paper claimed, The method is slow in computation as it is a bayesian method.\n- The paper only focus on a classical contrastive loss which is a margin-based loss, how about other cases? for example, the proxy-based losses?\n- Overall the performance improvement is not significant, and the computational efficiency is not competitive, compared with other uncertainty method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  First the experimental results only achieved limited improvement.\n\n**Strong UQ performance:** The reviewer is correct that the predictive performance does only improve slightly upon the baselines. However, we highlight that the OOD performance (the focus of the paper) across all 4 datasets improves significantly, e.g. for LFW, Deep Ensemble, PFE, MC dropout has 0.52, 0.03, 0.03 AUROC (recall 0.5 is a random baseline), whereas LAM (post-hoc) and LAM (online) yield 0.65 and 0.71 AUROC. Across all datasets, similar, large performance improvements are observed for uncertainty metrics on OOD and ID.  \n\n>  As the paper claimed, The method is slow in computation as it is a bayesian method.\n\n**Computational overhead:** It is true that our approach comes with a higher computational overhead than a standard feedforward network, but it also produces more information: a useful estimate of uncertainty. Depending on the application, this trade-off can be very worth making.\n\nAt inference time, we require N forward passes, thus requiring N times more compute than deterministic methods. In practice, however, these N forward passes can easily be parallelized, such that no time overhead might be observed.\n\n>  The paper only focus on a classical contrastive loss which is a margin-based loss, how about other cases? for example, the proxy-based losses?\n\n**Extension to other losses:** The focus of the paper is the contrastive loss, a very common loss in metric learning, which has been shown to perform on par with newer, more sophisticated losses [Metric Learning a Reality Check, Musgrave et al.]. We believe that the method is applicable to the proxy-anchor loss, and this it is an interesting direction to explore in future work. \n\n> Overall the performance improvement is not significant, and the computational efficiency is not competitive, compared with other uncertainty method.\n\nWe do not agree. See [Strong UQ performance] above.\n\n>  Questions:\n>  Could you provide a systematic analysis of the computational efficiency?\n\n[see above]\n\n> Proxy-anchor loss can be a good case to study with this uncertainty measurement.\n\n[see above]"}, {"review_id": "ZsWTkOCAJp", "reviewer": "Reviewer_Z8Se", "summary": "\nThe authors show that contrastive loss can be viewed as a likelihood after projection onto the spherical space. This consequently allows them to use the Laplace approximation to estimate the posterior over the parameters. To make the construction further amenable to estimation, the authors propose approaches to make the Hessian positive definite. Finally, the experimental results and ablations go into details about the design choices.", "questions": "1. In Line 46, the authors claim that they do not assume any distribution on the stochastic embeddings. But the choice of using Laplacian approximation on the parameters to construct the posterior implicitly makes an assumption on the constructed embeddings. Could the authors clarify this?\n2. As stated in Eq. (2), the Laplace approximation comes from a second-order Taylor expansion, where the first-order gradient vanishes due to $\\theta^\\star$ being the optima. Does this does really happen in practice, and how numerically close are we to zero for instance w.r.t. the size of models/data?\n3. Could the authors confirm if my understanding of the overall approach is correct? (a) Using online LA to construct the posterior (b) Use samples from the posterior to construct stochastic embeddings (c) Use embedding samples to construct $\\kappa$ for the von Mises-Fisher distribution which is used as a single number to quantify uncertainty.", "limitations": "Yes. See also weaknesses and questions.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The authors use a conceptually simple approach to constructing approximate posteriors on top of neural networks using the Laplace approximation.\n- The authors provide several analogies for various concepts introduced which makes the reader comfortable, and would be resourceful paper for the community.\n- Most design choices in the method are covered by ablations, which is much appreciated.", "weaknesses": "- The Laplace approximation hinges on an assumption of vanishing gradient which may not be true for modern large neural networks and large datasets used for representation learning. See Question 2.\n- The overall method involves a fair number of moving parts, and it would be good to reconcile them as a single algorithm or a list of bullet points for easy digestion for the reader.\n- The method can be computationally expensive, and the authors have to resort to construct stochasticity of the last-layer parameters only. Much of the post-hoc LA literature seems to be working with a similar setup, so I do not count this as a major weakness of this paper but rather the whole community.\n\n### Minor\n\n- Please use `\\citet` instead of `\\citep` for references directly referring to the paper, for instance in Line 164.\n- It would be great to have Figure 3 on the same page as the text on ensuring positive definiteness on Page 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The overall method involves a fair number of moving parts, and it would be good to reconcile them as a single algorithm or a list of bullet points for easy digestion for the reader.\n\nWe thank the reviewer for the suggestions and will include the following snippets of pseudo-code in the paper.\n```python\ndef train(x, y, hessian, prior_prec):\n\t# x is the input data\n\t# y is the target data\n\t# parameters is the network parameters\n\t# hessian is hessian of the network\n\t# prior_prec is prior precision. We set it to 1\n\n\tmu_q = parameters\n\tsigma_q = 1 / (hessian + prior_prec)\n\tnetwork_samples = sample_from_normal(mu_q, sigma_q)\n\tfor sample in network_samples:\n\t\temb = model(x, sample)\n\t\tpairs = miner(y)\n\t\tloss += contrastive_loss(emb, pairs)\n\t\thessian_batch += hessian_calculator(x, pairs)\n\t\n\tloss =/ len(network_samples)\n\thessian_batch =/ len(network_samples)\n\thessian = alpha * hessian + hessian_batch\n```\n```python\ndef inference(x, parameters, hessian):\n\t# x is the input data\n\t# parameters is the network parameters\n\t# hessian is hessian of the network\n\t# prior_prec is prior precision. We set it to 1\n\n\tmu_q = parameters\n\tsigma_q = 1 / (hessian + prior_prec)\n\tnetwork_samples = sample_from_normal(mu_q, sigma_q)\n\temb = []\n\tfor sample in network_samples:\n\t\temb.append(model(x, sample))\n\tmu,  sigma = vmf_from_samples(emb)\n```\nWe hope this will improve the clarity of the presentation.\n\n> The method can be computationally expensive, and the authors have to resort to construct stochasticity of the last-layer parameters only. Much of the post-hoc LA literature seems to be working with a similar setup, so I do not count this as a major weakness of this paper but rather the whole community.\n\nBayesian deep learning builds on approximations: We agree with the reviewer that the Bayesian literature tries to approximate the posterior in various ways to come up with computationally feasible and useful methods. We follow common practices in LA and rely on a last-layer, diagonal assumption.\n\n>Minor\n> Please use \\citet instead of \\citep for references directly referring to the paper, for instance in Line 164.\n> It would be great to have Figure 3 on the same page as the text on ensuring positive definiteness on Page 4.\n\nWe thank the reviewer for the suggestions to improve clarity and have updated the paper accordingly.\n\n>Questions:\n> In Line 46, the authors claim that they do not assume any distribution on the stochastic embeddings. But the choice of using Laplacian approximation on the parameters to construct the posterior implicitly makes an assumption on the constructed embeddings. Could the authors clarify this?\n\nThe reviewer is right that the Laplace approximation implicitly puts some assumptions on the embedding distribution. Note that these assumptions depend on the choice of the neural network architecture. So the proper statement would be “the Laplace approximation, when tied with a specific neural network architecture, implicitly puts some assumptions on the embedding distribution”. This is exactly what we observe in our case by only considering architecture with a normalization layer at the end: such architecture choice resulted in the implicit constraint of embedding distribution supported on the sphere.\n\nImportantly, no assumptions are present before conditioning on the network architecture. Our text was meant to contrast with existing variational methods that make the Gaussian assumption over the embeddings. Our model makes no such explicit choice, and different instances of the method will result in different assumptions. In principle, with a sufficiently flexible network architecture, it should be possible to obtain any embedding distribution from the Laplace approximation (in a similar spirit to change-of-variables in normalizing flows). In practice, we do observe non-Gaussian unimodal embedding distributions.\n\n> As stated in Eq. (2), the Laplace approximation comes from a second-order Taylor expansion, where the first-order gradient vanishes due to being the optima. Does this does really happen in practice, and how numerically close are we to zero for instance w.r.t. the size of models/data?\n\nThe reviewer is right that the gradient is usually ignored in the Laplace approximation, and the weight posterior is sampled from N(mu, H^{-1}). We experimentally find that if we do not ignore the gradient term, and instead sample from N(mu + grad * H^{-1}, H^{-1}), then we get similar results:\n\n|   |  map@1  |  map@5 |  map@10  |  auroc  | auprc | ausc |\n|---|---|---|---|---|---|---|\n| N(mu, H^{-1})  | 0.46 | 0.72 | 0.70 | 0.99 | 1.00 | 0.50 |\n| N(mu + grad * H^{-1}, H^{-1})  | 0.46 | 0.72 | 0.70 | 0.99 | 1.00 | 0.50 |\n\n\nHere, grad is the average gradient over the dataset estimated from one epoch. These results are obtained from LFW, similar to the rest of the ablation studies. We will include it in the ablation table. To answer, how numerically close the gradients are to zero, we provide the min and max of the average gradient: min = -0.0008, max = 0.0008. We believe that the Table and the absolute values of the gradients suggest that it is reasonable to ignore the first-order term. \n\n> Could the authors confirm if my understanding of the overall approach is correct? (a) Using online LA to construct the posterior (b) Use samples from the posterior to construct stochastic embeddings (c) Use embedding samples to construct for the von Mises-Fisher distribution which is used as a single number to quantify uncertainty.\n\nYour understanding is correct. However, note that the last step [fitting the vMF distribution] is optional, e.g. maybe you do not need a single measure of uncertainty but would prefer to work with the samples directly.\n"}, {"review_id": "LypaZzqVPn", "reviewer": "Reviewer_Stu9", "summary": "This paper presents a Laplace approximation-based probabilistic retrieval approach (aka. Bayesian metric learning for image retrieval). The author provides a probabilistic view of the contrastive loss based on the von-Mises Fisher distribution and corrections for the Hessian positive definiteness. Extensive experimental evaluations are performed, demonstrating the advantages of the presented approach in calibrated uncertainties, out-of-distribution detection, close-set and open-set retrieval, and ablations on a few parameters of design choices.", "questions": "The probabilistic view in the supplemental also seems a little bit weird to me since both argument and parameter are data-dependent. Would it be sufficient to just treat the probabilistic contrastive likelihood as a second-order differentiable loss, if they are mathematically equivalent anyway?  \n\nWhat are the main roadblocks that have been overcome in applying the Laplacian autoencoder (LAE) to probabilistic retrieval?\n\nAre there any other ways of approximating Hessian for contrastive loss, while still maintains scalability and positive definiteness? Is KFAC applicable, e.g., Ritter et al., 2018? \n\nRitter, H., Botev, A., & Barber, D. (2018, January). A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings (Vol. 6). International Conference on Representation Learning.", "limitations": "The authors discussed limitations about computation load which I think is reasonable. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "[Originality] This paper presents the use of Laplace approximation instead of amortized inference for probabilistic retrieval. Although similar ideas have been proposed to improve the amortization gap of variational autoencoders in several prior works, its development in the context of metric learning is novel. This includes the introduction of a probabilistic view of the contrastive loss and the utilization of the Hessian approximation based on GCN.\n\nOn the other hand, previous Bayesian metric learning approaches have also been proposed, such as hedge instance embedding (HIE) and probabilistic face embeddings (PFE). In this regard, the novelty of this work lies in the contrastive learning-based loss and the learning of stochastic embeddings based on the Laplacian autoencoder (Miani, M. et al., 2022).\n\nMiani, M., Warburg, F., Moreno-Muñoz, P., Skafte, N., & Hauberg, S. (2022). Laplacian autoencoders for learning stochastic representations. *Advances in Neural Information Processing Systems*, *35*, 21059-21072.\n\n[Quality] Extensive empirical evaluations, including a careful ablation study, are conducted. The results demonstrate that the proposed approach outperforms HIE and PFE in the considered cases. \n\n[Clarity] The paper effectively uses figures to demonstrate ideas. However, I do find the need of reading with jumps between the main text and the supplemental material in order to grasp the concept. The organization can be improved. \n\n[Significance] This paper demonstrates uncertainty quantification based on Laplace approximation in the context of probabilistic retrieval. This is an important topic for improving the robustness and mitigating the silent failure of deep neural network systems. The main contribution of this paper appears to be the empirical study, which can be informative and useful for practitioners.", "weaknesses": "It appears that this paper builds upon prior work, including the Laplacian autoencoder (Miani, M. et al., 2022), as well as several works on uncertainty in metric learning. Consequently, I am more concerned about the unique technical contributions of this work. In this regard, the necessity of the probabilistic view of the contrastive loss is not well presented and not well motivated.\n\n Another missing piece is how to perform test-time retrieval within the proposed framework and how its efficiency compares to the amortized approach. Given a query image, does the proposed approach learn a stochastic representation for it? Is the retrieval based on ranking the deterministic similarities between the query image and all other candidate images? If not, how is it performed? \n\nThe organization and clarity of this paper could be improved. If certain results are deemed important enough, they should be moved into the main text. The authors should focus on introducing the unique aspects of this work and provide a strong motivation for them, such as the Von Mises-Fisher distribution. The frequent references to the Appendix disrupt the logical flow of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It appears that this paper builds upon prior work, including the Laplacian autoencoder (Miani, M. et al., 2022), as well as several works on uncertainty in metric learning. Consequently, I am more concerned about the unique technical contributions of this work. In this regard, the necessity of the probabilistic view of the contrastive loss is not well presented and not well motivated.\n\nThe technical contribution: The paper proposes to use Laplace for metric learning. This requires (1) proving that the contrastive loss is a log-likelihood because this is a fundamental assumption of the Laplace approximation. This provides the probabilistic motivation that the reviewer request. (2) Identify approximations of the second-order derivative of the loss that ensures that it is semi-positive definite because otherwise, one might end up with a covariance matrix with negative variances. (3) A novel decomposition of the GGN approximation, and (4) “putting things together”, which is all too often trivialized. \n\n> Another missing piece is how to perform test-time retrieval within the proposed framework and how its efficiency compares to the amortized approach. Given a query image, does the proposed approach learn a stochastic representation for it? Is the retrieval based on ranking the deterministic similarities between the query image and all other candidate images? If not, how is it performed?\n\nThe model predicts stochastic representations. The test time retrieval is performed by finding the nearest neighbors using the mean of the representation. This gives a ranking and can be computed as efficiently as in the deterministic case. The uncertainty of the retrieved candidate is based on the variances (or concentrations for vMF distributions) of the query and the candidate. Thus, the retrieval system provides a ranking of the nearest neighbors and the uncertainty of each neighbor. \n\n> The organization and clarity of this paper could be improved. If certain results are deemed important enough, they should be moved into the main text. The authors should focus on introducing the unique aspects of this work and provide a strong motivation for them, such as the Von Mises-Fisher distribution. The frequent references to the Appendix disrupt the logical flow of the paper.\n\nClarity of the presentation: We thank the reviewer for highlighting that the frequent references to the Appendix disrupt the reading flow. We will update the manuscript, such that it can easily be read without referencing the appendix, e.g. move vMF motivation to the main text. \n\n> Questions:\n> The probabilistic view in the supplemental also seems a little bit weird to me since both argument and parameter are data-dependent. Would it be sufficient to just treat the probabilistic contrastive likelihood as a second-order differentiable loss, if they are mathematically equivalent anyway?\n\nThe classic contrastive loss is a second-order differentiable loss, but this is *not* sufficient to treat it as a log-likelihood. Sufficient conditions would be to prove positiveness and integrability, specifically:\n\n(1) $loss>0$ everywhere, which is equivalent to $probability=exp(-loss)<1$\n\n(2) $\\int exp(-loss) = C < \\infty$\n\nCondition (1) is feasible to prove directly. Condition (2) is more tricky, proving the existence of such constant C is feasible, finding the exact value of such constant is more annoying.\nImportantly, such constant C is in general different from 1 (this can be seen for example by considering trivial cases) and thus the actual probabilistic loss needs to be renormalized. This means that in any case we need to introduce a new loss\n\n$$ probabilistic loss = constrastive loss - log(C) $$\n\nThis is of course “equivalent” as losses, since we are simply adding a constant term that will be neglected by the gradient.\n\nThis approach (modifying the contrastive loss until it is a log-likelihood) is totally ok, and the reviewer may consider it simpler. We argue that our approach (defining a log-likelihood and then showing it is equivalent) is more elegant, and perhaps more fundamental. Building a probabilistic loss from the basics, i.e. repulsive and attractive terms for each pair, leads to a better intuition on what is going on and how the different pairs interact with each other. Moreover, it gives cleaner access to the explicit value of the normalization constant, as an explicit function of the von-mises-fisher normalization constants.\n\n> What are the main roadblocks that have been overcome in applying the Laplacian autoencoder (LAE) to probabilistic retrieval?\n\nThe main roadblocks are (1) showing that the contrastive loss is a log-likelihood, (2) prosing novel approximations to ensure that the second-order derivative is semi-positive definite, (3) proposing a novel decomposition of the Hessian, such that the l2 normalization layer is not linearized. \n\n> Are there any other ways of approximating Hessian for contrastive loss, while still maintains scalability and positive definiteness? Is KFAC applicable, e.g., Ritter et al., 2018?\n\nOur methods will work out of the box for KFAC. Although, using ReLU to ensure semi-positive definiteness is only sufficient for the diagonal case. Both fixed and pos approximations can be used. "}]

Author rebuttal:
We thank the reviewers for their positive and constructive feedback. The reviewers found the problem considered “interesting” [R1] and stated that it is an “important topic for improving the robustness and mitigating the silent failure of deep neural network systems.” [R3] The paper is “well organized” [R1] and with “good writing. Clearly presenting the idea” [R1] “effectively uses figures to demonstrate ideas” [R3], providing “several analogies for various concepts introduced which makes the reader comfortable, and would be a resourceful paper for the community.” [R2] “The use of Laplace approximation [...] in the context of metric learning is novel. This includes the introduction of a probabilistic view of the contrastive loss and the utilization of the Hessian approximation based on GCN.” [R3] They found that the method is “conceptually simple”[R2] yielding “improved results, especially on OOD examples.” [R1] The experimental section provides “Extensive empirical evaluations, including a careful ablation study, are conducted.” [R3] “Most design choices in the method are covered by ablations” [R2] “and ablation study is comprehensive” [R1]. “The results demonstrate that the proposed approach outperforms HIE and PFE in the considered cases.” [R3] We are excited of the reviewers positive reception of the paper, and will address their questions below.

Meta-review: The paper proposes a Bayesian metric learning approach for image retrieval that uses the Laplace approximation to learn a distribution over the weights. The reviewers remarked on the novelty of using a loss based on contrastive learning. They appreciated the empirical evaluation and ablation study, which gives confidence in the improvement made. In general, the idea was broad enough to be applicable to many problems and the reviewers found it likely to appeal to the ML community. Therefore, all reviewers voted to accept the paper.
Decision: Accept (poster)