SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Ablation on Hessian approximation and GGN decomposition. Online LA with the fixed approximation and Arccos distance performs best. Error bars show one std. across five runs.
mAP@5↑	AUROC ↑	AUSC ↑
Post-hoc	Euclidean fix	0.70 ± 0.0	0.57 ± 0.25	0.44 ± 0.01
Euclidean pos	0.70 ± 0.0	0.58 ± 0.23	0.45 ± 0.01
Euclidean full	0.70 ± 0.0	0.56 ± 0.26	0.44 ± 0.01
Arccos fix	0.69 ± 0.0	0.53 ± 0.20	0.46 ± 0.02
Arccos pos	0.70 ± 0.0	0.29 ± 0.11	0.48 ± 0.01
Arccos full	0.69 ± 0.0	0.55 ± 0.18	0.45 ± 0.01
Online	Euclidean fix	0.63 ± 0.01	0.77 ± 0.04	0.31 ± 0.02
Euclidean pos	0.70 ± 0.0	0.38 ± 0.10	0.47 ± 0.01
Euclidean full	0.67 ± 0.01	0.59 ± 0.04	0.42 ± 0.01
Arccos fix	0.71 ± 0.0	0.78 ± 0.18	0.50 ± 0.03
Arccos pos	0.70 ± 0.0	0.23 ± 0.03	0.46 ± 0.00
Arccos full	0.71 ± 0.0	0.70 ± 0.12	0.51 ± 0.01
(a) FashionMNIST
(b) CIFAR10
Figure 5: Calibration Curves.
eral strong probabilistic baselines on closed-set retrieval and a more challenging open-set retrieval.
Closed-Set Retrieval. OoD capabilities are critical for identifying distributional shifts, outliers, and irregular user inputs, which might hinder the propagation of erroneous decisions in an automated system. We evaluate OoD performance on the commonly used benchmarks [31], where we use (1)
Table 2: Closed-set results. LAM matches or outperforms existing methods in terms of predictive performance. It produces reliable uncertainties ID and OoD on two standard datasets FashionMNIST and CIFAR10. Error bars show one std. across five runs.
IMAGE RETRIEVAL	OOD	CALIBRATION
mAP@1↑	mAP@5↑	mAP@10↑	AUROC ↑	AUPRC ↑	AUSC ↑	ECE ↓
FashionMNIST	Deterministic	0.78 ± 0.01	0.73 ± 0.01	0.72 ± 0.01	—	—	—	—
Deep Ensemble	0.69	0.62	0.59	0.41	0.46	0.61	0.04
PFE	0.78 ± 0.00	0.74 ± 0.00	0.72 ± 0.00	0.53 ± 0.03	0.46 ± 0.01	0.65 ± 0.01	0.26 ± 0.02
HIB	0.69 ± 0.08	0.63 ± 0.09	0.61 ± 0.09	0.60 ± 0.12	0.60 ± 0.11	0.65 ± 0.08	0.54 ± 0.08
MC dropout	0.76 ± 0.03	0.71 ± 0.03	0.70 ± 0.03	0.93 ± 0.03	0.93 ± 0.03	0.84 ± 0.06	0.03 ± 0.04
LAM (post-hoc)	0.78 ± 0.00	0.74 ± 0.00	0.72 ± 0.00	0.96 ± 0.02	0.96 ± 0.02	0.86 ± 0.01	0.03 ± 0.00
LAM (online)	0.81 ± 0.00	0.77 ± 0.01	0.76 ± 0.01	0.98 ± 0.01	0.98 ± 0.01	0.89 ± 0.01	0.02 ± 0.00
CIFAR10	Deterministic	0.66 ± 0.00	0.59 ± 0.00	0.58 ± 0.00	—	—	—	—
Deep Ensemble	0.66	0.61	0.59	0.42	0.67	0.72	0.02
MC dropout	0.46 ± 0.01	0.37 ± 0.01	0.34 ± 0.01	0.60 ± 0.03	0.76 ± 0.02	0.61 ± 0.01	0.05 ± 0.00
HIB	0.11 ± 0.01	0.07 ± 0.00	0.05 ± 0.00	0.44 ± 0.17	0.70 ± 0.1	0.29 ± 0.03	0.04 ± 0.02
PFE	0.66 ± 0.00	0.60 ± 0.00	0.58 ± 0.00	0.21 ± 0.02	0.56 ± 0.01	0.56 ± 0.01	0.11 ± 0.01
LAM (post-hoc)	0.66 ± 0.00	0.60 ± 0.00	0.58 ± 0.00	0.50 ± 0.11	0.69 ± 0.07	0.81 ± 0.01	0.23 ± 0.01
LAM (online)	0.66 ± 0.01	0.60 ± 0.00	0.57 ± 0.01	0.78 ± 0.04	0.85 ± 0.03	0.83 ± 0.01	0.01 ± 0.00
FashionMNIST [49] as ID and MNIST [26] as OoD, and (2) CIFAR10 [21] as ID and SVHN [32] as OoD. We use, respectively, a standard 2- or 3-layer relu convolutional network followed by a single linear layer on which we compute LA with a diagonal Hessian.
FashionMNIST (ID) vs MNIST (OoD). Table 2 shows that both PFE and post-hoc LAM have a similar predictive performance to the deterministic model. This is not surprising, as both methods are initialized with the deterministic parameters, and then uncertainties are learned (PFE) or deduced (post-hoc LAM) with frozen weights. The awareness of uncertainties during training, grants the online LAM slightly higher predictive performance.
(a) FashionMNIST
(b) CIFAR10
Figure 6: Receiver Operator Curves. LAM assign high uncertainty to OoD observations.
PFE uses amortized inference to predict variances. This works reasonably within distribution but does not work well for OOD detection. This is because a neural network is trusted to extrapolate far away from the data distribution. Table 2 shows that MC dropout, LAM (post-hoc), and LAM (online) assign high uncertainty to observations outside the training distribution. Fig. 5 shows that both post-hoc and online LAM are near perfectly calibrated, giving very low ECE measures (Table 2).
CIFAR10 (ID) vs SVHN (OoD) is a slightly harder setting. Table 2 yields similar conclusions as before; Bayesian approaches such as MC dropout, LAM (post-hoc), and LAM (online) better detect OoD examples than neural amortized methods such as PFE. Online LAM has a similar predictive performance to state-of-the-art while having better ID (lower ECE and higher AUSC) and OoD (higher AUROC and AUPRC) performance. Fig. 5 shows the calibration plot for CIFAR10, where online LAM has near-perfect calibration. The CIFAR10 ROC curves (Fig. 6) show that online LAM is better at distinguishing ID and OoD examples.
Open-Set Retrieval. A key advantage of metric learning methods is that they easily cope with a large number of classes and new classes can be added seamlessly. We therefore evaluate LAM's performance on challenging open-set retrieval, where none of the classes in the test set are available during training. We first test with CUB200 [45] as ID and CAR196 [20] as OoD similarly to Warburg et al. [47], and second, test with LFW [17] as ID and CUB200 as OoD. We use a ResNet50 [16] with a GeM pooling layer [35] and a 2048 dimensional embedding and diagonal, last-layer LA [8].
CUB200 (ID) vs CARS196 (OoD). The CUB-200-2011 dataset [45] has 200 bird species captured from different perspectives and in different environments. We follow the zero-shot train/test split [30]. In this zero-shot setting, the trained models have not seen any of the bird species in the test set, and the learned features must generalize well across species. Table 3 shows that LAM matches or surpasses the predictive performance of all other methods. LAM (post-hoc) achieves state-of-the-art
Table 3: Open-set results. LAM matches or outperforms existing methods in terms of predictive performance and produces state-of-the-art uncertainty quantification for challenging zero-shot metric learning datasets LFW and CUB200. Error bars show one std. across five runs.
IMAGE RETRIEVAL	OoD	ID
mAP@1↑	mAP@5↑	mAP@10↑	AUROC ↑	AUPRC ↑	AUSC ↑
CUB200	Deterministic	0.62 ± 0.01	0.48 ± 0.01	0.42 ± 0.01	—	—
Deep Ensemble	0.21	0.11	0.07	0.47	0.55	0.21
PFE	0.62 ± 0.01	0.5 ± 0.01	0.43 ± 0.01	0.44 ± 0.16	0.5 ± 0.08	0.61 ± 0.02
HIB	0.33 ± 0.04	0.19 ± 0.02	0.14 ± 0.02	0.54 ± 0.12	0.61 ± 0.1	0.31 ± 0.07
MC dropout	0.61 ± 0.00	0.48 ± 0.00	0.42 ± 0.00	0.73 ± 0.08	0.68 ± 0.07	0.63 ± 0.01
LAM (post-hoc)	0.65 ± 0.01	0.52 ± 0.01	0.45 ± 0.01	0.56 ± 0.16	0.61 ± 0.11	0.66 ± 0.03
LAM (online)	0.61 ± 0.00	0.48 ± 0.00	0.42 ± 0.00	0.80 ± 0.03	0.75 ± 0.03	0.63 ± 0.01
LFW	Deterministic	0.44 ± 0.00	0.68 ± 0.00	0.65 ± 0.00	—	—	—
Deep Ensemble	0.36	0.57	0.54	0.52	0.64	0.33
PFE	0.44 ± 0.00	0.68 ± 0.00	0.65 ± 0.00	0.03 ± 0.02	0.41 ± 0.0	0.49 ± 0.01
MC dropout	0.42 ± 0.00	0.65 ± 0.01	0.63 ± 0.01	0.03 ± 0.01	0.41 ± 0.0	0.46 ± 0.01
LAM (post-hoc)	0.44 ± 0.01	0.68 ± 0.01	0.65 ± 0.00	0.65 ± 0.14	0.72 ± 0.11	0.45 ± 0.03
LAM (online)	0.46 ± 0.00	0.71 ± 0.00	0.69 ± 0.00	0.71 ± 0.22	0.78 ± 0.17	0.50 ± 0.02
Figure 7: Images with lowest and highest variance for PFE, post-hoc LAM, and online LAM across LFW (ID) and CUB200 (OoD) datasets. LAM associates high uncertainty to OoD examples, and vice versa for PFE. Shows the best-performing PFE and online LAM across five runs.
predictive performance, while LAM (online) matches the predictive performance of the deterministic trained model while achieving state-of-the-art AUROC and AUPRC for OoD detection.
$LFW$  (ID) vs  $CUB200$  (OoD). Face recognition is another challenging metric learning task with many applications in security and surveillance. The goal is to retrieve images of the same person as in the query image. Table 3 shows that online LAM outperforms existing methods both in terms of predictive performance and uncertainty quantification. Fig. 7 shows that PFE assigns higher uncertainty to images from the ID dataset (faces) than those from the OoD dataset (birds).
In contrast, both online and post-hoc LAM better associate high variance to OoD examples, while PFE predicts high variance to ID examples. Furthermore, online LAM seems to assign the highest variance to images in which the background is complex and thus camouflages the birds.
Visual Place recognition is important for the long-term operation of autonomous robots [7], where the goal is to retrieve images taken within a radius of 25 meters from a query image. The high number of unique places and varying visual appearance of each location - including weather, dynamic, structural, viewpoint, seasonal, and day/night changes - makes visual place recognition a challenging metric learning
Figure 8: Sparsification curve. Online and post-hoc LAM's sparsification curves monotonically increase, showing that they reliably associate higher uncertainty to harder observations.
Table 5: Results on MSLS. LAM yields state-of-the-art uncertainties and matches the predictive performance of deterministic trained models.
Validation Set	Challenge Set
R@1↑	R@5↑	R@10↑	M@5↑	M@10↑	AUSC↑	R@1↑	R@5↑	R@10↑	M@5↑	M@10↑	AUSC↑
Deterministic	0.77	0.88	0.90	0.61	0.56	—	0.58	0.74	0.78	0.45	0.43	—
MC Dropout	0.75	0.87	0.87	0.59	0.54	0.77	0.55	0.71	0.76	0.43	0.41	0.57
PFE	0.77	0.88	0.90	0.61	0.56	0.73	0.58	0.74	0.78	0.45	0.44	0.57
LAM (post-hoc)	0.76	0.86	0.89	0.60	0.55	0.74	0.58	0.74	0.78	0.45	0.44	0.59
LAM (online)	0.76	0.87	0.90	0.60	0.56	0.77	0.57	0.74	0.78	0.45	0.43	0.63
LAM (o)LAM (p)	Lower uncertainty	Higher uncertainty
problem. Reliable uncertainties and reliable out-of-distribution behavior are important to avoid incorrect loop-closure, which can deteriorate the autonomous robots' location estimate. We evaluate on MSLS [46], which is the largest and most diverse place recognition dataset currently available comprised of  $1.6M$  images from 30 cities spanning six continents. We use the standard train/test split, training on 24 cities and testing on six other cities. We use the same model as in open-set retrieval.
Table 5 shows that online LAM yields state-of-the-art uncertainties for visual place recognition measured with AUSC, while matching the predictive performance of the alternative probabilistic and deterministic methods on both the MSLS validation and the challenge set. Fig. 8 shows the sparsification curves on the challenge set. Both online and post-hoc LAM have monotonically increasing sparsification curves, implying that when we remove the most uncertain observations, the predictive performance increase. This illustrates that LAM produces reliable uncertainties for this challenging open-set retrieval task. Fig. 9 shows the queries associated with the highest and lowest uncertainty. LAM predicts high uncertainty to images with are blurry, captured facing into the pavement, or contain mostly vegetation. These images do not have features that are descriptive of a specific place, making them hard to geographically locate.
Ablations. We conduct ablations on LFW for online and post-hoc LAM. Table 4 shows that the predictive performance does not depend on the margin, however, OOD performance decreases significantly for larger margins. We find that LAM is robust to the number of pairs sampled per batch. We find that a larger latent space results in better-calibrated uncertainties. Online LAM is rather robust to the choice of memory factor  $\alpha$ . However, choosing a memory factor in the 0.0001–0.001 range gives both good AUROC and AUSC. This memory factor  $\alpha$  can be interpreted as a momentum-like parameter that relates to the learning rate and indicates how fast we should update the current Hessian [29].
Figure 9: Images with lowest and highest variance for PFE, post-hoc LAM, and online LAM across MSLS validation set. LAM reliably associates high uncertainty to images that are blurry, are captured facing the pavement, or contain vegetation. These images do not contain features that are descriptive of a specific place, making them especially challenging to geographically locate.
Margin
mAP@5	AUROC	AUSC
0.1	0.71	1.00	0.50
0.3	0.71	0.57	0.49
0.5	0.70	0.16	0.48
0.7	0.70	0.09	0.48
Number of pairs per batch
mAP@5	AUROC	AUSC
1	0.72	0.99	0.49
5	0.72	0.98	0.50
10	0.71	1.00	0.49
30	0.72	1.00	0.49
Latent dimension
mAP@5	AUROC	AUSC
128	0.71	0.99	0.47
256	0.71	1.00	0.49
512	0.71	1.00	0.50
2048	0.72	1.00	0.52
Memory factor α
mAP@5	AUROC	AUSC
0.1	0.70	1.00	0.41
0.01	0.71	0.99	0.46
0.001	0.72	1.00	0.48
0.0001	0.72	0.98	0.50
0.00001	0.72	0.87	0.51
Post-hoc tempering β
mAP@5	AUROC	AUSC
100	0.67	0.10	0.48
10	0.69	0.68	0.46
1	0.69	0.86	0.44
0.1	0.69	0.87	0.43
0.01	0.69	0.86	0.44
0.001	0.69	0.86	0.43
Table 4: Ablation on LFW.
Lastly, we perform cold-posterior tempering of post-hoc LA by scaling the log-likelihood hessian with a factor  $\beta > 0$ , such that the precision in Eq. 3 became  $\beta \cdot \nabla_{\theta}^{2}\mathcal{L}_{\mathrm{con}}(\theta^{*};\mathcal{D}) + \sigma_{\mathrm{prior}}^{-2}\mathbb{I}$ . We do not see any benefits from changing to other values than one.
Limitations. Similar to other Bayesian methods, LAM relies on  $n$  samples to obtain uncertainties. This makes inferences  $n$  times slower. Computing the Hessian at every step during online LAM also makes training time slower. To combat long training times and high memory usage, we use a last-layer LAM and thus only estimate and sample for a weight posterior of the last layer. The last-layer LAM training time is 3 hours for online LAM vs 2.3 hours for deterministic contrastive loss on LFW, and 30 minutes vs 15 minutes loss on CUB200 on an NVIDIA RTX A5000.
In this paper, we have introduced a Bayesian encoder for metric learning, the Laplacian Metric Learner (LAM), which uses the Laplace approximation. We prove that the contrastive loss is unnormalized negative log-likelihood on the spherical space, and develop three Hessian approximations, which ensures a positive definite covariance matrix. We propose a novel decomposition of the Generalized Gauss-Newton approximation that improves Hessian approximations of  $\ell_2$ -normalized networks. Empirically, we demonstrate that LAM consistently produces well-calibrated uncertainties, reliably detects out-of-distribution examples, and achieves state-of-the-art predictive performance on both closed-set and challenging open-set image retrieval tasks.
Acknowledgement. This work was supported by research grants (15334, 42062) from VILLUM FONDEN. This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 757360). The work was partly funded by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF20OC0062606).
[1] Relja Arandjelović, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5297-5307, 2016.
[2] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks, 2015.
[3] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5710-5719, 2020.
[4] Peter Ebert Christensen, Frederik Warburg, Menglin Jia, and Serge Belongie. Searching for structure in unfalsifiable claims. arXiv preprint arXiv:2209.00495, 2022.
[5] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8415-8424, 2021.
[6] Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into backprop. In International Conference on Learning Representations (ICLR), 2020.
[7] Andrew J. Davison, Ian D. Reid, Nicholas D. Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6): 1052-1067, 2007. doi: 10.1109/TPAMI.2007.1049.
[8] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace latex-effortless Bayesian deep learning. In NeurIPS, 2021.
[9] John Denker and Yann LeCun. Transforming neural-net output levels to probability distributions. Advances in Neural Information Processing Systems, 1990.
[46] Frederik Warburg, Søren Hauberg, Manuel Lopez-Antequera, Pau Gargallo, Yubin Kuang, and Javier Civera. Mapillary street-level sequences: A dataset for lifelong place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[47] Frederik Warburg, Martin Jørgensen, Javier Civera, and Søren Hauberg. Bayesian triplet loss: Uncertainty quantification in image retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12158-12168, 2021.
[48] Michael Wilber, Iljung S Kwak, David Kriegman, and Serge Belongie. Learning concept embeddings with combined human-machine expertise. In Proceedings of the IEEE International Conference on Computer Vision, pages 981-989, 2015.
[49] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, 2017.
[50] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks, 2021.