Hashing with Uncertainty Quantification via Sampling-based Hypothesis Testing
Authors: Yucheng Wang
Mingyuan Zhou
Xiaoning Qian
Venue: Accepted by TMLR ()

Abstract: To quantify different types of uncertainty when deriving hash-codes for image retrieval, we develop a probabilistic hashing model(ProbHash). 
Sampling-based hypothesis testing is then  derived for hashing with uncertainty quantification(HashUQ) in ProbHash to improve the granularity of hashing-based retrieval by prioritizing the data with confident hash-codes. HashUQ can drastically improve the retrieval performance without sacrificing computational efficiency. For efficient deployment of HashUQ in real-world applications, we discretize the quantified uncertainty to reduce the potential storage overhead. Experimental results show that our HashUQ can achieve state-of-the-art retrieval performance on three image datasets. Ablation experiments on model hyperparameters, different model components, and effects of UQ are also provided with performance comparisons. Our code is available at https://github.com/QianLab/HashUQ.

Published in Transactions on Machine Learning Research (10/2024)
Hashing with Uncertainty Quantification via Sampling-based Hypothesis Testing
Yucheng Wang
wanggnuchengStamu..d
Deportment of Electrical and Computer Engineering, Teaos A&M University
Mingyuan Zhou
miinguan.chondhhccoomb....uu.a
McCombs School of Business, The University of Tezas at Austin Department of Statistics and Data Sciences, The University of Tezas at Austin
Xiaoning Qian
zqian Stamu edu
Department of Electricol and Computer Engineering, Teras A&M University Department of Computer Science Engineering, Texas A&M University Computational Science Initiative, Brookhaven National Laboratory
Reviewed on OpenReview: https: // openreviee.. net/ formm? id= cc4v6u310f
Abstract
To quantify different types of uncertainly when deriving hash-codes for image retrieval, we develop a probabilistic hashing model (ProbHash). Sampling- hypothesis testing is then derived for hashing with uncertainty quantification (HashUQ) in ProbHash to improve the granularity of hashing- based retrieval by prioritizing the data with confident hash-codes. HashUQ can drastically improve the retrieval performance without sacrificing computational efficiency. For efficient deployment of HashUQ in real-world applications, we discretize the quantified uncertainty to reduce the potential storage overhead. Experimental results show that our HashUQ can achieve stale-of-the-arl retrieval performance on three image datasets. Ablation experiments on model hyperparameters different model components, and effects of UQ are also provided with performance comparisons. Our code is available at cortliaaaa//iaaa btips://githune htt.tttb.comm//m/ QianLaaa//////aaa (QjanLab/HashUQ, HashUQ
1 Introduction
Representation learning summarizes high-dimensional data into low-dimensional feature vectors with the key information preserved for conducting various downstream tasks, which has been successfully applied to different domains such as natural language processing (NLP) (Bahdanau et al. 2014), generative modeling (van den
Oord et al. 2017 Liu et al. 2019 2020), data compression (Ballé et al. 2016 Theis et al. 2017 Ballé
et al.] 2018 and multi-modal machine learning (Chen et al. 2020). Information retrieval can leverage
representation learning to efficiently return the most similar data samples within a database given a query. With the power of such representation models, Learning-to-Hasl (I.2H) (Weiss al.) 2008 Salakhutdinov
& Hinton 2009 Strecha et al.) 2011 Liu et al., 2011 2014: Li al. 2016) has become one commonly adopted group of algorithms, which are designed for efficient information retrieval based derived discrete representations in hashing codes (hash-codes). With the extremely compact binary hash-codes derived from high-dimensional data, L2H enables fast comparison and scarch by comparing the Hamming distance of hash- codes. More recently, the retrieval accuracy of L2H algorithms with different. lypes of data has experienced remarkable improvement, thanks to the rapidly evolving modern computer hardware and advanced deep neural network (DNN) architectures (Krizhevsky et al.] 2012: SSmonyyn Zisserman, 2014 He al.) 2016 Vaswani et al. (0117) with efficient training algorithms (Hinton al. 2012 Kingma & Ba 2014 Bengio et al.
2013 Jang et al.) 2016 Maddison et al. 2016 Yin & Zhou 2019 Yin et al. 2020

---

Published in Transactions on Machine Learning Research (10/2024)
(a) Prediction
(b) Database
1,1,0 m. 0,
Low
Image
Hash-codes
Uncertainties
11100 m. 0
Low
Input Image
Model
Predicted Success Rates
Binary Codes
Uncertainties
1, -0
High
(c) Retrieval
Hash--Codes
Hamming Distance:
9999 O0O
Uncertainttes:
Tew
High
High
00
1,0,0,
Imuges:
LDT IOLIE
0. 0,0, 0.0
High
Relovar
Query
Retrieval Order of Image Data
Figure 1: Schematic illustration of HashUQ with binarized uncertainties ("Low Uncertainty" v.s. "High Uncertainty"): (a) ProbHash can predict probability distribution of Bernoulli success rates of hash-codes, from which the binary codes and the associated uncertainty can be jointly inferred. (b) Database stores both hash-codes and associated uncertainties. (c) For images with the same Hamming distance based on the derived hash-codes to the query, the retrieval order is determined by the corresponding predicted uncertainty.
Despite the tremendous amount of research conducted on representation learning, few of them aim at quantify- ing the uncertainty of derived representations. As DNNs have long been criticized for their tendency to make overconfident. predictions and vulnerability to adversarial attacks, uncertainly quantification (UQ) in DNNs has received increasing attention and growing interest in the machine learning research community (Kendall & Gal 2017: Abdar et al.] (021). Scalable UQ via various approximate inference methods has boen developed for Bayesian Neural Networks (BNNs) (Lampinen & Vehtari, 2001 Titterington 20044: Neal 2012). Although they have greatly reduced the computational complexity of Markov chain Monte Carlo (MCMC) Neal et al.
2011 Welling & Teh 2011), few of them have been shown with promising UQ capability on representation and L2H models. Moreover, there still lacks practical way to utilize the quantified uncertainty for improving the accuracy for downstream information retrieval.
In this paper, we propose a scalable image hashing method with UQ for image retrieval. Our main contributions can be summarized as follows:
We develop supervised image hashing method with uncertainty quantification capability by jointly modeling the hash-codes and neural network weights random variables, which we term as Probabilistic Hashing (ProbHash). To the best of our knowledge, this is the first supervised image hashing algorithm with appropriately quantified uncertainty.
To jointly model different lypes of uncertainly, we propose I-test based measure for UQ, which provides statistically interpretable notion of uncertainty for hashing. Our UQ method can be readily applied to various types of existing L2H models for accurate data retrieval, which we exemplify with two representative image hashing models.
With our quantified uncertainty, we further design an image retricval algorithm by prioritizing images with condifent hash-codes, which we term Hashing with Uncertainty Quantification (HashUQ). Experiments several benchmark datasets show that this consistently improves the quality of the predicted image ranking without scarifying the speed and storage consumption.
We structure our paper by first describing our probabilistic framework, ProbHash, for hashing with two representative image hashing models given in Section 2 The information supplementary to our model construction included in Appendix A We detail our t-test based mcasure of uncortainty and uncortainty aware retrieval algorithm, HashUQ, Sections and Experimental evaluation by comparing with other state-of-the-art. (SOTA) models and ablation studies of our methods are then presented in Section 5 with the
2

---

Published in Transactions on Machine Learning Research (10/2024)
detailed experimental setups and supplementary results provided in Appendix B Lastly, we provide literature review on Learning to Hash (T.2H), the existing Uncertainty Quantification (UQ) schemes for deep learning, as well as the information bottleneck methods in Section G We also include a comparison between our proposed f-test based mcasure of uncertainty with Shannon's s entropy in Appendir C and provide retrieval examples of our developed methods in Aypendir D
Figure provides a schematic summary of HashUQ.
2 Probabilistic Modeling for Image Hashing
2.1 Mathematical Notations
Unless explicitly specified, we use the normal font letter æ, bold letter æ and bold letter in upper cases X to denote scalar, vector, and matrix, respectively. We use letter æ with subscript i, k: Di,k, to represent the wah entry of vector 2;, which is the jth data point in the dataset. We use the superscripts x", x to denote a pool data point and query, respectively. N, Nsamples: and K denote the number of data points, number of samples, and bit-length of hash-codes, respectively.
2.2 Problem Settings
Suppose we have image dataset {x1, .2...... ..... ØN with corresponding labels {y1, 32...., yyr our objective is to find mapping ʃ Rexhxw - 1,1 which can encode all the training images {mr) as well as possibly unseen query pool images { 1 into binary hash-codes {b.) and { such that images with similar content can be encoded into hash-codes with smaller Hamming distances, and vise versa.
Recent image hashing methods [Zhu et al. 2016 Cao et al. 2017 Su et al. 2018 Yuan et al. 2020 Fan
ct al. 2020 Tian Hoe et al. 2021 usually adopt a foed-forward neural network with parameters ab and binary quantization function sgn( ). which we denote as fw. to model the hash encoding function f. Based on the lcarning objective and the availability of image labels, we categorize the existing image hashing methods into
Llabel-Target" (Cao et al. 2017 Su et al., 2018) and "Center-Targee methods (Yuan al. 2020 Fan et al.,
2020
Tian Hoe et al. 2021).
In "Label-Target" methods, the label yi for each image æ; is dircctly used as the learning target of binary hash-code represenlation b, such that yi can be reconstructed from the binary representation with decoder network. The "Center-Targe Target methods, on the other hand, minimize the distance of hash-code b, to its "center" q, To generate the center for each image 2ú, a Hadamard matrix with dimension K, for which we briefly review the basics in the context of hashing in Appendir A.1 is first constructed and each row of the matrix will be assigned to one of the M classes (Yuan et al. 2020 The "center" e; for image æ; is then the row vector corresponding to image label yı. Some other randomized algorithms have also been used for the cases when M >>> K (Yuan et al. 2020).
2.3 ProbHash
Learning the hashing function f can inevitably introduce different types of uncertainties to different phases in model construction, training, as well as prediction. To give an example, applying the pre-pparameteriee DNN models will indispensably lead to the uncertainty in model parameters. Moreover, the learning targets to model are also noisy, which can be shown either from a Hadamard matrix or some randomized algorithm and further randomly assigned to cach class. This motivates us to build probabilistic (in contrast to existing deterministic) hashing model which can consider the uncertainty with different sources and characters. We therefore develop BNN-based probabilistic hashing (ProbHash) by modeling &b as random variables. The hash-codes b are further modeled as another Bernoulli random vector given a realization of ab and an input image data sample æ. We illustrate our ProbHash modeling using probabilistic graphical model in Figure
Ceenter-Target" setup where the centers M M are generated
While we focus on developing our method based on two representative supervisod image hashing models, ProbHash a general framework that can be seamlessly extended to the majority of L2H models with various wetwork structures, data types, likelihood assumptions, and various supervised and unsupervised formulations.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 2: Illustrations of our probabilistic hashing (ProbHash) in graphical model.
Different variational families, such dropout Gal & Ghahramani 2016 with learnable data-dependent
rates (Gal et al. 2017 Boluki et al. 2020: Fan et al. 2021) and factorized Gaussian (Blundell et al. 2015). may also be adopt on vp to achieve better flexibility. The probability distribution p (1b) and (bb x) can be interpreted to represent epistemic uncertainly and aleatoric uncerlainly in hash-code prediction, similar as the categorization in literature (Kendall Gal 2017 Hüllermeier Waegeman 2021). Our probabilistic model for L2H can offer more concise formulation comparod to many existing methods (Zhu al. 2016 Cao et al. 2017 Su et al., 2018) Yuan et al.
2020 Fan et al., 2020; Tian Hoe et. al., 2021 which adopt diflerent. regularization tricks to minimize the quantization error in model training. Moreover, the introduction of construction, which can be easily optimized without the straight-throus through heuristic (Bengio et al. 2013) as detailed in Section 2.4
p(bly,, æ) will lead to closed-form log-like ligllikel function in "Center-Targe
2.4 Bayesian Learning for Image Hashing
Our main objective is to learn p (b/ær, 1b), which is the conditional distribution of b by neural worw hashing function parametrized via ab after observing image æ, and (æ/(2,, yi J.) which is the posterior distribution of neural network parameters after obscrving the training data {æ,, V To jointly learn p (blzr 4b) and p (vb({ yi ) in the Bayesian paradigm while avoiding the intractability of computing the marginal distribution p(y]z)) we resort to the amortized variational inference with the variational distributions qbb|z, vb) and ((b).. We learn the variational parameters +p by maximizing the Evidence Lower BOund (ELBO):
logp p 1xx
ll (+) (1)
> (
(g(b..1..)) [logp (y: [b))]
Σ DKL (a (bilvb,,;)) a;)) ||v (b. (z:))
Dri (I
We provide by-teb derivation of Expression in Appendir
We set the prior distribution p (blæ)) in 11 to be factorized Bernoulli distribution with the success probability p being a hyperparameter p(blz) p(b) Bernoulli(b; p) independent of æ. This will lead to the closed-form KL-divergence term in
Σ a (fw.k (a:)) log o (fu. (xx)) + (-fx.t (2:)) log T (- fur. (2x:)
K
DEL (g (balas Dr, ab) 11 (b:))
Pk
Pk
1=1
Next, we give our detailed model constructions and provide the specific optimization objectives for "Label- Target" and "Center-Target", respectively.
ProbHash with Label-Target Construction: For "Label- Targot" construction, we assume the likelihood p (b) to be categorical distribution parameterized by a neural network go (-) with the softmax activalion:
y ~ Categorical (y: Softmax (ge (b))),
where 0 are the parameters of the likelihood model. The variational posterior q (b|z, v)) is then constructed
to be a factorived Bernoulli distribution parameterized by neural network with parameters ub: f (-) and the sigmoid activation function o (-):
q (b) ar, 1b) Bernoulli (b; (fw (m))).
4

---

Published in Transactions on Machine Learning Research (10/2024)
Let Wi be the neural network weights at layer I, and +b (Wi) We assume the independence of the weights of each layer q (+b) = II (Wi). For simplicity, we adopt the dropout variational distribution with the dropout rate at layer I to be Pt E [0, 1], following Gal & Ghahramani (016)) The neural network weights Wi for dropout variational distribution
be written
Wi = M1 Z1,
where denotes the Hadamard (entry-wise) product and Zi is the dropout mask, each row containing cither all is Os with probability pt and 1 Pt- {Mi) denote the variational parameters, which will be learned from data jointly with 0. We omit the subscript. for {Mi) and only use g(*)) to denote the net work weight variational distribution throughout our discussion. The KL-divergence of the variational posterior with the Gaussian prior Dx (xz((()))lp) (<<))))>>))>))))>))>)) can be approximated using a weight docay tcrm Gal & Ghahramani 2016). which we also omii the following discussion for simplicity. We also nole here that there are other possible amortized variational inference solutions. As an ablation study, an empirical comparison with another widely used variational distribution family of fully factorized Gaussian weights is included in Appendix B.5
With the training data (2;; Vi Di we solve the following loss minimization problem with the estimated ELBO:
CELBO ([Mi))) Eg((s)
[xbb,/m.,3)) [log pe (y b:)] + A DEL (b, +)) Ilp (b.lzz))
(2)
where Ả is a hyperparameter reflecting the trade-off between model fitness to the training data and discrepancy to the prior distribution.
ProbHash with Center-Targer Construction: To derive ProbHash for existing "Center- "Center-Target" meth- ods (Vuan et al. 2020), we assume the likelihood distribution p (ys(bb)) proportional to the exponential of the Hamming distance between b, and the predefined "center" vector for 7th data, C;:
p (u.lb.) ((.(b,)) exp ( (-oD (ci, b:))
(3)
Z
where Z is the normalizing constant. A discussion and comparison with other likelihood choices is included in Appendir A.3. The negative ELBO minimization objective L' (LBO((P) can be similarly derived to be:
N
N
L' PIBBO (bb) = Ey((s)
Σ
(. [logp (ci(b:)) + A
Σ
DKL (g (bilar, +b) lip (b. (bll):)))
(4)
Notice that by further incorporating following closed form:
3 into
44. the log-likelihood term in the negative ELBO has the
Ey(b, ())
| log p(c: 64)]
(0) c;
bil
log 7|
K
= -B (2))
be,al - llog 7 10,,
| 20,,00 (fo. (xx)) c,s + 1] - 1logllog222 Z.
where log Z is a constant irrelevant to the variational parameters vb. This closed-form log-likelihood can be easily estimated and optimized with any package equipped with automatic differentiation. Our negative ELBO minimization objective with both log-likelihood and KT--divereence terms in closod-forms can lead more principled training with better retrieval accuracy, compared to existing works (Zhu et al. 2016 Cao
et al. 2017 Su et al. 2018 Yuan et al. 2020 Fan et al. 2020: Tian Hoe et. al. 2021 where model optimization
5

---

Published in Transactions on Machine Learning Research (10/2024)
by biased gradient estimation of the discontinuous function ab with the heuristic straight-through (ST) trick (Bengio al. 2013) can lead degraded performance, as shown empirically in Section 5.4.1
Connection to Information Bottleneck: While originally derived from the perspective of Bayesian inference, our proposed ProbHash with "Label-Target" construction can be considered as realization of variational information bottleneck (Alemi et al., (016) on Learning-to- toahgg-tt-Hashh with a Bernoulli latent representation and a Bayesian neural network encoder. Minimizing the loss (2). which has a similar format. as the variational information bottleneck objective in [llemi et al. (2016), is also equivalent to maximizing the mutual information between latent hash-codes b and labels y while minimizing the mutual information between latent hash-codes b and inputs æ.
2.5 Predicting the Binary Hash-codes
By solving the optimization problems discussed in Section 2.4 we obtain two variational posteriors qu (b|ær, 1h) and q (1)). Our goal is to find mapping from imput data to binary vector for image hashing, We caleulate the probability distribution of hash-code b given new test image by marginalizing out the model paramcters bb:
q (Blx) - /
2b) (
[blē,
(vb) d'b
55)
where y (b)z)) Bernoulli distribution whose success probability can be estimated using Monte-Carbee sam- pling. With Sample samples of neural network parameters {th, Ns the Bernoulli success probability of hash-code can also be sampled: for (fw. ( Nsumute, and the marginal distribution of hash-code distribution can be estimated by:
Σ
-= Nsampic (fua (i))
q (6/$) Bernoulli (6: Ess (or (far (8))) Bernoulli
6
Nsamplee
We use the marimum-a a-stttmri---oooiiooooo (MAP) estimation of the variational posterior (6(()) to predict the most probable hash-code of the input image. This is equivalent thresholding the predicted Bernoulli success probability at 0.5:
argmax ( = (0))200 - 1E (00))
6E(-1.1)*
3
Quantifying Uncertainties in Hashing
3.1 Uncertainties in Hashing
Our probabilistic modeling for both &b and b bjr, sb leads hierarchically constructed distribution for hash-code b with two groups of uncertainties, & in our learned model, with g(tb) being similar to Dirac delta function, indicates high confidence over model parameters. On the other hand, given vb and x, the predicted Bernoulli success probability closer to 0 and or 0.5 respectively means the model is certain or uncertain of the specific entry of hash-codes. Suppose NSample with each sb ~ q(bb). For the ease of discussion, we will use ( to denote the Bernoulli success rate of t entry given input image & with n sample ;bn::
mentioned in Section 2.3 In particular, low uncertainty
we have Nsample samples of neural network parameters (tbn) )
( = (f
Consider four different cases when the mean value of π. to be either 0.6 or 0.9, with either low or high variance. If model consistently predicts π.A = 0.9, this indicates the model elcarly towards prodicting 6. to be "1". In hir cases when predicting 0.6 consistently or 0.9 on average with high variance, the model is still favor of deciding on "1" but with more uncertainty. In the last case when the model outputs with the mean value of π. to be 0.6 with high variance, the model less confidence for the decision on "1". This motivates us to quantify the uncertainty by conducting hypothesis testing on the samples of predicted Bernoulli success rates of hash-codes, which detail in Section 3.2

---

Published in Transactions on Machine Learning Research (10/2024)
3.2 Measure of Uncertainty by t-test
For any of the 4th entry of a hash-code b.s, the significant difference between the Bernoulli success and failure probabilities indicates confident prediction while having similar success and failure probabilities implics uncertain prediction. Given the sampled probabilities of b,, being "1": { ) and "0": {1- kn, - Ngampte, we conduct student's t-test of the mull hypothesis that the mcan probabilitics of being "I" and "0" are equal on each of the kthh entry. We represent the uncertainty in J using the p-value of the test, which we denote as Px((). A smaller p-value in favor of rejecting the null hypothesis indicates a more significant difference between the Bernoulli success and failure rates, reflecting our confidence in b. The total uncertainty on the hash-code given an input image & is represented using the summed log p-value over all the K entries: Σk= log Ps (.)). The summed log p-value over all entries provides an aggregated measure of bash-code uncertainty for one specific input data & which can also be interpreted as the logarithm of the joint tail probability given the mull hypothesis under the factorization assumption.
We adopt the paired sample I-test considering the dependency of the success and failure rates. This test is also equivalent to performing one sample t-test of the null hypothesis that the sample mean of Bernoulli success rate equals 0.5, which can be interpreted similarly.
3.3 Discussion
The idea of using statistic hypothesis lesting as the measure of uncertainty is not totally new, and has already been adopted in some previous works in other applications (Fan et al., 2021), which we find to be potentially suitable for quantifying the uncertainties of hash-codes when the two aforementioned uncertainties are presented together. The above metric also matches our intuition about uncertainties in L2H discussed in Section 3.1 as the I-statistics becomes higher when the Bernoulli success rates n² is less variant and closer to 1 or O, which will further lead to the lower tail probabilities and p-values, and implies a higher confidence. In Appendiæ C we provide detailed analysis and numerical comparison between our proposed paired sample t-test based measure with Shannon's entropy based measure of uncertainty. In Appendix D we further provide histograms and Q-Q plots of the difference between paired samples of each Bernoulli success rate of hash-code vectors generated using 5 query images from the ImageNet dataset, which validate the assumption for paired sample t-test that the difference between the paired values is normally distributed.
4 Hashing with Uncertainty Quantification
4.1 Hashing in the Presence of Uncertainty
Consider a database of Np data entries {ar 2..... ,2, }, with Np on the order of millions or even trillions with the corresponding hash-codes {b%. _6. ........ by J. When performing nearest neighbor search for the query x with code 89, compared to the data entries which are predicted to be relevant to the query but with high uncertainty, we prefer entries which are relevant based on the derived hash-codes with high confidence. This motivates us to take the quantified uncertainties into account for retrieval. We propose our uncertainty aware retrieval to prioritize the data entries with confident hash-codes while deprioritizing the data samples with uncertain hash-codes, which we term as Hashing with Uncertainty Quantification (HashUQ)-
In particular, when the query æ is provided along with the hash-code ba, we rank each entry xy in the database based the Hamming distance between 6 and b%, which denote as DHamming (b", b²), from low to high. In the meanwhile, we rank each entry based on quantified uncertainty associated with hash-code. The retrieval is performed by first comparing the Hamming distance DHamming (b', "."). When we encounter two entries with the same Hamming distance to the query, we further compare the quantified uncertainties of hash-codes and the entry with more confident hash-code will be retrieved with higher priority. We can oquivalently describe our method as retrieving a pool data sample af in sequential order based on the sorting results of the following expression in the ascending order over all the pool data in the database:
K
DHamming (b*,b') +
log Pa (x
km
7

---

Published in Transactions on Machine Learning Research (10/2024)
where is a positive real number small enough such that Ja Ek= log (+)) < 1. Notice that here is not a. hyper-parameter in either our model or algorithm. Given the fact that the length of hash-code K is typically much smaller than Np. thousands of database entries whose hash-codes have exactly the same Hamming distance to the query when performing nearest neighbor search can now be retrioved with the model confidence also laken into account.
4.2 Efficient Storage of Uncertainties
Our proposed test Bernoulli samples measures uncertainty in real value, which is typically stored in the "floating-point format in computer systems. Considering that the L2H is typically applied to the retrieval system which is sensitive to the storage space and rumning time, the space for storing the quantified uncertainty is unignorable compared to the hash-code which takes only tens or hundreds of bits.
We address this practical challenge by discretizing the quantified uncertainties. A simple example is to quantize the uncertainties into binary, with each hash-code to be either "Low Uncertainty" or "High Uncertainty" namely. When we discretize the quantified uncertainties into d quantized levels, the storage space reduces to log, d-bit complexity.
4.3 Computational Complexity
Assume that we have a total of N data samples in database and M queries for retrieval. For each query, only r data samples need to be retrieved from the database and ranked accordingly, with r & N. When the vanilla daamiing-distance-aaeed hashing strategy is applied, the total computational complexity for M retricvals is MN) for Hamming distance computation and OC22MN log N) for sorting, with C₁ C2 being two constants and C1 >> C2. As the quantified uncertainty is only associated with each data point and does not need to be computed each time when the retricval is performed, our uncertainty- ranking strategy will introduce almost no additional computation overhead to sort the quantified uncertainty when multiple retrieved images have the same distance to the query.
4.4 Discussion
Why Prioritizing Confident Pool Data: Given confident pool data sample and an uncertain data sample with the predicted Hash-codes having the same Hamming distance to the query. The uncertain pool data more likely to be either more relevant to the query, or irrelevant to the query compared to the confident one. However, given the fact that these Hamming distances are small when we are performing nearest neighbor search, it is more likely that the latter one hold true, and we should choose the confident. pool data sample to achieve better retrieval accuracy, which we validate experimentally in Appendir B.6
Introduced Computation Consumption: Different from other applications of machine learning where their inference time will be the main concern regarding computational budget, the hash-codes of pool data can be pre-computed and stored in hashing- based retrieval setup, and therefore the retrieval time and storage consumption are the main computational concerns. The inference time of the query data will remain the same as other non-probabillistic methods as we are not quantifying the uncertainty of query data. While we mainly focus on using the uncertainty values of pool data throughout the discussion above, the uncertainty values of query data can also be used to help improving retricval accuracy either in a scenario where the computational resources allow, or by adopting some other uncertainty measures which does not require intensive sampling, such Shannon's Entropy. We show in Section 5.3 that the query data with higher hash-code uncertainty will be more likely to retrieve irrelevant documents, which can be potentially combined with the aforementioned method to warn the users of the potentially erroneous retrievals.
5 Experiments
We empirically evaluate our ProbHash and HashUQ, comparing with other SOTA L2H methods. We first describe the experimental setup for performance evaluation by providing the evaluation matrices and baseline We empirically evaluate our ProbHash and compare with other SOTA T.2H
models in Sections 5.1 and 5.2
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table Comparison of HashUQ with both SOTA "Label-Target" and "Center-Target" baselines on ImageNet, MS COCO, and NUS WIDE datasets. The best performing models for each setup (Except for full HashUQ) are illustrated in bold font. denotes the overall best performing model.
ImageNet(mAAP11000
MS Coco (55000
NUS WIDE
Target
Method
16 bits
32 bits
64 bits
16 bits
32 bits
64 bits
16 bits
32 bits
64 bits
GreedyHash
0.611
0.657
0.680
0,654
0.702
0.714
0.776
0.796
0.812
HashNet
0.335
0.502
0.607
0.612
0.661
0.714
0.752
0.808
*0.844
Label
ProbHash
0.595
0.645
0.673
0.653
0.703
0.726
0.775
0.800
0.816
HQ-hUQ-Bin
0.618
0.653
0.675
0.663
0.709
0.729
0.790
0.806
0.818
HashUQ
0.623
0.655
0.678
0.669
0.712
0.730
0.799
0.812
0.822
CSQ
0.595
0.668
0.699
0.651
0.732
0.769
0.789
0.820
0.834
OthorHash
0.587
0.669
0.712
0.624
0.689
0.718
0.792
0.825
0.841
HSWD
0.593
0.675
0.700
0.662
0.736
#0.771
0.788
0.824
0.835
Center
ProbHash
0.606
0.680
0.706
0.665
0.731
0.764
0.790
0.820
0.835
HashUQ- Bin
*0.623
*0.690
*0.714
*0.681
*0.739
0.768
*0.805
*0.829
0.839
HashUQ 0.631
0.696
0.718
0.688
0.743
0.771
0.814
0.834
0.843
methods as well as the UQ capability by HashUQ in Soction 5.3
Lastly in Section 5.4.1
we analyze the effects
of different model components and hyper-parameter sensitivity. Additionally in Appendia B
we provide
detailed description of benchmark datasets and model training, well the supplementary results on model evaluation and uncertainty quantification
5.1 Evaluation Metrics
Mean Average Precision: In ranking-based retrieval, Average Precision (AP) is the integrated performance measure jointly considering precision and recall. Let P(r) be the precision of the top images, and rel(r) the indicator function that the r-th image is relevant, and [Relevant]] the number of all the relevant images to the query the database. We have:
ΣN P(r) rel(r) AP [[Relevant]]
We use the mean Average Precision&r (mAP@r) on the whole test set to evaluate the retrieval performance of our uncertainly aware image hashing, with r retrieved images when we calculate the average precision.
Following previous works (Cao et al.] 2017 Su et al.) [018 Yuan et al. 2020 Fan et al.] 2020 Tian Hoe we set r = 1000 for ImageNet, and r = 5000 for both MS COCO and NUS WIDE.
ct al. 2021
5.2 Baseline Models
For each of the "Label- Target" and "Center- Target" method, we implement our ProbHash model based on "GreedyHash" (Su et al. 2018) and "CSQ" (Yuan et al. 2020), respectively. We test ProbHash with and without HashUQ, and denote cach of them as "ProbHash" " and "HashUQ" throughout this section. The retrieval algorithm with model confidence by HashUQ has been discussed in Section As HashUQ Lakes significantly more storage space compared to all other methods, we further binarize the uncertainties quantified using the paired sample t-test as detailed in Section 4.2 which we denote "HashUQ-Bin" We compare with the following SOTA baselines: HashNet (Cao et al. 2017), GreedyHash (Su et al. 2018). CSQ (Yuan et al. 2020), OrthoHash (Tian Hoe et al. 2021), and HSWD (Doan et al.) 2022). We reproduce the results of Greedy Hash, CSQ, and HSWD, and implement HashUQ performance evaluation and comparison based on the implementation from HampHaash-ytoorde with PyTorch 1.8.1 (Paszke al. 2019). We reproduce OrthoHash results using the implementation from its official GitHub repository? (Tin Hoe et al. 2021
httpe://gtthub..om/wwwwxy
9

---

Published in Transactions on Machine Learning Research (10/2024)
5.3 Experimental Evaluation of Proposed Methods
ProbHash Achieves Competitive Performances: Table summarizes the empirical performance of baseline models and different implementations of our proposed model. Compared to the corresponding baseline models, "GreedyHash" and "CSQ", our ProbHash achieves competitive retrieval accuracy. Across two constructions and three datasets, our ProbHash achieves similar or better performance on five setups and only perform slightly worse some of the experiments under one setup (Label-Target on ImageNet).
ProbHash Enables Uncertainty Quantification: To show that our ProbHash can provide reasonable uncertainty estimation for the predicted hash-code of a supervised image hashing model, we provide the boxplot of retrieval accuracy for the query images with respect to the quantile of their predicted hash-code uncertainty with both "Center-Target" and "aabeet" construction with hash-code length K = 32 on ImageNet in Figure 3 The corresponding plots for K = 16 and K = 64 are included in Figure 5 of Appendix B.2 In bothFigures Sa and 3b we can clearly observe that the predicted uncortainties are typically small for hash-codes of query image with high relrieval accuracy. The median, 25% and 75% quantiles of retrieval accuracy will all decrease the predicted hash-code uncertainties of query images increase, which shows that high estimated uncertainty will be typically associated with erroneous prediction.
1.0
1.0
Avcrage Preesion 1000
Aterrage Predsiion 1000
0.8
0.6
0 600
0.4
0.4
0.2
0.0
0.0
20%--10%
40%--60%
80%--00%
20%-40%
40%-860%
60%--00%
Quartile of Hash code Unceriaiity
Quamille of Hash code Uncertainty
(a) Center-Target
(b) Label- -arget
Figure 3: Boxplot of retrieval accuracy with respect the quantile of hash-code uncertainty with K 32 of our ProbHash with "Center- "Center-Target" and "Label- constructions ImageNet.
HashUQ Further Achieves State-of-the-art Retrieval Accuracy: By prioritizing data with high confidence, HashUQ-Bin consistently delivers superior performance across nearly all tested settings of the 'Label-Target' and 'Center-Target' constructs, demonstrating the broad applicability of our method. HashUQ-Bin under "Center-Targee can achieve prominent retrieval performance with the best retrieval accuracy in of the 9 settings tested, and is also comparable to the best performing models in the other two settings. Our HashUQ- -in improves the retrieval accuracy of the corresponding ProbHash model with a similar amount as the full HashUQ, demonstrating its practical effectiveness exploiting the benefits from UQ.
HashUQ is especially effective for retrievals with short hash-codes, as there will typically be more retrieved data samples whose hash-codes have the same Hamming distance from the query, compared to retrievals with long hash-codes. Although have only shown the results with two representative L2H baseline models, our ProbHash and HashUQ can be combined with other advanced neural network architectures (Yuan el al. 2020 improved likelihood function design (Tian Hoe et al. 2021). and more sophisticated regularization strategies (Tin Hoe et al.) 2021 Doan et al. 2022). We expect similar empirical performance improvement. In Scction 5.4.4] we demonstrate that Hash UQ outperforms bascline models in retrieval results, even when these models utilize more bits than HashUQ. This further validates the superior performance of HashUQ-Ba relative the baselines.
Discussion:
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Retrieval accuracy of our ProbHash with KL-regularization and HashUQ included and excluded, trained with different gradient estimators on ImageNet. We for experiments without a checkmarker À and 1.0 otherwise.
Component
ImageNet(mAPQ
Gradient Estimator
UashUQ
16 bits
32 bits
64 bits
Brraight-through
0.573
0.660
0.689
Straight-through Closod-forn
0.592
0.666
0.693
0.604
0.679
0.707
Closed-form
0.631
0.696
0.718
5.4 Ablation Studies
5.4.1 Effects of Different Model Components
We study the effects of each component of our model and different optimization strategies by running the experiments with different components included and excluded. All the ablation experiments are performed on "Center-Target" implementation. The corresponding retrieval performances on ImageNet are reported in Table 2 We use "Straight-throug""
to denote a model trained with the gradient of ag estimated by the Thrrouh trick, and "Closed-form" to denole the same model trained by optimizing the closed-form ELBO in Section 2.4 Each component ceapaueett-loose-eeee closed-form ELBO optimization, prior regularization, and UQ-based ranking
to improved retrieval accuracy. Optimal retrieval performance is achieved when all these elements are combined.
5.4.2 Sensitivity of Hyperparameter À
Our model ouly introduces one more hyperparameter for the trade-off between data belief and prior. To study the sensitivity of our model performance with respect X, we run our model with different values on ImageNet, with the corresponding retrieval accuracy reported in Figure 4a As long as À is set to be within appropriate range, the retrieval accuracies of ProbHash and HashUQ are not sensitive to X Moreover, our UQ based ranking stralegy can consistently improve the retrieval accuracy with all different À values and all different hash-code bit-length K's.
5.4.3 Comparison of Different UQ Measures
To compare different UQ measures in terms of the effectiveness in improving retrieval accuracy, we evaluate the UQ-based retrieval strategy using different UQ measures. The results on different datasets with K-bit hash-codes reported in Table More information about other two measures and comparison can be found in Appendix B.4 While the uncertainty quantified in all of the three measures can help improve the retrieval accuracy, our 1-test-based UQ consistently performs the best among all the measures on all three datasets with different K's. A possible explanation is the full consideration of two types of uncertainties in the adopted p-value compared to other measures. For example, Shannon's Entropy mostly just considers the uncertainty of b, while the summed variance only considers the uncertainty of sb. This further justifies our approach for the L2H task and our preference for the t-test-basod UQ methodology.
5.4.4 Effects of Number of Quantization Levels
We further study the effects of the number of quantization levels d discussed in Section 4.2 on the retrieval accuracy with our UQsba ranking strategy. We determine the quantization levels by making the same number of data samples to be discretized to each of the quantization levels. Figure 4b plots the retrieval accuracy by ProbHash and HashUQ with different mumbers of quantization levels when K = 32, We also include the performance of ProbHash with more bits for each case to compare the performance within the same storage. The corresponding figures with K 16 and K 64 can be found in Appendic B.3 While storing the quantified uncertainty with higher precision can lead to more accurate retrievals, with the best.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Retrieval accuracy of our UQ-based retrieval with different UQ measures: t-test based metric (denoted
as "t-test"), Shannon's 's Entropy ("Entropy"), and the summed variance of Bernoulli success probability ("Vari- ance") on ImageNet, MS COCO and NUS WIDE datasets.
Dataset
UQ Measure
16 bits
32 bits
64 bits
Entropy
0.628
0.695
0.717
ImageNet
Variance
0.617
0.688
0.712
I-test
0.631
0.696
0.718
Entropy
0.682
0.740
0.769
MS Coco
Variance
0.673
0.735
0.766
I-test
0.688
0.743
0.771
Entropy
0.814
0.833
0.842
NUS WIDE
Variance
0.795
0.824
0.837
1-test
0.814
0.834
0.843
0.70
0.70
ImageNet mAP@1000 1000
ImageNet mAP@1 1000 mAP@1000
0.68
0.68
0.66
0.66
HashUQ (Quantization)
HashUQ (No Quantization)
0.64
0.64
ProbHash
ProbHash (32 bits)
HashUQ
ProbHash (34 bits)
0.62
0.62
0
4
6
8
10
2
8
16
Quantization Level
(a)
(b)
Figure Retrieval accuracy of HashUQ with (a) different coefficient À values (b) quantization levels with K 32 on ImageNet.
performance when no quantization is implemented, our UQ-based ranking method can bring significant and consistent performance improvement to the implementation only considering the distances. They achieve better performance even with the binarized or quaternarized uncertainty, which only takes 1 or extra bits for storage and almost no extra computational overhead.
When we train deep hashing model with few more bits but it in the way without considering the quantified uncertainty, the benefit brought by the extra bits is almost marginal compared to uncertainty aware retrieval, as can be clearly observed in Figure 4b As model re-training is needed for HashUQ long as the base model being a probabilistic model, our HashUQ can be applied to systems whose storage allows us to have additional memory to include more quantization levels, which adds on the flexibility as more spareable resources can utilized to improve the accuracy and user experience. The reason that the extra bits may bring almost no benefit data retrieval can be possibly explained as the retrieval accuracy of an L2H model can be highly dependent on the minimum pairwise Hamming distance between pre-specified "center" vectors (Tan Hoe et al. 2021 over all pairs, whose largest possible value will rarely change when K only increases for a few more bits
hhtpps:/ http:://wwwwwnn.tute aeb/coddes/biaggg biinbbanaaaliiii html
12

---

Published in Transactions on Machine Learning Research (10/2024)
6 Related Work
and a database with N data points (af, a2...... ,x),, the main objective of information retrieval is to generate a ranking of all the pool data p (2?;; {=]))=1) for all the query data point, such that pool data similar to the query can get higher ranking order while pool data different from the query will be ranked lower in each of the predicted ranking. The Learning-to to to-Hash (L2H) based information retrieval methods achieve this by learning an encoding function f which can map the query and all of the pool data into K-bit. binary hash-codes bb".b..... b%} (Weiss et al., 2008 Salakhutdinov & Hinton 2009; Strecha et al., 2011 Liu et al.) 2011 2014 Li et al. 2016). The retrieval is then performed by ranking all the pool data points according to the Hamming distance of their hash codes to the hash code of the query.
Information Retrieval and Learning to Hash: Given query data
Pioneering works of L2H using traditional handeraft features include Spectral Hashing (Weiss et al. 2008) ) Linear Discriminant Analysis (LDA) Hashing (Strecha et al. 2011), and Graph Hashing (Liu et al.) 2011 2014). Semantic Hashing (Salakhutdinov & Hinton 2009) is among the earliest works using DNNs for hashing, where a two-stage procedure is developed to train deep auto-encoder for document retrieval in fully unsupervised manner. For high-dimensional complex data such as images and video, some early deep hashing models include Deep Hashing (DH) (Erin Liong et al. 2015), Deep Bairwise-Supervee
Hashing (DPSH) (Li ct al.] 2016), and Superssue Temporal Hashing (SSTH) Zhang ct al.) 2016). Rocent research on image hashing mostly focuses on solving the discrete optimization problem for principled training (Cao al. 2017
Su et al. 2018), reducing the information loss in code quantization (Doan et al. 2022 improving the loss
function for orthogonal and disentangled hash code generation (Tian Hoe et al.
2021
Doan et al. 2022). and
designing novel and flexible learning schemes (Kang et al. 2019 Yuan et al.
2020 Fan et al. 2020
Uncertainty Quantification in Deep Learning: The primary notion of uncertainty in deep learning is to use a probability distribution as the model prodiction instead of a point estimation. Most of uncertainty can be categorized into two major groups based on the sources and characteristics (Kendall & Gal 2017 Hüllermeerr & Waegeman 2021]: aleatoric uncertainty, which represents the uncertainty due the intrinsic randomness of the physical process and is typically modeled by either the softmax outputs in classification tasks or the Gaussian distributed outputs in regression tasks [Nix & Weigend 1994); and epistemic uncertainty, which is the uncertainty due to the lack of knowledge and is typically modeled by replacing the frequentist DNNs with Bayesian Neural Networks (BNNs) (Lampinen & Vehtari 2001 Titterington 2004 Neal 2012). BNNs model network parameters or activations as random variables whose distributions are learned through the posterior updates using Bayes' theorem. To solve the intractability of posterior inference in many situations, various approximate inference methods based on Markov Chain Monte Carlo (MCMC) (Neal et al., 2011 Welling & Teh 2011) and variational inference (VI) (Blei et al. 2017) have been developed, including the Monte Carlo dropout (MC Dropout) (Gal & Ghahramani 2016 Gal et al.) 2017 Boluki et al. 2020 Fan et al. 2021 and
Bayes-By- Ba-Baccprop (Bundell et al. 2015). Some UQ methods developed from frequentist's point of view include conformal prediction (Angclopoulos & Bates 2021 and quantile regression (Koenker 2005), with the point estimation replaced by an interval representing the confidence of the model prediction.
Our work differs from previous works of learning-to-hast in a way that our work is the first probabilistic framework for supervised hashing. We also propose a new measure of uncertainty designed for hashing basod student's 's I-test. which can simultaneously quantify the aleatoric and epistemic uncertainties. Additionally, our work is also one of the few works that demonstrate the potential applicability of quantified uncertainties improving real-world applications.
7 Conclusion and Discussion
In this paper, we propose ProbHash, a probabilistic framework for modeling hashing function in image retrieval along with a t-test based measure of uncertainty which jointly takes different sources of uncertainty into account. We further develop HashUQ, an uncertainty aware hashing strategy, and show that considoring the quantified uncertainty is an effective and efficient way to enhance retrieval accuracy with little computation and storage overhead.
13

---

Published in Transactions on Machine Learning Research (10/2024)
One drawback of our current work lies in the fact that approximate the posterior of network parameters and hash-oode using the variational distributions with the mean-field assumption to make it tractable, which has been shown to possibly underestimate the variance (Blei et al. 2017). Although adopting a deep neural network with its advanced representational capacity and fiexibility can mitigate this issuc, future rescarch directions include extending to more complex variational distributions with fewer restrictions, and exploring diffusion models' capabilities uncertainty quantification (Han et al. 2022 for hashing.
Broader Impact Statement
This paper presents a novel approach for image retrieval, contributing to the ongoing advancements in the intersection of machine learning, computer vision, and information retrieval. Our primary goal is to providing the capability of uncertainty quantification to hashing models, and further enhancing the efficiency and accuracy of retrieval systems. By doing so, we aim for valuable tools in applications ranging from content. organization to visual search.
The potential broader impact of our work extends to various societal dimensions. Improved image retrieval systems can have positive impact for fields such as content management, recommendation, and searching systems. However, we also recognize the responsibility associated with deploying such technology, especially concerning privacy, bias, and ethical considerations related to image content.
Acknowledgments
This work supported in part by the U.S. National Science Foundation (NSF) grants SHF-2215577 IIS-2212418, and 11S-2212419. Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Rescarch Computing.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavanzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, et al. review of uncertainty quantification in deep lcarning: Techniques, applications and challenges. Information fusion, 76:243-297, 2021.
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint 111:000000 2016.
Anastasios Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint wwX.::2107.07511, 2021.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.00 0478, 2014.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv preprint 1xXi::611.01100, 2016.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. arXiv preprint wwXiv:1802.01..0 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXir:1308.3432, 2013.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859 877, 2017.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference machine learning, pp. 1613 1622. PMLR, 2015.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Shahin Boluki, Randy Ardywibowo, Siamak Zamani Dadanch, Mingyuan Zhou, and Xiaoning Qian. Learnable bernoulli dropout for bayesian deep learning. In International Conference on Artificial Intelligence and Statistics, pp. 3905 3916. PMLR, 2020.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Hashnet: Deep learning to hash by contimuation. In Proceedings of the IEEE international conference on computer vision, pp. 5608-5617, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learningg pp. 1597 1607. PMLR, 2020.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. Nus-wide: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR 009), Santorini, Greece., July 8-10, 2009.
J. Dong, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Largo-Scale Hicrarchical Image Database. In CVPR09, 2009.
Khoa D Doan, Peng Yang, and Ping Li. One loss for quantization: Deep hashing with discrete wasserstein distributional matching. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pultern Recognition, pp. 9447 9457, 2022.
Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre Moulin, and Jie Zhou. Deep hashing for compact binary codcs lcarning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2475 2483, 2015.
Lixin Fan, Kamn Woh Ng, Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. In IJCAI, pp. 825-831, 2020.
Xinjie Fan, Shujian Zhang, Korawat Tanwisuth; Xiaoning Qian, and Mingyuan Zhou Contextual dropout: An efficient sample-dependent dropout module. arXiv preprint (rXic:2103.04181 2021.
Andrew Foong, David Burt, Yingzhen Li, and Richard Turner. On the expressiveness of approximate inference in bayesian neural networks. Advances in Nearal Information Processing Systems, 33:15897-15908, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout bayosian approximation: Representing model uncertainty in deep learning. In indernational conference machine learning, pp. 1050 1059. PMLR, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout, 2017. URL URLLttpp:://arrrrroo///bbb/
Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. CARD: Classification and regression diffusion models. Advances in Newral Information Processing Systems, 35:18100-18115 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a
overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457 506, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611001144, 2016.
Rong Kang, Yuc Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Maximum-margin hamming hashing. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 8252-8261, 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? A Advacces in Newral Information Processing Systems, 30, 2017.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1rXii:1412.6900 2014.
arXiv preprint
R. Koenker. Quantile Regression. Econometric Society Monographs. Cambridge University Press, 2005. ISBN 9781139444712. URL httts:://book: wow.ooo..o.ooog....ooooooooooo.. com/books woookss?.dwwoddaAAAA.....
Alex Krizhevsky, llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Jouko Lampinen and Aki Vehtari. Bayesian approach for neural networks review and case studies. Neural Networks, 14(3):257 274, 2001.
Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. In Proceedings of the Twenly-Fifth International Joint Conference on Artificial Intelligence, IJCAF16, pp. 1711-1717. AAAI Press, 2016. ISBN 9781577357704.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
W. Liu, Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. Hashing with graphs. Tn International Conference Machine Learning, 2011.
Wei Liu, Cun Mu, Sanjiv Kumar, and Shih-Fu Chang. Discrete graph hashing. In Z. Ghahramani, M. Welling, C. Corles, N. Lawrence, and K.Q. Weinberger (eds.), Advances Nearal Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL httpsss//prceeeingg..u.... www.:us..errreccoooeooo. httpppppp..c/appp/////////eeeee. paaapprrpfff..........
Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan. Reference based face super-resolution. IEEE Access, 7: 129112-12912,, 2019.
Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan. Photo-realistic image super-resolution via variational autoencoders. IEFE Transactions on Circuits and Systems for video Technology. (44):1155)))36, 2020.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A contimuous relaxation of discrete random variables. arXiv preprint arXic:6611.0011. 2016.
Radford M Neal. Bayesian learning for neural nelworks, volume 118. Springer Science & Business Media, 2012.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neurul Networks (ICNN'94), volume 1, pp- 55 60. IEEE, 1994.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perfopeerormance deep learning library, 2019.
Ruslan Salakhutdinov and Geoffrey Hinton. Semantic hashing. International Journal of Approzimate Reasoning, 50(7):969 978, 2009.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition. arx preprint ExXxx1140.. 1556, 2014.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Christoph Strecha, Alex Bronstein, Michael Bronstein, and Pascal Fua. Ldahash: Improved matching with smaller descriptors. TEFE transactions on paltern analysis and machine intelligence, 34(1):66-78 78, 2011.
Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cmn. Advances in neural information processing systems, 31, 2018.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arViv preprint arXiv: 170.. 00395, 2017.
Jiun Tian Hoe, Kam Woh Ng, Tianyu Zhang, Chee Seng Chan, Yi-Zhe Song, and Tao Xiang. One loss for all: Deep hashing with single cosine similarity based learning objective. arXiv e-prints, pp. arXiv 2109, 2021.
D Michael Titterington. Bayesian methods for neural networks and related models. Statistical Science, pp- 128-139, 2004.
Aaron van den Oord, Oriol Vinyals, and koray kavukeuoglu. Neural discrete representation learning.
I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnelt (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yucheng Wang, Mingyuan Zhou, Yu Sun, and Xiaoning Qian. Uncertainly-aware unsupervised video hashing. In International Conference on Artificial Intelligence and Statistics, pp. 6722-677 6740. PMLR, 2023.
Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In D. Koller, D. Schuur- mans, Y. Bengio, and L. Bottou (eds.), Advances in Newral Information Processing Systems, vol- ume 21. Curran Associates, Inc., 2008. URL https://proceeeding. www.igg...urrr..... wwtp:eeewererrrrrpeapeeeee http:/pcccccappp//////////
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICMI-11), pp. 681-688. Citeseer, 2011.
Mingzhang Yin and Mingyuan Zhou. ARM: SEAIIIERREIFOORC-EEe-eee gradient for stochastic binary networks. In International Conference Learning Representations, 2019. URL htttss:/
net/ffrrmmidd-l1g0JJee..
Mingzhang Yin, Nhat Ho, Bowei Yan, Xiaoning Qian, and Mingyuan Zhou, Probabilistic best subset selection via gradient- based optimization. ar preprint arxiv:2006. 06448, 2020.
Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jic, Wei Liu, and Jiashi Feng. Central similarity quantization for elficient image and video retrieval. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 3083-3092, 2020.
Hanwang Zhang, Meng Wang, Richang Hong, and Tat-Seng Chua. Play and rewind: Optimizing binary representations of videos by self-supervised temporal hashing, In Proceedings of the 24th ACM international conference on Multimedia, pp. 781 790, 2016.
Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. Deep hashing network for efficient similarity retrieval. In Proceedings of the AAAI conference Artificial Intelligence, volune 30, 2016.
17

---

Published in Transactions on Machine Learning Research (10/2024)
In the appendix, we first include brief introduction to Hadamard Matrices, which are commonly used to construct hashing centers in previous "Center-Target" methods in Section A.1 We then provide the step-by-step derivation of the training loss function for ProbHash with LLabl--Targer construction and experimental comparison with other likelihood choices Section A.2 and Section A.3 respectively. We also include detailed evaluation pipeline, additional experimental results along with an empirical comparison of different variational distributions for Bayesian neural networks Section In Section D we provide comparison of some exemplar image retrieval results with and without uncertainty quantification. Lastly in Section E provide histograms and Q-Q plots of the difference between paired samples of each Bernoulli success rate of hash-code vectors generated using 5 query images from the ImageNet dataset to support the applicability of paired samples t-test on image hashing problems.
A SUPPLEMENTARY INFORMATION FOR MODEL CONSTRUCTIONS
A.1 Brief Introduction to Hadamard Matrices and Its Application on Learning-to-Hast
A Hadamard matrix square matrix dimension 2 , E Z+ which can be defined by induction:
H1 [1]
1
H2 = -1
(6)
H2
H2 -22--1
H2 H24-1
Each row or column of a Hadamard matrix orthogonal all other rows or columns. The row vectors of matrix H -H] where H is Hadamard matrix of dimension 2 will form set of 2 dimeenniioaal vectors
with the thcoretically maximum minimum mutual Hamming distance, which has been empirically shown to affect the retrieval accuracy of "Center-Target" methods (Tian Hoe et al. 2021). These Hadamard matrices have been used to construct the hashing centers previous "Center- "Center-Target" methods (Yuan et al. 2020
Tian Hoe et al. 2021: Doan al. 2022
A.2 Derivation of Expression in the Main Text
We provide the detailed derivation of Expression (L) in Section 2.4 of the Main Tert. The derivations corresponding to the "Label-Target" construction and "Center- Teanter-Targe'' construction be casily obtained by replacing p( [am
vb) with Ma J 1b) and
log
U1 Mi N
2 EEg(,)
log p( p((yi bb) DKL (d()))p((((
-kg(()) log / p((u)) (b.) (2 / )dd(bb)
DE (x))
-E,(w)
log
16. P(
bddbb
b){(b,
DKL (2ww))
NV >Eg(w)
N
DKL (a
A.3 Choice of Likelihood for "Center- " Construction
In Bayesian statistics, the likelihood represents our beliefs about what data we expect to see for each setting of parameters of the model. The choice of likelihood distribution will depend the data type well as the
18

---

Published in Transactions on Machine Learning Research (10/2024)
convenience of computation. The likelihood function should take the higher value when data is consistent with the model while the lower value when the data is inconsistent with the model. For example, the isotropic Gaussian distribution is typically chosen for continuous data, and the categorical distribution is typically used for modcling the discrete data. Both belong to the exponential family and have easy-to-compute log- ligliikeihood lunction. We choose the likelihood distribution:
p(c:(b,) HaaaiDiiaaGaa.... (C,; (c);
for the following reasons:
This probability distribution matches with our aforementioned principle for the likelihood choice: Both C; and b, are binary vectors and the Hamming distance is indeed a proper choice of distance measure for binary vectors. The adopted likelihood distribution reflects our belief that the hash-code center e; corresponding its class-label will be more likely to have small Hamming distance to the unobserved hash-code b, Our adopted likelihood distribution takes the highest value when C, matches with b, and decreases when C, gradually deviates from b,-
This probability distribution belongs to the commonly adopted likelihood distributions: Just. like Gaussian or categorical distribution, our adopted likelihood distribution a Boltzmann distribution, which is one of the most widely used probability distribution in statistical analysis and machine learning modcling. Our adopted distribution also belongs to the exponential family, which has a simple form of log likelihood function.
This probability distribution has computational advantages over other choices of like- lihood distributions: This specific choice of our likelihood distribution will also lead the closed-form optimization objective as emphasized in Section 2.4 of our manuscript. The benefit of this closed-form objective is two-fold: (1) it reduces the computational consumption for conducting multiple forward Monte Carlo samples; (2) it is unbiased and will reduce the variance for stochastic gradient descent, which will lead to more efficient model training.
We compare our adopted likelihood distribution with another Boltzmann distribution:
p(c](b,) exp(
(0l-oDj DD b:))
This distribution takes similar format Gaussian distribution, and the closed-form log- likelihood can also be derived as:
Eg( |- log (exp(c;bbb)
(e [alc: bil² 2 log 2]
-Eg(b.ler))
ci, bii log
(7)
K
c + 20r +1 402,60 (fr. (2:))) log Z.
This means that this distribution also satisfies all the three principles for selecting likelihood function mentioned previously. The main difference between the distribution above and the likelihood distribution we adopt in the main paper is how the probability decays as the Hamming distance gets larger. We run supplementary experiments on ImageNet dataset lo compare the distribution above (D) and the likelihood distribution we adopt in the main (aperr(Homming))) and include the results in Table g We observe the retrieval accuracy to be slightly affected with the likelihood distribution in the main paper replaced with the likelihood discussed above. The experiment results also show that our HashUQ strategy can help improving the retrieval accuracy regardless of the likelihood choice.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Method
16 bit
32 bit
64 bit
ProbHash- D Fammming
0.606
0.680
0.706
HashUQ- D amminn
0.631
0.696
0.718
ProbHash-1D Famming
0.606
0.678
0.705
HashUQ-Di Hamming
0.631
0.692
0.716
Table 4: Comparison of different likelihood distributions for "Center- "Center-Target" Construction.
B SUPPLEMENTARY INFORMATION FOR EXPERIMENTS
B.1
Supplementary Information for Evaluation Pipeline
B.1.1 Datasets
We empirically evaluate image retrieval performances based on different supervised hashing methods to demon-
strate our uncertainty aware HashUQ's superiority on three benchmark image datasots: ImageNet (Deng et al. 2009). MS COCO (Lin et al.) 2014), and NUS WIDE (Chua et al., July 8- 8-10, 2009). Each dataset has been further split. into training, query and pool (database) sets, with the details given below:
ImageNet is large-scale image dataset (Dng et al. 2009), which contains more than 10,000,000 images labeled with 20,000 different synsets in WordNet. We follow Cao et al (011 and use the images from 100 selected categorics for training as well as evaluation.
MS COCO is image dataset with more than 200, 000 labeled images from 80 categories (Lin al. 2014h. Multiple labels are typically associated with one image instance, which makes the retrieval task typically more challenging than ImageNet. We follow Zhu et al. (2016) and use a total of 132,218 data samples for training and evaluation.
NUS WIDE an image dataset with total of 269, 648 data samples collected from Flickr, labeled with 81 ground-trute truth concepts (Chua et al. July 8-10, 2009h. We follow [Cao et al. (2017) and use images from the 21 most frequent concepts for model development and testing, with each concepts containing at least 5000 images.
One major difference bet ween our evaluation pipeline and those adopted in previous works is that we explicitly include a validation set for model selection while most of the previous works does not differentiate the
validation set with the test set (Cao et al., 2017 Su al.) 2018 Yuan et al., 2020 Fan et al. 2020 Tian Hoc
al. 2021
The dataset statistics with the adopted data splits summarized in Table
Table 5: Statistics of data splits in ImageNet, MS COCO, and NUS WIDE datascts.
Dev.
Test
Datasct
# cls
Train.
Val.
Query
Pool
ImageNet
11700
1300
5000
128,503
100
MS Coco
9000
1000
5000
117,218
80
NUS WIDE
9000
1500
2100
193,734
21
B.1.2 Hyperparameters and Training Details
We use the AlexNet Krizhevsky et al. 2012) backbone for all the experiments be consistent with the evaluation pipelines of previous works Cao et al. (2017); Su et al. (2018); Yuan et al. (2020): Tian Hoe et all (2021); Doan et al. (2022). Three extra fully connected layers with the latent dimensions 4096 are added at the end of neural network backbone. We regularize the last three layers with dropout in each of the hidden layer for all the methods benchmarked, with the neurons reweighed deterministically in baselines and dropped randomly in our ProbHash and HashUQ at test time. The the dropout rates are set to be 0.5. We optimize the neural networks using RMSprop (Hinonn et al. (2012) optimizer with the learning rate 1e-5 and weight decay le for all the models. We use derived closed-form ELBO the minimization objective
20

---

Published in Transactions on Machine Learning Research (10/2024)
for Ceanter-Target construction and estimated ELBO using Monte-Carlo sampling the minimization objective for "Tageel-Tarett" construction with the Straight-Through (ST) trick to estimate the gradient of We choose fair Bernoulli distribution p(b, = 1) = p(b., = -1) = 0.5 as the prior for each entry of the hash-codes and set as max; E log (x())) in all of our simulation. The coefficient balancing between data-fitting and prior belief À is set to be 1.0 unless specified. Notice that the Φ in "center- "center-target" construction can be absorbed into X, and we use a fixed @ = 2 throughout the paper without loss of generality. We perform model evaluation using 100 samples from the learned dropout variational distribution. The best performing model on the validation sets during the first 100 training epochs are chosen to be evaluated on query and pool datasets.
discontinuous function 8b
B.2 Supplementary Results for Uncertainty Quantification
1.0
1.0
3
Rrcction
0.6
0000
Aeerage
0.4
0.4
000
0
0.0
0.0
0%-220%
20%-4406
4016-00%s
60%6-80%
80%-100%
0h5-20%
20%-000%
601--60%
60%-80%
80%-100%
Quantlle Hash-code Uncertainty
Quantile of Hash code Uncortainty
(a) Center-Target, Bits
(b) Center- Target, 64 Bits
1.0
0.8
0.8
Average Preces 1000
80
CUD DO
Average
0.4
0,4
0.2
076-207
20%-4376
40%-400%
60%6-80%
BON-100%
0%-20%
20%-400%
60%
60%-80% 80%-100%
Quartille Hash-code Uncertainty
Quantil of Hash Uncertainty
(c) Target, Bits
(d) Label- Target, 64 Bits
Figure 5: Boxplot of retrieval accuracy with respect to the quantile of hash-code uncertainty with = 16, K = and K of our ProbHash with "Center-Target" and "T.abel- "arree" constructions on ImageNet.
We present more results showing the uncertainty quantification capability our proposed method supple- mentary to experiments in Scctions 5.3 of the Main Tert. The boxplots of retrieval accuracy for the query images with respect the quantile of their predicted hash-code uncertainty with different constructions with K = 16 and K 64 provided in Figure 5 We can observe similar trend in the Main Text: A higher
21

---

Published in Transactions on Machine Learning Research (10/2024)
estimated uncertainty of hash-code will be typically predicted with the present of query with erroncous retrieval results.
B.3 Supplementary Results for Ablation Experiments
We include more experimental results supplementary to the experiments Sections 5.4.2 and 5.4.4 of the Main Test. We plot the retrieval accuracy with respect to different coefficients in Figures Ga and 6b and the ones with different numbers of quantization levels in Figures 6c and 6d on the ImageNet dataset with K = 16 and K 64. We observe similar trend in Sections 5.4.2 and 5.4.4 of the Main Text: Both of our ProbHash and HashUQ provide retrieval performances insensitive to X, long as À is set within appropriate range. Our HashUQ can consistently provide performance improvement over ProbHash regardless of X, K and quantization level settings. Our HashUQ also brings more retrieval accuracy improvement compared to a model without considering the quantified uncertainty as the allowed bits increase.
0.64
0.74
ProbHash HashUQ
ImageNet mAP@1000 @10000
0..2
ImageNet mAP@) 1000
0.72
0.60
0.58
0.70
0.56
0.68
0.54
ProbHash
HashUQ
0.52
0.66
10
A
A
(a) K = 16
(b) K 64
0.64
0.74
ImageNet @10000000
0.62
0.72
0.60
ImageNet mAP@
0.58
0.70
HashUQ (Quantization)
HashUQ (Quantization)
0.56
HashUQ Quantization)
HashUQ (No Quantization)
0.68
0.54
ProbHash (16 bits)
ProbHash (64 bits)
ProbHash (18 bits)
ProbHash (66 bits)
0.52
0.66
2
4 Quantization Level
8
16
16
Quantization Level
(c) K 16
(d) K 61
Figure 6: Retricval accuracy with respect to different (a) coefficient: À values with K = 16 (b) number of quantization levels with K = 16 (c) coeflicient. values with K = 64 (d) number of quantization levels with = 64 on ImageNet.
B.4 Supplementary Information for Different Metrics
We here provide more information about the comparative study different metrics for measuring the uncertainties hashing complementary to Scction 5f4.3 of the Main Text. Given a bunch Bernoulli success rates ( fns Ngample one way measure the uncertainty is calculate the variance of nh: Eg E (<=====<<i (=(x>()) This type of uncertainty measure will mostly quantify the effect of the uncertainty of variational distribution
22

---

Published in Transactions on Machine Learning Research (10/2024)
Method
16 bit
32 bit
64 bit
ProbHash-FFF
0.597
0.647
0.662
ProDHash-MCD
0.606
0.680
0.706
HashUQ-FFG
0.633
0.677
0.686
HashUQ-MCI
0.631
0.696
0.718
UQshUQ-PFG-
0.627
0.673
0.678
HashUQ-MCD-BB
0.623
0.690
0.714
Table 6: Comparison of Fully Factorized Gaussian weights (FFG) and MC Dropout (MCD) inference for ProbHash and HashUQ on ImageNet dataset.
of model parameters vp on the Bernoulli success rates n² We report the experimental results with proposed HashUQ based retrieval on different. image datasets in Table 3 which we denote as Variance".
Another way to quantify the uncertainty considering the adopted factorized Bernoulli distribution for hash- code the Shannon's 's Entropy, which has been adopted in |Wang et al.] (2023). As no uncertainty of & is considered in Wang al. (2023), here we generalize the metrics for our method and consider the Shannon's Entropy of q(b)á, b)): (( +b) = 1 (()a)))))))))) log (6) vb)]]. This will measure the uncertainty of hash-code 6* averaged over the distribution of neural network parameters (( We also test this metric with the results denoted as "Entropy" in Table 3 In both of the above cases, we sum the variance and entropy over all the K entries to have an aggregated notion of uncertainty for hash-code of one input image considering the factorized assumptions of our variational distribution.
B.5 Comparison of Different Variational Distributions for Bayesian Neural Networks
Last but not least, we provide an empirical comparison between dropout, the variational distribution we adopted, to another widely used distribution for Bayesian neural network approximation: Fully Factorized Gaussian weights (FFG). For each entry Wi,i, of neural network weight matrix Wi, the independent Gaussian distribution is assuned:
Wf. N Wij PL..j. TRAs)
with {prs.d. } as the variational parameters. We specifically adopt the Bayes- By--Byyy-BBBopppp (Blundell ict al. 2015) implementation for Fully Factorized Gaussian weights, in which the variational distribution is reparameterized using the standard Gaussian distribution and variational parameters optimized by maximizing the Evidence Lower BOund (ELBO) of training data through gradient-based optimization. We model the weight matrices of the last {1, 2, fully connected layers as factorized Gaussian and set the learning rates to 10 10-4). A standard Gaussian prior is adopted and {1, 5, 10) Monte Carlo samples
be {1 x 10-5,6 5 x 10-5, are used to reduce the variance of gradient estimation, with the best performing model in the aforementioned configurations on ImageNet dataset choosen to be reported in Table
The quantified uncertainties of Fully Factorized Gaussian weights can similarly be used to help enhance the retrievals as the dropout weights. We observe worse empirical performance of Fully Factorized Gaussian weights compared to dropout in most of the sctups with or without using uncertaintics. In the meanwhile, the performance improvement due uncertainties quantified from the Fully Factorized Gaussian weights is slightly more significant compared to dropout. While some more flexible distribution families may better approximate the posterior and model the uncertainties (Foong et al.) 2020). it also brings computational challenges in optimization, which may require further analysis on the ELBO maximization objective and gradient based optimization methods.
B.6 Comparison between Two Strategies: Prioritizing Confident Pool Data or Prioritizing Uncertain Pool Data
In the Table 7 we include the retrieval accuracy of our ProbHash, HashUQ with the confident pool data prioritized (HonsiUQ)Confideet)) and HashUQ with the uncertain pool data prioritized
23

---

Published in Transactions on Machine Learning Research (10/2024)
Method
16 bit
32 bit
64 bit
ProbHash
0.606
0.680
0.706
HashUQ- CoCofident
0.631
0.696
0.718
HashUQ- QUnnertai
0,563
0.656
0.689
Table 7: Comparison between two strategics: prioritizing confident pool data or prioritizing uncertain pool data.
100
Probabiiiy Predicting
Probabiity Predicting
Frobs of Predicting "I
12-
10-
Frequency
60
Frequencyy
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Probability
Protability
(a) Case A: the posterior samples always predict the (b) Case B: the values of posterior samples fluctuating probabilities 0.3 and 0.7.
from (0.2, 0.8) 00.4,0.6).
Figure 7: Illustaaioon of exemplar scenario considered where the t-test can be effectively used to rank uncertainly.
ImageNet. We observe worse retrieval accuracy of Hash U Uncertain compared to HashUQ-Cooffident and ProbHash. This implics that the uncortain pool data is more likely to be irrelovant to the query compared to not only the confident ones, but also most of the other pool data between when Hamming distances are small.
C SUPPLEMENTARY ANALYSIS OF UNCERTAINTY MEASURES
C.1
Comparison between (-test and Shannon's entropy based measures of uncertainty
The paired sample t-test measures whether the population mean of two groups of samples equal. Here connect the estimated uncertainty by I-test statistics to Shannon's 's entropy based measure of uncertainty. We present an exemplar scenario where the two-sample t-test can be effectively used to rank uncertainty:
In Case A, for a given input, the posterior samples always predict the probabilities of predicting '-1' and '1" as (xt,:: m (0.3, 0.7).
In Case B, while the average probabilities remain (0.3,0.7), the values of (T' x"(e) fluctuate, ranging from (0.2,0.8) 0.64,
Figure 7 illusttaate the two cases in the scenario considered. Clearly, the uncertainty in Case B is greater, and the two-sample t-test captures this difference effectively while Shannon's entropy do not capture this increased uncertainty in Case B.
24

---

Published in Transactions on Machine Learning Research (10/2024)
Paired sample t-test based Uncertainty Measure: We conducted the paired sample t-test with the null hypothesis Ho: T. = x.k. The 1-statistic of our adopted paired sample test is calculated as follows:
T T
t(T t(T =*) Std(k', T. DIVN
where T', - x², and s =' is the number of samples. The uncertainty of the k-th entry is quantified the tail probability of observing ((T..)) under the null hypothesis Ho:
represent the sample mean and standard deviation of (F - "(,))) N
mall((('',,s = P(t(( ="))) 2 "T m"x) ~ Ho)-
We use the log scale, log (p r specifically as the measure of uncertainty for k-th entry.
Uncertainty Measured using Conditional Shannon's Entropy: We calculated the conditional Shannon's entropy as follows:
E E (b(bb,,, J log + (1 10-m,ppa
T',x) log(1 x
Some previous works use this conditional Shannon's is entropy to measure aleatoric uncertainly.
Uncertainty Measured using Total Shannon's Entropy:
H(blz) R - I'''.log'''.... (1 11--t log(( =
Some previous works use this total Shannon's entropy to mcasure total uncertainty. An induced epistemic uncerlainly can be calculated by substracting total Shannon's Entropy to conditional Shannon's i's entropy:
Hbblz) Ex-a(v) (blb))) x)]]
Connection and Difference: Both of the t-test and Shannon's entropy give the highest estimation of uncertainty when the model consistently predicts ,., x 0.5 (corresponding high aleatoric uncertainty) while the lowest estimation of uncertainty when the model predicts π. 0 or πτ, = 1 (corresponding to low aleatoric uncertainty). When the Shannon's 's Entropy based mcasure of uncertainty mentioned above is adopted, the total uncertainty do not change and the alcatoric uncertainty decrease as the variation of æ increase. The uncertainty measured using 1-lest will increase when the variation of æ increase. We include a numerical comparison between paired sample t-test and Shannon's entropy based measures uncertainty using 100 samples from truncated Gaussian distributions Figure and to show how cach of these two uncertainties change as the mean 7' and variance Var(=') changes.
D RETRIEVAL EXAMPLES
We provide retrieval examples of our ProbHash and HashUQ in Figures 10a. and 10b using 5 exemplar query images from TmageNet. On the left-most of each row is the query image, and the corresponding retrieved images are plotted on the right-hand-side of the dividing line. We plot each retrieved images relevant to the query in green bounding boxes, and images irrelevant to the query in red bounding boxes. To illustrate the main idea and show the difference ween the predicted ordering of the retrieval models with and without uncertainty, here we number of retricved samples r 10. (This means that we only retrieve 10 images from all the 128, 503 entries in the database based on the hash-codes Hamming distance for each query, and re-rank these 10 images by the predicted uncertainty of hash-code accordingly.)
By comparing Figure 10a and Figure 10b we see that our HashUQ will prioritize images with relevant concepts to the query while de-prioritize images with irrelevant concepts, which lead to more principled retrieval ordering. In real case, by rank the data with the same hash-code Hamming distance to the query using the predicted uncortainty, the relevance of top data to be retrieved will also be improved, and the retrieval accuracy will be affected not only by the retrieval ordering but also in terms of the data to be retrieved.
25

---

Published in Transactions on Machine Learning Research (10/2024)
-50
p-value
150
0.6
200
0.4
Total
300
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.00
0.09
(a)
(b)
0.30
1.0
Conditonal Shanren's Entrpy
Uncertainty Sharnon's
0.08
0.06
0.6
0.4
0.2
0.02
0.04 0.06
0.09
0.02
0.05
0.07
(d)
Figure 8: Uncertainty measured with (a) paired sample t-test, (b) conditional Shannon's entropy, (c) total Shannon's 's entropy and (d) induced epistemic uncertainty based Shannon's entropy using 100 samples from Gaussian distributions N(H,a²) with respect to standard deviation o changing from 0.01 to 0.1. The mean = 0.3 and the Gaussian distribution is truncated at (a, b) with the minimum and maximum value set to be a = 0.2, = 0.4.
26

---

Published in Transactions on Machine Learning Research (10/2024)
-50
p-value
150
0.6
200
0.4
Total
300
0.10
0.15
0.25
0.30
0.35
0.40
0.45
0.10
0.15
0.20
0.25
0.30
0.35
0.45
(a)
(b)
0.30
1.0
Conditonal Shanron's Entrpy
Uncerfainty Sharnon's
0.08
0.06
0.6
0.4
0.2
0.02
(d)
Figure 9: Uncertainty mcasured with (a) paired sample t-test, (b) conditional Shannon's entropy, (c) total Shannon's entropy and (d) induced epistemic uncertainty based on Shannon's entropy using 100 samples from Caussian distributions. NN,,, x²) with respect to mean changing from 0.1 to 0.5. The standard deviation = 0.05 and Gaussian distribution is truncated at (a, b) with the minimum and maximum value (a, b) set to a = p 0.1, = 4 + 0.1.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Que
Duery
Query
Quarry
Query
Query
Quary
Query
(b)
Figure 10: (a): Exemplar image retricval results using 5 query images from the ImageNet datasct. The retrieval performed without considering the uncertainty of hash-codes (b) Image retrieval using the same queries with proposed HashUQ, On the left side of the dividing line is the query image while on the right side of the line is the retrieved data, with the decreasing predicted relevance from left right. We plot success retrievals in green bounding boxes and failure retrievals in red bounding boxes. Below each retrieved image is the Hamming distance of the hash-code to the query image. To illustrate the main idea, here we only retrieve 10 images for each query and re-rank them based on the quantified uncertainty.
28

---

Published in Transactions on Machine Learning Research (10/2024)
E
SUPPLEMENTARY FIGURES
-41
(a)
sanga Quaran
-13
45
33
.23 -11
04
Thenetica
Thenscaiaarii
68-
60
64
05
66 -01
83 01
03
84
05
82 21.
62
05
26
37
02 24
82 84 85
AT
CO.
36-
-30
-25
-044
-02
-24
-32
O
04
-55
-33 -32
32
31
-6.2
G1
CUREM
(b)
29

---

Published in Transactions on Machine Learning Research (10/2024)
of
-44
-22
-67
04
51
22
01
-35
-33
-04
+43
22
-31
-63 -04
-62
-55
CA 05
4.7
+5
-67
83
Transue QAN2OI
(d)
30

---

Published in Transactions on Machine Learning Research (10/2024)
193-
(e)
2.7
33
Saven
-36
-37
-17
02-
(sackia Garchs
C.
(5
43
61
-07
35
45
20
-22
-55
-5
-6
-0
-2
01
05
63
CA
66
03
04
65
-22
12-
04
05
0A
24
28 -A .27
-14
(222
05
16
85
FA
32
5.5
Sveres
C3
43
-24
Taaetia GARERA
(f)
31

---

Published in Transactions on Machine Learning Research (10/2024)
10
6%
(g)
-020
2.4
33
-359
-355
-35
-24
-23
-10
-22
C5
34
-55
-34
-33
33
45
31
-22
-24
.11
-21
-53
-22
31
65
-21
-21
.32
07
05
64
65
635
-0.5
C39
-22
-0.0
-645
025
41 C2
22
-04
-33
-12
035 c5 Lit
ado 150
-54 -33 <32
0.5
2.4
5.6
626
-42
-24
03
(h)
32

---

Published in Transactions on Machine Learning Research (10/2024)
AB
4
32
Dffecie aend
of 200
ii
6.1
-02
Semple Guvnie
-044
12
-05
00
06
05
203 21 -22 -0
-2
203
-22
-41
-28
-03
-22
60-
35-
65
-11
02
0.
04
-24 -x.
-41
02
01
24
30
-63-
60-
44.
<55
-24
-3 -1
-2.
-05
-32
01
02
CA
65
26
43
-3
05
C4
Transed Cuatlnl
Thanesalqaraaa
(i)
33

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 11: Histograms and Q-Q plots the difference between paired samples of each Bernoulli success rate of hash-code vectors generated using query images from the ImageNet dataset. (a)))) Histograms of the difference between paired samples. (f)Ld) Q-Q plots of the difference between paired samples with respect to the Gaussian distribution with the same mean and variance. We use the same images to generate the plots in Section D with K = 16.
34


Reviews:
[{"review_id": "deIANaBg9G", "reviewer": "Reviewer_79if", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper proposes a novel method that leverages model uncertainty to improve image hashing for retrieval problems. The contribution and the motivation of the method are well justified.\n\n- Sufficient experiments and ablation studies are performed to support the arguments.\n \nWeakness:\n\n- The description of the methodology is not quite clear. The paper proposes two methods ProbHash and HashUQ, but ProbHash is actually a part of HashUQ. It would be better if they were described and compared in the form of ablation instead of two methods.  Also, the logic is not quite coherent. Sections 4 and 5 are titled with \"Quantifying the Uncertainty of Hashing\" and \"Hashing with Uncertainty Quantification\", which talk about the same thing but are split into two sections. \n\n- The ambiguities in the paper make it hard to reproduce. In Sec 3.4, how is prior $p(b)$ selected? How does it impact the performance? In Sec 5.1 \"$\\alpha$ is a positive real number small enough such that...\", what is $\\alpha$ and how is it set?\n\n- Some general questions about the problem setup - For this image hash problem, will training a classification network and then hash the predicted class better than directly predicting hash codes? Will codes with 4 8-bit classes perform better than 32 binary codes?\n\n- HashUQ requires significantly more computation for inference on the image hash database, making the comparison with the baselines not truly fair.\n\n- Why does Table 1 use bold font on the best model except for full HashUQ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel image hashing method to improve retrieval tasks by quantifying model uncertainty. The model ranks by both distances between the query image and images in the database and the uncertainty of query images. The model uncertainty is quantified by sampling from a Bayesian network. The proposed method obtains better performance on three datasets of image retrieval.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please provide more details of methods and improve writing to clarify the ambiguities.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yp81xGwRwR", "reviewer": "Reviewer_Qmeu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe paper contains two strong ideas:\n- Using a variational distribution over the hashes, which, in the \"center-target\" variant of the method, allows the authors to analytically evaluate the model's expected likelihood. Therefore, the gradient of the expectation is also available exactly, and there is no need for a straight-through estimator, which is both biased and has a higher variance.\n - Using uncertainty estimates to augment information retrieval.\n\nI also found the experimental evaluation and ablation studies mostly satisfactory. I appreciated that the authors paid attention to their method's increased storage requirements and, therefore, attempted to make fair comparisons with competing methods.\n\n## Weaknesses\nMethods-wise, I have three main concerns:\n- The authors do not compare Monte Carlo dropout to 1) only using dropout to regularise training or 2) to a fully-factorized Gaussian weight posterior. These comparisons would be important for two reasons: 1) doing variational inference is generally more expensive than using point estimation, so it would be good to establish whether the extra computation is worth it, and 2) Monte Carlo dropout as a variational approximation has well-known shortcomings for uncertainty quantification (see, e.g. [1]). Hence, it would be good to see if using a better approximation could be used to improve performance.\n - There is no motivation for using the t-test as the basis for uncertainty quantification; measuring entropy instead seems a lot more sensible. Based on the paper's contents, my impression is that the authors chose it because it performed slightly better than other measures in the experiments they conducted (though the statistical significance is unclear). One sentence I found particularly problematic was the authors saying, \"We find [hypothesis testing] to be especially suitable for quantifying the uncertainties of hash-codes considering the nature of the problem.\" - there is no explanation of what particular nature of the problem should make it suitable.  \n - There isn't a clear motivation for the authors' ranking criterion. While using the uncertainty values as a secondary ordering for data whose hashes collide is certainly sensible, it also seems arbitrary. Could the authors provide a principled justification for why ordering data this way should work better in some cases than something else? Furthermore, one dissatisfactory thing about the authors' method is that the training and the test/retrieval procedures seem to differ. Concretely, the authors use the uncertainty values to extend the hashes of the data in the pool set but don't use them for the query point. Could the authors comment on this?\n\nContent-wise, the method appears to be an instance of the variational bottleneck framework [2], but the authors do not discuss this in the related works section. Could the authors comment on this?\n\nFinally, the paper's presentation needs improvement. Many statements are imprecise, have information missing or are potentially misleading:\n - In Section 3, when presenting the necessary background, the authors note that the methods they base their work on use the \" binary quantization function $\\mathrm{sgn}$\". The standard definition of this function maps to $\\{-1, 1\\}$. The next paragraph mentions that they generate the centroids for their method's \"center-target\" variant using a Hadamard matrix. Firstly, the authors should define what they mean by the Hadamard matrix in the paper, as I don't think it is a widely-known object. Furthermore, there are two conventions for these matrices, one which defines that the entries are $\\{-1, 1\\}$ (the one I was familiar with) and $\\{0, 1\\}$. However, the authors adopt the $\\{0, 1\\}$ convention in the rest of the paper, so I found the whole method quite confusing on the first couple of reads; please update the paper to avoid this confusion.\n - The main text does not define the performance metric the authors use in the experiments (mAP@1000).\n - Table 2 uses unnecessary abbreviations (Opt, ST, CF) that are not defined in the table caption. Please just spell out the relevant phrases.\n - The authors below Eq (2) state, \"λ is a hyperparameter to reflect our belief of the data relative to the prior.\" I don't understand what this should mean; could the authors please clarify? On a related note, do the authors tune the prior parameters?\n - The related works section (Section 2) is essentially just a literature review that does not specify how the works mentioned are related to the authors' work or how they differ; please clarify this. Furthermore, I found this section quite dense straight after the introduction; I suggest the authors move it after the experiments section.\n\n## References\n - [1] Foong, A., Burt, D., Li, Y., & Turner, R. (2020). On the expressiveness of approximate inference in bayesian neural networks. Advances in Neural Information Processing Systems, 33, 15897-15908.\n - [2] Alemi, A. A., Fischer, I., Dillon, J. V., & Murphy, K. (2016). Deep variational information bottleneck. arXiv preprint arXiv:1612.00410.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider using neural networks to learn efficient hash functions for information retrieval using labelled data (specifically labelled images in the paper). The authors propose a variational information bottleneck architecture: the encoder network maps data to binary vector-valued latent random variables, which serve as the basis for the hash of the data. Then, during training, a separate decoder network aims to reconstruct the label or a proxy for the label; the authors specifically map the labels to certain error-correcting code vectors in the hash-space, simplifying the decoding and training procedure. \n\nThe authors propose using Monte Carlo dropout (leaving dropout turned on during inference time and averaging results over multiple forward passes through the network) to quantify the epistemic uncertainty in their network's prediction. Furthermore, they develop an information retrieval procedure that considers the predicted uncertainty, which ranks data that map to the same hash based on the model's confidence in the prediction. The authors also account for their method's storage requirements and experiment with different quantization levels for the predicted uncertainty, which they save along the hash of each data point.\n\nThey empirically verify the proposed approach's efficiency on three image datasets, and their method is competitive with state-of-the-art ones. Moreover, they conduct several ablation studies to show the value brought by each new component they propose.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Necessary:\n - Please address the concerns I outline above, and update the paper accordingly.\n - Additionally, please increase the line thickness in the Figures, as they are currently hard to read on a laptop screen without zooming in.\n \n## Nice to have:\n - The presentation of the \"label-target\" and \"center-target\" variants of the method are not separated cleanly enough; please signpost them better using bold paragraph headings or putting them in different subsections. I would actually suggest that the authors present only a generic framework and specialize the algorithm at the end of the section. \n\n## Typos/style:\n- introduction: \"uncertainty-awared\" -> \"uncertainty aware\"\n- Please format the argmax above section 4 so that the optimization variable is below the word \"argmax\" and not to the side.\n - notation: $x^q$ clashing with $x^k$,  please use a double subscript $x_{i,k}$ instead\n - first sentence of section 3.4: \"$x$ and $\\psi$ and $p(\\psi)$\" - writing $\\psi$ not needed.\n - In many equations (e.g. eq (1)): please use `\\left[` and `\\right]` for the brackets.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dh8iwITovw", "reviewer": "Reviewer_3tbV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method shows that considering the quantified uncertainty is an effective and efficient way to enhance retrieval accuracy.\n2. It is natural and intuitive to use statistical hypothesis testing as the measure of the uncertainty quantification of hash-codes.\n3. Numerical experiments and analysis demonstrate the effectiveness of considering the quantified uncertainty in improving the retrieval performance.\n\nWeaknesses:\n1. The proposed method assumes that the likelihood distribution of each class follows a categorical distribution parameterized by a neural network with the softmax activation and the variational posterior follows a factorized Bernoulli distribution parameterized by a neural network with the sigmoid activation function. However, the rationality of the assumption that these random variables follow various distributions has not been explained. \n2. The rationality of using neural networks to model the parameters of these distributions is not discussed and analyzed, which makes the proposed method not convincing enough. In addition, can the selection of different network structures significantly affect the retrieval performance of the final model? Such problems require further numerical experiments, analysis and discussion. At least some empirical discussion and explanation about the selection of different network structures are necessary.\n3. To derive ProbHash for existing “Center-Target” methods, the likelihood distribution is assumed to be proportional to the exponential of the Hamming distance, the rationality of this assumption needs to be further discussed and analyzed since the lack of theoretical analysis on the consistency.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "For the problem of the uncertainty quantification of hash-codes for image retrieval, a novel probabilistic hashing model named ProbHash is proposed to take different sources of uncertainty into account. In addition, an uncertainty-aware hashing strategy named HashUQ is proposed to improve the retrieval performance with little computation and storage overhead. The experimental results on three image datasets validate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses for details.", "broader_impact_concerns": "This work does not seem to have any potential negative societal impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies using neural networks to model hash functions for uncertainty-aware image retrieval. The proposed framework, HashUQ, first predicts codes and uncertainty estimations for input images in the encoding stages. The retrieval process can then leverage the uncertainties for better ranking results. The reviewers agree that using uncertainty estimation to augment information retrieval is a strong idea (79if, Qmeu), and the experiment mostly shows its effectiveness (all reviewers). Meanwhile, reviewers also raise several concerns regarding result comparisons with specific baselines (Qmeu), motivation of some algorithm designs (Qmeu, 3tbv), and clarities & presentation issues (79if, Qmeu). The author makes a good effort in the rebuttal phase and addresses most of the concerns. Additionally, reviewer 3tbv also requests an analysis of the likelihood distribution assumption made in the center-target method. The AE checked the discussion threads and revised paper, finding that the authors include a discussion in Appendix A.3 of the latest revised version. Given the consensus of all reviewers,  the AE recommends the acceptance of this paper, with the request of performing an experimental comparison between the selected Boltzmann distribution versus other commonly used distributions, as requested by reviewer 3tbv.
Decision: Accept (poster)