One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective
Authors: Jiun Tian Hoe
Kam Woh Ng
Tianyu Zhang
Chee Seng Chan
Yi-Zhe Song
Tao Xiang
Venue: NeurIPS 2021 Poster (2021)
Keywords: hashing
deep hashing
learning to hash
image retrieval
instance level retrieval

Abstract: A deep hashing model typically has two main learning objectives: to make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such as bit balance and code orthogonality, it is not uncommon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only $\textit{a single learning objective}$. Specifically,  we show that maximizing the cosine similarity between the continuous codes and their corresponding $\textit{binary orthogonal codes}$ can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using a  Batch Normalization (BN) layer and multi-label classification is also straightforward with label smoothing. The result is a one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly,  extensive experiments show that our model is highly effective, outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins. 

One Loss for All:
Deep Hashing with a Single
Cosine Similarity based Learning Objective
Jiun Tian Hoe"
Kam Woh 2g
Tianyu Zhang Tao Xiang
Chee Seng Chan
Yi-Zhe Song
CISIP, Universiti Malaya, Malaysia CCSSP, University of Surrey, U.K.
FilyTekk Tellyeek--urree Joint Research Centre Artificial Intelligence *Geekt, China
Abstract
A deep hashing model typically has two main learning objectives: make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such bit balance and code orthogonality, it is not uncom- mon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only a single learning objective. Specifically, we show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using Batch Normalization (BN) layer and multi-label classification is also straightforward with label smooth- ing. The result is an one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly, extensive experiments show that our model is highly effective, outperforming the ttate-of-the--a multi-loss hashing models on three large-scall scale instance retrieval benchmarks, often by significant margins. Code is available at withhttt//// btttp://githhtt com/ ham////vooo/rhhhaaa) orthohash
1 Introduction
A key building block of real-world large-scale image retrieval system is hashing. The objective of image hashing is represent the content of an image using binary code for efficient storage and accurate retrieval. Recently, deep hashing methods 48
23 have shown great improvements 19 Furthermore, deep hashing methods can be grouped by how the similarity of the learned hashing codes are measured, namely pointwise
over conventional hashing methods [46] [5 16 36
37
49 54 40, 12 [50, pairwise (25 23 5 4. trie 45 32], or listwise (52). Among them,
pointwise methods have O(N) computational complexity, whilst the complexity of the others of at least ( for N data points. This means that for large-scale problems, only the pointwise methods are tractable [49]. They are thus the focus of most recent studies.
deep hashing neural network naturally has multiple learning objectives. Specifically, given an image input, the network outputs a continuous code (feature vector) which is then converted into binary
"quul contribution.
corresponding author (cs.chan @ um. edu.emdmmy..
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

50
25
25
50
-75
-1
-100
-125
-150
-60
-40
-20
20
40
-5.0
-2.5
0.0
2.5
5.0
-2
(a) CE
(b) CE+BN
(c) Proposed
Figure 1: We train simple CNN model on CIFARIO with only first 4 classes and 2-bits. The continuous codes v are visualized before sgn. (Left) The model trained with cross entropy (CE) only. Although it can separate the 4 classes in Euclidean space, the output is not bounded and thus indicating high quantization error and sub-optimal the Hamming space. (Middle) By appending a batch normalization (BN) layer after the hash codes are now balanced. (Right) Now the model (proposed) is trained maximize the cosine similarity between and its corresponding binary target The black arrows are the binary orthogonal target, denoted for each class. It can be seen that the continuous codes exhibit lower intra-class variance and quantization error as compared with the CE+BN models (middle).
hash code using a quantization layer (usually sign function). There are thus two main objectives. First, the final model output, i.e., the binary codes must be discriminative, meaning the intra-class hamming distances are small, while the inter-class ones big. Second, quantization error minimization objective is needed to regularize the continuous codes. But the learning is constrained by the vanishing gradient problem caused by the quantization layer. Although the problem can be avoided by deploying some relaxation schemes [5] 23 25), these schemes often produce sub-optimal hash codes due the introduction quantization error (see Figure IT. Hence, most recently deep has an explicit quantization error minimization learning objective.
hashing methods 41 25 4
53 50
still not enough. In particular, ensure the quality of hash codes, many other losses are employed by existing methods. These include bit balance loss weights constraints maximize Hamming distance [54]. code orthogonality [1 (32. Further, losses are designed to address the vanishing gradient problem caused by the sign function used obtain binary codes from the continuous ones [41 40 27]. As result, the ofate-of-the-art hashing models typically have a large number (>4) losses. This means difficulties in optimization which in turn hamper their effectiveness.
Having these main objectives/losses
533
49
40),
In this work, for the first time, a deep hashing model with a single loss is developed which removes any needs for loss weight tuning and is thus much casier to optimize. As mentioned carlier, deep hashing model needs to be trained with at least two objectives, namely binary code discriminativenss and quantization error minimization. So how could that the two objectives are closely related and can be unified into one. More concretely, we show that both objectives can be satisfied by maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal target, which can be formulated as cross-entropy (CE) loss. Our model, dubbed OrthoHash has loss only which maximizes the cosine similarity between the Lg-normalized continuous codes and binary orthogonal target to maximize inter-class Hamming distance and minimize quantization error simultaneously. We show that this single unifying loss has number of additional benefits. First, we can leverage the benefit of margin [42] [0] to further improve the intra-class variance. Second, since conventional CE loss only works for single-label classification, we can easily leverage Label Smoothing (38 to modify the CE loss to tackle multi- labels classification. Finally, we show that code balancing can now be enforced by introducing a batch normalization [17] (BN) layer rather than requiring different loss. Extensive experiment results JUS-WIDE and MS-COCO, our model is on par with the SOTA. More importantly, on the large-scale instance-level retrieval tasks, our method achieves the new SOTA, beating the best results obtained so far on GLDv2, ROxf and RParis by 0.6%, 9. % and 17.1% respectively.
loss only? The answer lies in the fact
suggest that on conventional category-level retrieval tasks using ImageNet100,

---

2 Related Work
Hashing methods. Conventional hashing methods can be categorized into many streams. Data- independent methods such as Locality Hashing (LsH) 16 13), and its kernelized version (KLsH) [22] have contributed many of the fundamental concepts for hashing such as the requirement of code balance, uncorrelated bit, and similarity preserving. In contrast, data-dependent methods 36 37 aim to learn hash codes that are more compact yet more dataset-specific [7]. Recently, deep learning based hashing methods [25] 48 23 dominated the hashing research due to the superior learning ability of DNN. Various learning objectives are developed to learn hash codes using a training dataset. The objective functions include i) task learning objective which can be further categorized into pointwise 49 [54] 40
[46 21 15
19
41 [2] [50,, pairwise [5 25 [23] triplet-wise A5 (2), listwise [52] and unsupervised [2 14: ii) quantization error minimization such as the loss designed to minimize the p-norm (usually p 2) between continuous codes and hash codes; iii) code balancing for more detailed review.
27 40
We refer readers to learning to hashing surveys 44
Binary optimization. Hashing is NP-hard binary optimization problem [46., and is prone to the vanishing gradient problem due to the discrete and non-differentiable binary hash functions. Early methods solved the problem by discarding the discrete constraints (e.g., designing penalty loss term to generate feature as binary as possible [25] 23 solve with continuous relaxation, i.c,, to optimize in continuous space using sigmoid tanh for approximation (5)). Some methods also utilized coordinate descent method in the training 28 24]. Nevertheless, these methods have increased the complexity of learning due need for tuning of hapee-paramettrs balancing different learning objectives.
Bypassing vanishing gradient. Greedy Hash 41 designed new coding layer which uses the sign function the forward pass to generate binary codes, and gradients are backpropagated using straight-through estimator [1] during optimization. (27] designed a parameter-free coding layer Bi-half, to maximize the bit capacity by shifting the network output by median (each bit can have a 50% chance of being +1 or -1). These methods typically requires the modification of computational graphs, in the sense that the original graph is no longer end-to-end trained, hence further complicates the original optimization objective. Ours the other hand incorporates a neat one-loss design that removes all such complications.
used random assignment scheme to generate target vectors with maximal inter-class distance, then optimized with hinge-like polarized loss. Central Similarity Quantization (CSQ) (50) uses Hadamard matrix as "hash centers", then optimized with binary cross entropy. Both methods have similar overall objective, i.e., the continuous codes are learned to be as similar as the target vectors (or "hash centers"). Our model also employs a hash target, but uniquely it is used in a single cosine similarity based single objective.
Learning hash codes with pre-defined target. Deep Polarized Network (DPN) 12
Cosine similarity. According to [6), which is a theoretical analysis for Locality-sensitive Hashing (LsH) [6 13,, if two samples have high angular similarity, then we have high probability of obtaining the same hash codes well. Hence, while most works focus on hashing images with various constraints, we reformulate the problem deep hashing in the lens of cosine similarity. By following the same principle, a similar work is done by (2] which described the hashing problem under pairwise constraint while our work describes the problem under pointwise constraint. As inspired by [51] 14] which utilize cosine similarity to find closest approximate binary or ternary representation, we also interpret the quantization error in terms of cosine similarity. Moreover, deep hypersphere embedding learning methods (c.g., SphereFace [35), CosFace [42] and ArcFace (10) imposed discriminative constraints on a hypersphere manifold and proposed to improve decision boundary by cosine or angular margin. Inspired by them, we also leverage the benefit of margin to improve intra-class variance.
3 OrthoHash: One Loss for All
we reformulate the problem of deep hashing in the lens of cosine similarity, i.e., interpreting both Hamming distance retrieval and quantization error in cosine similarity. In Section we propose to maximize cosine similarity between the continuous codes and binary orthogonal target under single classification objective (for both single-label and multi-labels classification).
In Section 3.1
3

---

Scaled Cosine Similarity
Inputs
Continuous Codes
vK
| I
(U 0,
Classification Output (N x c)
Classification Label
(N K)
0.8
0.5
-0.1
BN
0.7
+0.99
0.9
Backbone
Softmax Loss
Network
:
+0.6
0.7
+0.9
Cross Entropy
Continuous Codes
Orthogonal Targets
We first obtain continuous codes V = (v E N from our backbone network. then passed through batch normalization (BN) layer obtain zero-mean continuous codes. Next, we compute scaled cosine similarity between the continuous codes and their binary orthogonal targets O for )= E +11,+1)cxx where C number of classes. Finally, the scaled cosine similarity will act as a classification output and we minimize a cross entropy loss. See Section 3.2 for details.
Figure 2:
Finally, we deseribe why adding batch normalization layer after the continuous codes will achieve code balance in Section 3.2.3 Our method is illustrated in Figure
Let first formally define the deep hashing problem. Let d-dimensional data, X {xn) J.Y RNxd where N is the number of training samples, and Y
J {0, 1 one- hot training labels of C' classes (for multi-labels, Yn A Yni = [yn1,-.. , .uuc], whose Yhi 1 if any i-th class are assigned to the n-th sample and 0 otherwise). Our objective is to learn a set of K-bit binary codes B
J E (11,1) Nxc for each training point x which is converted from the continuous codes Vn through a sgn function. v can be computed by a latent layer H(x) Wo(x)= E R ((-) a deep neural network (backbone network) to compute q- g-dimensional nonlinear feature representation = 6(x) E R", W R is the weights of the latent layer and sg*( if k-th bit of v, 2 0 and -1 otherwise. our work, binary orthogonal targets Oy. (o1. TC] 0 E (-1, +*]CcK K where denotes a column vector belongs to i-th class. Ideally, for any two rows, 1 S i,j S C, o, and o, are orthogonal each other. We use or A to represent scalar, to represent column vector, and A to represent matrix. Both and j are often used as index.
3.1 Reformulating Deep Hashing in the Lens of Cosine Similarity
Interpreting Hamming Distance as Cosine Similarity. Typically, Hamming distance can be com- puted using logical xor operation between binary codes b, and by. followed by popcount. b is represented by {-1,+) +111 then Hamming distance can also be computed mathematically as:
D(b, b,) K b,b,
2
Geometrically, the dot product b, can be interpreted as:
b bj = |ball |/bي|| cos Aij,
(2)
which ||-|| the Euclidean norm and 0, the angle between b, and by. As both ||b:]| and ||by||
are constant (i.e., |b|| = K)), equation (I) can then be viewed as:
K B3j
D(b,, b,))
1 cos 0,y).
Since K is constant, we that the retrieval is now will be only based the angle between two hash codes i.e., similar hash codes will have a similar direction, yield lower angle between them, and hence a lower hamming distance.
Interpreting Quantization Error as Cosine Similarity. Typically, converting continuous codes to binary codes b will Iead to information loss, which is also known as quantization error. Therefore,

---

most of the existing hashing methods have included quantization error minimization in their learning objective such as -norm, -n-no and p-norm (c.g.. p in Greedy Hash (4 usually in the form of:
(4) where I. is the supervised learning objective such as Cross Entropy and Q the quantization error between and b. However, it is difficult to control the scale X, i.c. low Ả might not be effective, while a high might lead to underfitting. As a result of this, careful tuning is needed and yet the tuned À may varies in different tasks. To overcome this cumbersome practise, let us first interpret quantization error geometrically:
min L + XQQ
min || bll² s.t. b (-1,1)*;
(5)
in which v is in continuous space, b sgn(v) is in binary space. We expand equation (5) to get:
||v bl 2 ||v|| + ||b]| ||v|| ||||| cos Pub.
(6)
According to equation B. retrieval is only based on the similarity in the direction of two hash codes. Hence, we can ignore the magnitude of v by normalizing it to have the same norm with b, i.c., ||v]| = /K and interpret the quantization error as to only the angle Pyb between v and
||v bil 2K 2K cos
cos Oub 2K (1 cos Oub)-
Since 2K is constant, will lead to a low quantization error, leading to a better approximation in the hash codes.
then conclude that maximize the cosine similarity between v and b
3.2 Discriminative Hash Codes with Orthogonal Target
According [6), the probability of two samples x; and x, to have the same hash code under family F of hash functions using random hyperplane technique can be described as Prher(((x)) = h(xx)]] = 1 Su, where h(()) is a hash function and Oij is the angle between x; and Xy. Therefore, based on the same principle, can be derived that if the two continuous codes V; and V, from latent layer H. have high cosine similarity, then the hash codes b, and by should also have high chance of obtaining the same hash codes. Beside that, as described in Section 3.I cosine similarity can also be used to justify the retrieval performance using both the hash codes and quantization error between the continuous codes and hash codes. Given these two circumstances, we therefore propose to maximize the cosine similarity of the continuous codes v, and its corresponding binary orthogonal target, oy.. E [o,..... oc = O E {-1,+1 K where this can be achieved by maximizing the posterior probability of the ground- -ouud class using softmax (cross-entropy) loss:
cxp (OT V..)
L
Σ log
1==1 exp (o!
where v, denotes the deep continuous codes of the n-th samples from DNN é and both oy.,. O; E O denote the ground-trute class Yn and the i-th class of the binary orthogonal targets. For simplicity, we omit the bias term from equation (8). follows that under the framework of deep hypersphere embedding 35 42 10 we can transform the logit v.v,, = ||o:|| [/,|| cos On where Oni is the
angle between the continuous codes Va and the binary orthogonal target o, Next, we perform L2
normalization on vn so that ||vx2|| = 1, and ||o.|| = vK K since it is in binary form. Now our loss function can be rewritten as:
N L=
exp (VK K cos By.))
N log
Σ
1=1 exp (VK cos cos yy..) E Ei exp (V K cos bns)
As such, instead of introducing the quantization error minimization in the learning objective (equation (4), our proposed method unifies both the learning objective and quantization error minimization together under a single classification objective as shown in the loss function (equation ( Further- more, since the binary orthogonal targets attain maximal inter-class Hamming distance and that loss function also aims to minimize the intra-class variance, we can leverage on cosine or angular
See Appendix B supplementary material for proof.
5

---

margin that have been proven to be beneficial in CosFace A42 and ArcFace [TO], to further improve the minimization of intra-class variance (we set m 0.2 in all of our experiments unless mentioned explicitly). With this, our method is able to perform end-to-end training to learn highly discriminative hash codes without both the sophisticated training objectives and computational graph modifications.
3.2.1 Binary Orthogonal Target
The maximization of the expectation of inter-class Hamming distance will help to increase the recall rate during retrieval as there will be lesser chance to retrieve incorrect items, because the aim is to retrieve more similar items (intra-class), and avoid to retrieve incorrect items (inter- class). That is, given K-bit Hamming space H E [-1,+1] for any two binary vectors b; by sampled with probability p for +1 on each bit, the expectation of Hamming distance is E[D(b,, b 2 K p(1 - p) and it achieves the upper bound of K with p 0.5 12 55) (See Appendix in supplementary material for details.). Hence, hash codes b, and by must orthogonal so that we can get D(b,, by) = in equation
Orthogonal Targets Generation. Hadamard matrix naturally contains orthogonal rows and columns, which guarantees the maximum Hamming distance of K between any two rows (50 29]. However, is restricted when K is not 1, 2, multiple of 4. Hence, a simple solution is to sample the targets from (055) which every sampled bit has the probability = 0.5 to be +1. The result is the K expectation of Hamming distance between any two rows equals to 2 which indicates orthogonality. One limitation that 2 < C,, the nearest rows in the sampled targets will be identical, which causes performance degrade. Hence, simple solution is to increase K. In supplementary material (Appendix D.3), we show that the two nearest rows has Hamming distance closed to K as K is higher. We also generate the targets with the objective of maximum inter-class Hamming distance heuristically, it indeed improved the performance at lower K, but the improvement higher K are negligible.
2
3.2.2 Multi-labels Hash Codes Learning
As conventional cross-entropy loss only works for single-label classification, we leverage the concept of Label Smoothing (38 generate labels for multi-labels classification. A standard cross entropy
(CE) loss is mathematically formulated as:
L
N Σ Yni log(p((ns(x,,))
(10)
in which You if i-th class assigned the n-th sample in a single label multiclass classification
task. In [38]. the target label becomes soft-target such that non-target class has a small "smoothing" value to regularize overconfident samples and we leverage this concept for multi-labels. To adopt CE for multi labels classification, we set Yns = > 0 if any i-th class are assigned to n-th sample. The constant is determined such that > Iru = 1, e.g.,
0.5 and yn [0, 0.5, O, 0.5] when the 2nd and the 4th classes are the assigned classes. Our motivation is that the model should maximize the probabilities of the target classes, which can optimize the hash codes to be as similar as the binary found out empirically that replacing softmax
targets from assigned classe In experiments, with sigmoid for multi-labels are not effective") A likely explanation is that softmax will intrinsically suppress the lower activated class unit (i.c., scaled cosine similarity) with lower probability and increase the highly activated class unit with higher probability, while sigmoid will treat each class unit an individual unit. As result, maximizing probability of class might not lead to minimizing the probability of other classes. Therefore, we propose to leverage the concept of Label Smoothing to generate labels so that we can use cross entropy loss for learning.
*Cosine margin will transform exp (vK cos 0g..) ) to exp (V(cos(s,.... m)) and angular margin will transform the same exp (v cos(Oy.. + m²)).
* 'Note, we cannot guarantee that the final hash codes are the center hash codes the target classes. Instead we let the optimization algorithm to find the best hash codes.
See Appendix D.4 in supplementary material for details.

---

3.2.3 Code Balance
Although binary orthogonal target helps in code balancing, since every bit has 50% of chance being +1 or -1, there no guarantee that the model will learn to output a balanced code. Therefore, we propose to add batch normalization (BN) layer after the continuous codes to ensure the code balance. If En Vak- = O, then we can see that Σn bns = for the k-th bit. Because the distribution of v has been normalized to have zero-mean and variance of 1, with b sgn(v), (v), the hash codes b will follow uniform binary distribution with 50% chances on both +1 and -1. Empirically, we found that it improves the retrieval performance with a model with normal cross entropy loss (see Table DT Note that the Bi-half method [27] shifts the continuous codes by their median, followed by converting the continuous codes to binary codes for optimization. However, it will have to modify the computational graph in order to have proxy derivative to the solve vanishing gradient problem. In contrast, appending BN layer will not modify the computational graph, therefore enabling straightforward mnd-to-end training.
ImageNet100 by about 17-20% as compared
4 Experiment
Training Setup. We select 7 different deep hashing methods for comparison (5 point-wise, pair- wise and 1 triplet-wise). For fair comparison, we use the same learning rate of 0.0001, Adam optimizer 18 and 100 epochs for all methods. For SDH-C (11, we have modified it from pair-wise objective to point-wise objective, while all penalty terms are kept (i.e., quantization loss, bit variance loss and orthogonality on projection weights).
23 34 and choose ImageNett10 [ NUS- WIDE [8] and MS-COCO [30] for category- retrieval experiments. For a more practical yet challenging large-scale instance-level retrieval task (i.e., tremendous number of classes), we evaluate
Datasets. We follow prior works 12, 33 40 48
on the popular GLDv2
47 ROxf and R.ar 39
Architecture. For category-level retrieval, following the settings in 41 12 we use pre-trained AlexNet (20) as the network backbone initialization. The output from last fully connected with ReLU (4096-dimension vector) acts as input to the latent layer; various supervised deep hashing methods then applied to generate binary codes. The image size is 224 x 224. For instance-level retrieval, due to the expensive cost of training from scratch, we use pre-trained mode (ode (R50-DEL clean) from DELG [3] to compute the 2048ddimension global descriptors. We then train a latent layer H to compute hash codes where inputs are the global descriptors. For GLDv2, the images input are 512 x 512. For ROxf and Raar, we use 3 scales { ( v2) to produce multi-scale representations. These are subject to L2 normalization, and then average-pood peragg- to obtain a single descriptor as done by 3 A GLDv2-trainee latent layer is used to compute hash codes for the evaluations.
Details of training setups, datasets and architecture can be found in the supplementary material (Appendix C).
4.1 Results on Category-level Retrieval
R3 we use mean average precision (mAP@R) which is the mean of average precision scores of the top R retrieved items. Table I offers performance comparison amongst all selected hashing methods and our methods (+variants). CE denotes model trained with cross entropy only, the hash codes are computed from sign of continuous codes. CE+BN denotes CE model with BN layer [77 appended after the latent layer. CE+Bihalf denotes CE model with Bihalf" appended after the latent layer. OrthoCos denotes model trained with cosine margin and binary orthogonal target. OrthoCos+ Bihalf denotes variant of OrthoCos. and with Bihalf layer appended. Ortho denotes a variant of OrthoCos, and with BN layer appended. OrthoAre+BN denotes variant of CrthhoCos+BN, trained with angular margin.
For performance evaluatior
e [7 layer
Overall. It can be observed that both our OrthoCos+ BN and OrthoAre+BN perform better than recent state-of-the-art, DPN [2] and CSQ (50) On multi-labeled datasets (i.e., NUS-WIDE and MS
watrrreeeaattereeeeeee wttp//eeearccceell
"See Appendix C.3 supplementary material for evaluation detail.
"Note that Bihalf was originally proposed for unsupervised hashing, however, we think it is worth to compare Bihalf is layer for code balancing instead of training objective.
7

---

ImageNet100 (mAP@ IK)
NUS-WIDE (mAP@5K)
MS COCO (mAP@5K)
Methods
16
32
64
128
16
32
64
128
16
32
64
128
HashNet
S
0.343
0.480
0.573
0.612
0.814
0.831
0.842
0.847
0.663
0.693
0.713
0.727
DTSH 15
0.442
0.528
0.581
0.612
0.816
0.836
0.851
0.862
0.699
0.732
0.753
0.770
SDH-C
31
0,584
0.649
0.664
0,662
0.763
0.792
0.816
0.832
0.671
0.710
0.733
0.742
Hash
0.570
0.639
0.659
0.659
0.771
0.797
0.815
0.832
0.677
0.722
0.740
0.746
JMLH
40
0.517
0.621
0.662
0.678
0.791
0.825
0.836
0.843
0.689
0.733
0.758
0.768
DPN1 12
0.592
0.670
0.703
0.714
0.783
0.818
0.838
0.842
0.668
0.721
0.752
0.773
CSQ
50
0,586
0.666
0.693
0.700
0.797
0.824
0.835
0.839
0.693
0.762
0.781
0.789
CE²
0.350
0.379
0.406
0,445
0.744
0.770
0.796
0.813
0.602
0.639
0.658
0.676
CF+BN
0.533
0.586
0.612
0.617
0.801
0.814
0.823
0.825
0.697
0.721
0.729
0.726
CE+Bihalf
27
0.541
0.630
0.661
0.662
0.802
0.825
0.836
0.839
0.674
0.728
0.755
0.757
OrthoCos
0,583
0.660
0.702
0.714
0.795
0.826
0.842
0.851
0.690
0,745
0.772
0.784
Orthals+BBihalf'
0.562
0.656
0.698
0.711
0.804
0.834
0.846
0.852
0.690
0.746
0.775
0.782
DN!
0,606
0.679
0.711
0.717
0,804
0.836
0.850
0.856
0.709
0.762
0.787
0.797
OrthoAre+BN
0.614
0.681
0.709
0.714
0.806
0.833
0.850
0.856
0.708
0.762
0.785
0.794
Table I: Performance of different methods for 4 different bits on different benchmark datasets. All
results are run by us. The superscript 1, 2 and 3 indicate point-wise, pair-wise and triplet-wise method respectively. Bold values indicate best performance in the column.
GLDv2
(mAP@ 100)
ROxf-Hard (mAP@all)
RParis-Hard (mAP@all all)
Methods
128
512
2048
128
512
2048
128
512
2048
2
0.018
0.069
0.111
0.034
0.058
0.307
0.133
0.190
0.490
DPN 12
0.021
0.089
0.133
0.053
0.184
0.303
0.224
0.399
0.562
GreedyHash!
41
0.029
0.108
0.144
0.032
0.251
0.373
0.128
0.531
0.652
CSQ [0
0.023
0.086
0.114
0.093
0,284
0.398
0.245
0.541
0.649
OrthoCos+BN
0.035
0.111
0.147
0.184
0.359
0.447
0.416
0.608
0.669
R50-DEL.GH
1.15
0.471
0.682
R50-DEI
1.18
0.510
0.715
Table 2: Performance of different methods for 3 different numbers of bits on different instance-level benchmark datasets. All results are run by us. The superseript and indicate point-wise and pair-wise method respectively. Bold values indicate best performance in the column. indicates using 512 x 512 image inputs, hence different performance as reported by DELG []. R50-DEL.GH denotes Hamming distance retrieval using the sign of extracted descriptors. R50-DELG-CC denotes Cosine distance retrieval using the extracted descriptors.
COCO), DTSH 45
(triplet based method) performed the best with 0.851 and 0.862 with 64 and 128-bits hash codes in NUS-WIDE followed by our method (e.g., OrthoCos+BN achieves 0.850 and 0.856 in the same settings), while OrthoCos+BN and OrthoArc+BN performed the best on MS-COCO with at most 1% improvement over previous deep hashing methods.
Code Balance. Although retrieval performance of CE models performed the worst, but by appending BN layer after the latent layer (CE+BN), we were able to observe 5-20% improvement over all settings (dataset and number of bits). Bihalf 27 layer (zero-median features) has a proxy derivative to learn hash features, hence getting 0.1-4 1114-.9 4.99% improvement than CE+BN. This indicates that without sophisticated training objectives, code balance itself a very important factor in improving Hamming distance based retrieval. However, OrthoCos+Bihll' does not show significant improvement over OrthoCos+BN. but is comparable with OrthoCos. We thus conclude that our method can achieve code balance without explicitly engineering the computational graph.
Cosine and Angular Margin. In experiments, we observed that cosine margin (OrthoCos+BN) slightly outperform angular margin (OrthoArc+BN) by about 0.2% on average.
4.2 Results on Instance-Leve Retrieval
For evaluation metries, we adapt the evaluation protocol of 3 39 The baseline performance of GLDv2, ROxf-Haad and Hard from the pre-trained R50-DELL-GLDD2--LE are 0.138,
8

---

Bormakzzed hraquing
0.20
Intro class Inter-class
0.20
Intro class Inter-clss
Bormalizze Prequa 20
0.20
Intra class Inter-clas
0.15
0.15
0.15
0.10
Normellzed
0.10
0.10
24
40
24
64
(a) HashNet [5]
(b) Greedylaas
4
(c) OrthoCos+BBN
Figure 3: Histogram of intra-class and inter-class Hamming distances with 64-bits ImageNet100. The arrow annotation is the separability in Hamming distances, E[Dettr] [ntraa We normalized the frequency so that sum of all bins equal to 1.
40
0,7
0.8
0.8
0.8
6
0.s
0.5
20
Greed Grestptant
Or-anner
octec Din
co
Ordeabrs
Romiina
erady ok Sk-anerr
Ce-Ra
Jusas
Orthetar tak
ras
DY3H
dres drencipiint
Octac ei
csp
onteaber rese
APM
(a) Quantization emoor
(b) Separability
(c) Orthogonality
Figure 4: Analysis of retrieval performance 64-bits ImageNet100. (a) Quantization error: dub-
(b) Separability: E[Dsnter] D[ (e) Orthogonality: + HHT I|. Blue solid line denotes
mean average precision (mAP@ 1000) and orange dotted line denotes the respective analysis score.
0.510 and 0.715 respectively. Table summarizes the performance of different deep hashing methods and our method. For all the datasets, our method outperforms all previous deep hashing methods on all bits. This suggests that our method has better generalization ability on unseen instances than previous deep hashing methods. In particular, our model significantly outperforms previous deep hashing models by 0.6%, 9.1 % and 17.1% respectively on the datasets with 128-bits hash codes.
Orthogonal Transformation. For GLDv2 2048-bits hash codes, surprisingly it can achieve a much better performance than the pre-trained 2048-dimensions descriptors (by 1.1 % improvement over R50-DELG-CC, We then analyze the separability in cosine distances, i.e., the difference in the mean of intra-class cosine distance and the mean of inter-class cosine distance before and after the transformation (similar to Figure 3). We observe that the separability in cosine distances increases after the orthogonal transformation, i.c., before it is 0.142 and after it increases to 0.167. The results thus show that learning orthogonal hash codes can transform the inputs to be more discriminative.
Domain shifting with BN. As the model is trained with GLDv2, the running mean and variance in the BN layer might experience domain shifting problem 26 when testing directly on different datasets (e.g., ROxf and Par)). We empirically found that using running mean and variance from GLDv2 will lead to a large performance drop in Hamming distance retrieva IT] One simple solution to recompute the mean and variance from all continuous codes in the database, then update the running mean and variance with the computed mean and variance. The performances of ROxf and RPar Table 2 are obtained with running mean and variance of the respective database.
4.3 Further Analysis
Histogram of Hamming Distances. Figure summarizes the histogram of intra-class and inter- class distances. We compare our method OrthoCos+BN with pair-wise method HashNet (5) and point-wise classification based GreedyHash 4T) Although the distribution of inter-class distances are about the same for all the 3 methods (close to Hamming distance of K/2 = 32), we can see that the larger the separability i.c., the difference in the mean of intra-class distance (the blue dotted line) with the mean of inter-class distance (the orange dotted line), the better the performance.
Sese Appendix D.5 in supplementary material for details.

---

Performance Improvement Analysis. We further analyze the reasons behind performance improve- ments of different deep hashing methods, and summarizes the results Figure 4 We conclude main reasons that contribute to the improvement in deep hashing methods: i) quantization error; ii) the separability in Hamming distances; and iii) orthogonality in hash centers. For quantiza- tion error, we measure the angle Pub between the continuous codes v and the hash codes b, i.e., vTb )). For separability, we measure the difference in the mean of inter-class 0,6. = deg(arccos [|v]|[|]]
distances and the mean of intra-class distances, i.c., i[Dinter] E [[DDmtral For orthogonality, we first compute the hash centers H (-1, +1) + K for every class (by taking the sign of average hash codes in every class), then we measure the orthogonality with || b HHT - Ill (lower is better). When the quantization emror reduces, the separability increases and the hash centers has better orthogonality, resulting in better performance.
5 Conclusion & Future Work
We propose to unify training objectives of deep hashing under single classification objectíve. We show this can be achieved by maximizing the cosine similarity between the continuous codes and binary orthogonal target under cross entropy loss. For that, first reformulated the problem of deep hashing the lens of cosine similarity. We then demonstrated that we perform L2 -g-normalization on the continuous codes, then end-to-end training of deep hashing is possible without any extra sophisticated constraints. Moreover, we leverage the concept of Label Smoothing to train multi- labels classification with cross-entropy loss and batch normalization for code balancing. Extensive experiments validated the efficiency of our method in both category-level and instance-level retrieval benchmarks.
Nonetheless, the proposed method might fail when the number of bits too small (<8 bits), especially when number of classes is much greater than the number of bits. In this case, there will be overlapping in the generated target code (i.c., the number of maximum unique codes is equal to 2 where K number of bits). In such condition, the target code will also not guarantee to be orthogonal. Overcoming this limitation is part of the future work. Also, we are exploring how to learn better feature representations to improve the retrieval performance by using hash codes through unsupervised learning.
Broader Impact
Hashing remains a key bottleneck in practical deployments of large-scale retrieval systems. Recent deep hashing frameworks have shown great promise in learning code that are both compact and diseriminative. Yet saate-of-the-and frameworks are known to be difficult train and to reproduce largely owing to their complex loss designs that dictates hyperparameter tuning and multi-stage training. In this work, we set out to change that we attempt to unify deep hashing under single objective, therefore simplifying training and help reproducibility. Our key intuition lies with reformulating hashing in the lens of cosine similarity. We report competitive hashing performance on all common datasets, and significant improvements over state-of- the-artss arts the more challenging task of instance-level retrieval.
-
Acknowledgments and Disclosure of Funding
This research is partly supported by the Fundamental Research Grant Scheme (FRGS) MoHE Grant FP021-2018A, from the Ministry of Education Malaysia. We also thank Kilho Shin for helpful discussions and recommendations.
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308..34 2013.
[2] Levi Boyles, Aniket Anand Deshmukh, Urun Dogan. Rajesh Koduru, Charles Denis, and Eren Manavoglu. Semantic hashing with locality sensitive embeddings. 2021.
10

---

[3] Bingyi Cao, André Araujo, and Jack Sim. Unifying deep local and global features for image search. In Andrea Vedaldi, Horst Bischof. Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 726-743, Cham, 2020. Springer International Publishing.
[4] Yue Cao. Mingsheng Long, Bin Liu, and Jianmin Wang. Deep cauchy hashing for hamming space retrieval. In 2018 IEFF//VF Conference on Computer Vision and Pattern Recognition, pages 1229-1237, 2018.
[5] Zhangjic Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. Hashnet: Deep learning to hash by continuation. In 2017 IEEE International Conference Computer Vision (ICCV), pages 5609-5618, 2017.
[6] Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380-388, 2002.
[7] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint 140.::31..33331..
[8] Tat- Fat-Seng Chua, Jinhui Tang, Richang Hong. Haojie Li, Zhiping Luo, and Yan- Zheng. Nus-wide A real weal-world web image database from national university of singapore. Proceedings of the ACM international conference on image and video retrieval, pages 1 9, Santorini, Greece., July 8-10, 2009.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 248-255, 2009.
[10] Jiankang Deng. Jia Guo, Xue Niannan, Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In 2019 JEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
[11] Shiv Ram Dubey, decade survey content based image retrieval using deep learning. 2020.
[12] Lixin Fan, Kam Woh Ng. Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. Christian Bessicre, editor, Proceedings of the Twenry-Ninth International Joint Conference on Artificial Intelligence, ISCA/-20, pages 931-831. International Joint Conferences on Artificial Intelligence Organization, 2020. Main track.
[13] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search high dimensions via hashing. In vldb, volume 99, pages 518-529, 1999.
[14] Yunchao Gong, Sanjiv Kumar, Vishal Verma, and Svetlana Lazebnik, Angular quantization based binary codes for fast similarity search. In F. Pereira, C. J.C. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[15] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization: procrustcan approach to learning binary codes for large-scale image retrieval. JEEE transactions on pattern analysis and machine intelligence, 35(12):2916 2029, 2012.
[16] Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposiun on Theory of Computing, STOC *98 page 604-613, New York, NY, USA, 1998. Association for Computing Machinery.
[17] Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 Proceedings of Machine Learning Research, pages 448 456, Lille, France, 07-09 Jul 2015. PMLR.
[18] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
[19] Weihao Kong and Wu-jun Li. Isotropic hashing. In Pereira, C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey F., Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS' 12, page 1097-1105, Red Hook, NY, USA, 2012. Curran Associates Inc
[21] Brian Kulis and Trevor Darell. Learning to hash with binary reconstructive embeddings. In Y. Ben- gio, D. Schuurmans. J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22, Curran Associates, Inc., 2009.
11

---

[22] Brian Kulis and Kristen Grauman. Kernelized senssive hashing for scalable image search. In 2009 IEEE 12th international conference on compuler vision, pages 2130-2137. IEEE, 2009.
[23] Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan. Simultaneous feature learning and hash coding with deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3270-3278, 2015.
[24] Qi Li, Zhenan Sun, Ran He, and Tieniu Tan, Deep supervised discrete hashing. In I. Guyon, U. V. Luxhurg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[25] Wu-Jun Li, Sheng Wang, and Wang- Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels, Proceedings of the Twenty-Fifhh International Joint Conference on Artificial Intelligence, IJCAI' 16, page 1711-1717. AAAI Press, 2016.
[26] Yanghao Li, Naiyan Wang. Jianping Shi. Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 50:109-117 2018.
[27] Yungiang Li and Jan van Gemert. Deep unsupervised image hashing by maximizing bit entropy, Proceed- ings of the AAAI Conference on Artificial Intelligence, 2021.
[28] Guosheng Lin, Chunhua Shen, David Suter, and Anton van den Hengel. A general two-step approach to lcarning-basee based hashing. In 2013 IEEE International Conference on Computer Vision, pages 2552-2559, 2013.
[29] Mingbao Lin, Rongrong Ji, Hong Liu, and Yongjian Wu. Supervised online hashing via hadamard codebook learning. In Proceedings of the 26th ACM International Conference on Multimedia, MM '18, page 1645,1643, New York, NY. USA. 2018. Association for Computing Machinery.
[30] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 20/4, pages 740-755, Cham, 2014. Springer International Publishing.
[31] Venice Erin Liong, Jiwen Lu,, Gang Wang, Pierre Moulin, and Jie Zhou. Deep hashing for compact binary codes learning. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2475-2483, 2015.
[32] Bin Liu, Yue Cao, Mingsheng Long, Jianmin Wang, and Jingdong Wang. Deep triplet quantization. In Proceedings of the 26th ACM international conference on Multimedia, pages 755-763, 2018
[33] Haomiao Liu, Ruiping Wang. Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2064-2072, 2016.
[34] Wei Liu, Jun Wang, Rongrong Ji, Yu fu-Gang Jiang, and Shih-Fu Chang. Supervised hashing with kernels. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2074-2081. IEEE, 2012.
[35] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6738-6746, 2017.
[36] Mohammad Norouzi and David J. Fleet. Minimal loss hashing for compact binary codes. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML 11, page 353-360, Madison, WI, USA, 2011. Omnipress.
[37] Mohammad Norouzi, David Fleet, and Russ R Salakhutdinov. Hamming distance metric learning. In Advances in neural information processing systems, pages 1061-1069, 2012.
[38] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 1701, 17010065888 2017.
[39] Filip Radenovic, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In 2018 IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 5706-5715, 2018.
[40] Yuming Shen. Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, and Ziyi Shen. Embarrassingly simple binary representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. Oct 2019.
12

---

[41] Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cnn. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[42] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji. Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5265-5274, 2018.
[43] Jingdong Wang. Ting Zhang, jingkuan song, Nicu Sebe, and Heng Tao Shen. A survey on learning to hash. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):769-77) 2018.
[44] Jun Wang, Wei Liu, Sanjiv Kumar, and Shih-Fu Chang. Learning to hash for indexing big data a survey, 2015.
[45] Xiaofang Wang, Yi Shi, and Kris M Kitani. Deep supervised hashing with triplet labels. Asian Conference on Computer Vision, 2016.
[46] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2009.
[47] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-leve recognition and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2575 2584, 2020.
[48] Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. Supervised hashing for image retrieval via image representation learning. Proceedings of the AAAI conference on artificial intelligence. volume 28, 2014.
[49] Huci-Fang Yang, Kevin Lin, and Chu-Song Chen. Supervised learning semantics prrsrieiiprreeiiigggg hash via deep convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 40<2):437-451, 2017.
[50] Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng. Central similarity quantization for efficient image and vidco retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3083-3092, 2020.
[51] Tianyu Zhang, Lci Zhu, Qian Zhao, and Kilho Shin. Neural networks weights quantization: Target none-retraining ternary (tnt), 2019.
[52] Fang Zhao, Yongzhen Huang, Liang Wang, and Tieniu Tan. Deep semantic ranking based hashing for multi-label image retrieval. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1556-1564, 2015.
[53] Xiangtao Zheng, Yichao Zhang, and Xiaoqiang Lu. Deep balanced discrete hashing for image retrieval. Neurocomputing, 403:224-236, 2020.
[54] Chang Zhou, Lai- Man Po, Wilson Y. F. Yuen, Kwok Wai Cheung, Xuyuan Xu, Kin Wai Lau, Yuzhi Zhao, Mengyang Liu, and Peter H. W. Wong. Angular deep supervised hashing for image retrieval. IEEE Access, 7:127521-127532, 2019.
13


Reviews:
[{"review_id": "jkQIsiYF9Y", "reviewer": "Reviewer_MNiD", "summary": "This paper presents OrthoHash, a new deep hashing model that is trained by maximizing the cosine similarity between the continuous codes and the corresponding binary orthogonal target. In addition, code balancing is achieved by using batch normalization. Different from prior works that typically involves multiple loss functions, the training objective in this paper is simpler and easy to optimize. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Neat idea and good novelty: the proposed training objective is well designed with the idea of cosine similarity. The final formulation with softmax-like loss function is very interesting, and it is simpler than the existing multi-loss approaches. Additionally, the use of BN for balancing code is elegant. \n- Strong results on multiple datasets. It is highly competitive compared to multi-loss approaches. \n- Good ablation studies to demonstrate the performance impact of different factors.\n- The paper is well-written and easy to understand.\n\nWeakness:\n- One possible weakness is that it may have large performance degrade when the bit length is short. That is because in the target code generation, the target is sampled from Bern(0.5). One may expect larger information loss with short code.\n- Evaluation: For ROxf and RPar, the authors use multi-scale features for instance retrieval. It is unclear to me if the compared exiting deep hashing methods also use such a multi-scale scheme. Does it a fair comparison?\n- Evaluation: The authors use mAP@1K for ImageNet100, but mAP@5k for NUS-WISE and COCO. In instance retrieval, the authors use mAP@100 and mAP@all instead. They are inconsistent and not easy to interpret the performance. The authors may want to provide some intuitions/discussions about their design choices. For example, how many returned images are required to better reflect the  retrieval performance?  What's the effect of evaluating all returned images?  \n- In current presentation, the proposed method performs only on par with prior works in category-level retrieval. This gives an impression that only ordinary improvements are achieved. I believe this paper has many advantages that are not well demonstrated in current form. One shining point should be its scalability to very large dataset. I was hopping to see at least one experiment at a large scale, to clearly differentiate itself from pair-wise and triplet-wise methods. This would strengthen the paper significantly. \n\n\nOverall:\n- Overall this is a nice paper. My main concerns are in their evaluation. But this paper has merits that outweigh the flaws.\n- This paper presents a simpler learning objective that can reduce the difficulty of training deep hash models. Given its effectiveness and the simplicity, I think this paper could serve as a strong baseline to facilitate future research. In my opinion, it would bring insightful contributions to our community. My initial recommendation is Accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "QOEFXuLgS5r", "reviewer": "Reviewer_N6CJ", "summary": "The paper proposes a deep hashing model with only a single learning objective. It avoids the problem that existing deep hashing models are difficult to train due to a large number (>4) of losses. Specifically, it maximizes the cosine similarity between the continuous codes and their corresponding binary orthogonal codes to ensure both the discriminative capability of hash codes and the quantization error minimization. Besides, it adopts a Batch Normalization layer to ensure code balance and leverages the Label Smoothing strategy to modify the Cross-Entropy loss to tackle multi-labels classification. Extensive experiments show that the proposed method achieves better performance compared with the state-of-the-art multi-loss hashing methods on several benchmark datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a deep hashing model with only a single learning objective. It avoids the problem that existing deep hashing models are difficult to train due to a large number (>4) of losses. Specifically, it maximizes the cosine similarity between the continuous codes and their corresponding binary orthogonal codes to ensure both the discriminative capability of hash codes and the quantization error minimization. Besides, it adopts a Batch Normalization layer to ensure code balance and leverages the Label Smoothing strategy to modify the Cross-Entropy loss to tackle multi-labels classification. Extensive experiments show that the proposed method achieves better performance compared with the state-of-the-art multi-loss hashing methods on several benchmark datasets.\n\nStrengths: \n1. This work is meaningful for effectively optimizing the deep hashing models. It unifies the training objectives of deep hashing under a single classification objective.\n2. The motivation is well presented by the authors in the manuscript.\n3. The paper carries out extensive experiments and the supplementary material is rich in content.\n\nWeaknesses:\n1. According to the experimental results in Table 1, the proposed method does not consistently outperform the baseline methods, which is inconsistent with the sentence\"outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins\" mentioned in Abstract. In addition, the performance improvement of the proposed method is not significant.\n\n2. As the main contribution of this work is to propose a unified objective function, to verify its superiority, the authors should compare the results of the following two experiments:\na) Baseline methods without any change \nb) The variant baselines which replace the multiple loss functions with the proposed single loss \n\n3. There are several issues that the authors may want to explain.\na) In the second paragraph of section 3 OrthoHash: One Loss for All, the authors write \"o_i denotes a column vector belongs to i-th class\". Since the dimension of the binary orthogonal targets O is C×K, I think o_i is a row vector.\nb) The proposed method is supervised. Please expain the reason why using the unsupervised Bihalf as the baseline?\nc) Why mAP@1k is used for ImageNet100 and mAP@5k for MS COCO for performance evaluation, especially when the two datasets are close in size. Besides, the metrics of performance evaluation for instance-level benchmark datasets are also different.\n\n4. For experimental evaluation with SOTA methods, the authors should compare more newly deep hashing methods in the experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. According to the experimental results in Table 1, the proposed method does not consistently outperform the baseline methods, which is inconsistent with the sentence\"outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins\" mentioned in Abstract. In addition, the performance improvement of the proposed method is not significant.\n\n2. As the main contribution of this work is to propose a unified objective function, to verify its superiority, the authors should compare the results of the following two experiments:\na) Baseline methods without any change \nb) The variant baselines which replace the multiple loss functions with the proposed single loss \n\n3. There are several issues that the authors may want to explain.\na) In the second paragraph of section 3 OrthoHash: One Loss for All, the authors write \"o_i denotes a column vector belongs to i-th class\". Since the dimension of the binary orthogonal targets O is C×K, I think o_i is a row vector.\nb) The proposed method is supervised. Please expain the reason why using the unsupervised Bihalf as the baseline?\nc) Why mAP@1k is used for ImageNet100 and mAP@5k for MS COCO for performance evaluation, especially when the two datasets are close in size. Besides, the metrics of performance evaluation for instance-level benchmark datasets are also different.\n\n4. For experimental evaluation with SOTA methods, the authors should compare more newly deep hashing methods in the experiments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2cChhs1A-Dj", "reviewer": "Reviewer_7uiu", "summary": "In this paper, authors propose a novel deep hashing model with only a single learning objective and claim that state of the art papers generally use a large number (>4) of losses. Specifically, authors show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hashcode discriminativeness and quantization error minimization. Authors show significance of their method using extensive experiments. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Proposed method is simple but novel. The submission is technically sound and well supported by experimental results. \n2) Under what conditions the proposed method would fail? Can authors comment more on this?\n3) Using orthogonal targets is a simple but clever idea. Can authors comment on how can modify this method when one doesn’t have categorical labels but have some pairwise constraints and the goal is to learn quantized representation learning for better retrieval? \n4) In Table 1, proposed method is not always the best. From the discussion it was not clear why? Can authors comment more on why? Is it because the state of the art methods were already close to the best possible results or there is some drawback of the method that does not improve the results on some datasets? \n5) Angular margin used in the paper seems very related to semantic hashing [1] where angular similarity was used to generate representations better for hashing retrieval. Can authors compare and comment? \n6) In Figure 6, not only the distance between two distributions (intra class and inter class) matter but also the overlap between them matters. Can authors calculate and comment on the overlap between two distributions?  \n\n[1] https://openreview.net/pdf?id=sFDJNhwz7S ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a novel deep hashing model with only a single learning objective which is a simplification from most state of the art papers generally use lots of losses and regularizer. Two of the three reviewers liked its effectiveness and the simplicity, they believe it would bring insightful contributions to our community. Reviewers liked the final formulation with softmax-like loss function. They also found the use of Batch normalization for balancing code elegant.

One of the reviewers raised certain concerns.  The authors in rebuttal clarified that most of the negative argument raised by reviewers are misunderstanding and are not valid. It is clear that the concerns raised were already addressed in the main draft. 
.
Overall, the merits of the paper justifies the publication.  
Decision: Accept (Poster)