One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective
Year: 2021

Abstract:
A deep hashing model typically has two main learning objectives: to make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such as bit balance and code orthogonality, it is not uncommon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only a single learning objective. Specifically, we show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using a Batch Normalization (BN) layer and multi-label classification is also straightforward with label smoothing. The result is an one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly, extensive experiments show that our model is highly effective, outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins. Code is available at https://github.com/kamwoh/orthohash

heading: 1 Introduction
text: A key building block of a real-world large-scale image retrieval system is hashing. The objective of image hashing is to represent the content of an image using a binary code for efficient storage and accurate retrieval. Recently, deep hashing methods [48, 23] have shown great improvements over conventional hashing methods [46, 15, 16, 22, 36, 37, 19]. Furthermore, deep hashing methods can be grouped by how the similarity of the learned hashing codes are measured, namely pointwise [49, 54, 40, 12, 50], pairwise [25, 23, 5, 4], triplet-wise [45, 32], or listwise [52]. Among them, pointwise methods have a O(N) computational complexity, whilst the complexity of the others are of at least O(N2) for N data points. This means that for large-scale problems, only the pointwise methods are tractable [49]. They are thus the focus of most recent studies.
A deep hashing neural network naturally has multiple learning objectives. Specifically, given an image input, the network outputs a continuous code (feature vector) which is then converted into a binary
∗equal contribution. †corresponding author (cs.chan@um.edu.my).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
hash code using a quantization layer (usually a sign function). There are thus two main objectives. First, the final model output, i.e., the binary codes must be discriminative, meaning the intra-class hamming distances are small, while the inter-class ones are big. Second, a quantization error minimization objective is needed to regularize the continuous codes. But the learning is constrained by the vanishing gradient problem caused by the quantization layer. Although the problem can be avoided by deploying some relaxation schemes [5, 23, 25], these schemes often produce sub-optimal hash codes due to the introduction of quantization error (see Figure 1). Hence, most recently deep hashing methods [41, 25, 4, 53, 50] has an explicit quantization error minimization learning objective.
Having these two main objectives/losses are still not enough. In particular, to ensure the quality of hash codes, many other losses are employed by existing methods. These include bit balance loss [53, 49, 40], weights constraints to maximize Hamming distance [54], code orthogonality [31, 32]. Further, losses are designed to address the vanishing gradient problem caused by the sign function used to obtain binary codes from the continuous ones [41, 40, 27]. As a result, the state-of-the-art hashing models typically have a large number (>4) losses. This means difficulties in optimization which in turn hamper their effectiveness.
In this work, for the first time, a deep hashing model with a single loss is developed which removes any needs for loss weight tuning and is thus much easier to optimize. As mentioned earlier, a deep hashing model needs to be trained with at least two objectives, namely binary code discriminativenss and quantization error minimization. So how could one use one loss only? The answer lies in the fact that the two objectives are closely related and can be unified into one. More concretely, we show that both objectives can be satisfied by maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal target, which can be formulated as a cross-entropy (CE) loss. Our model, dubbed OrthoHash has one loss only which maximizes the cosine similarity between the L2-normalized continuous codes and binary orthogonal target to maximize inter-class Hamming distance and minimize quantization error simultaneously. We show that this single unifying loss has a number of additional benefits. First, we can leverage the benefit of margin [42, 10] to further improve the intra-class variance. Second, since conventional CE loss only works for single-label classification, we can easily leverage Label Smoothing [38] to modify the CE loss to tackle multilabels classification. Finally, we show that code balancing can now be enforced by introducing a batch normalization [17] (BN) layer rather than requiring a different loss. Extensive experiment results suggest that on conventional category-level retrieval tasks using ImageNet100, NUS-WIDE and MS-COCO, our model is on par with the SOTA. More importantly, on the large-scale instance-level retrieval tasks, our method achieves the new SOTA, beating the best results obtained so far on GLDv2, ROxf andRParis by 0.6%, 9.1% and 17.1% respectively.

heading: 2 Related Work
text: Hashing methods. Conventional hashing methods can be categorized into many streams. Dataindependent methods such as Locality-sensitive Hashing (LsH) [16, 13], and its kernelized version (KLsH) [22] have contributed many of the fundamental concepts for hashing such as the requirement of code balance, uncorrelated bit, and similarity preserving. In contrast, data-dependent methods [46, 21, 15, 19, 36, 37] aim to learn hash codes that are more compact yet more dataset-specific [7]. Recently, deep learning based hashing methods [25, 48, 23] dominated the hashing research due to the superior learning ability of DNN. Various learning objectives are developed to learn hash codes using a training dataset. The objective functions include i) task learning objective which can be further categorized into pointwise [49, 54, 40, 41, 12, 50], pairwise [5, 25, 23], triplet-wise [45, 32], listwise [52] and unsupervised [27, 14]; ii) quantization error minimization such as the loss designed to minimize the p-norm (usually p = 2) between continuous codes and hash codes; iii) code balancing [27, 40]. We refer readers to learning to hashing surveys [44, 43, 11] for more detailed review.
Binary optimization. Hashing is a NP-hard binary optimization problem [46], and is prone to the vanishing gradient problem due to the discrete and non-differentiable binary hash functions. Early methods solved the problem by discarding the discrete constraints (e.g., designing a penalty loss term to generate feature as binary as possible [25, 23]; solve with continuous relaxation, i.e., to optimize in a continuous space using sigmoid or tanh for approximation [5]). Some methods also utilized coordinate descent method in the training [28, 24]. Nevertheless, these methods have increased the complexity of learning due to need for tuning of hyper-parameters balancing different learning objectives.
Bypassing vanishing gradient. Greedy Hash [41] designed a new coding layer which uses the sign function in the forward pass to generate binary codes, and gradients are backpropagated using straight-through estimator [1] during optimization. [27] designed a parameter-free coding layer – Bi-half, to maximize the bit capacity by shifting the network output by median (each bit can have a 50% chance of being +1 or−1) . These methods typically requires the modification of computational graphs, in the sense that the original graph is no longer end-to-end trained, hence further complicates the original optimization objective. Ours on the other hand incorporates a neat one-loss design that removes all such complications.
Learning hash codes with pre-defined target. Deep Polarized Network (DPN) [12] used a random assignment scheme to generate target vectors with maximal inter-class distance, then optimized with hinge-like polarized loss. Central Similarity Quantization (CSQ) [50] uses Hadamard matrix as "hash centers", then optimized with binary cross entropy. Both methods have similar overall objective, i.e., the continuous codes are learned to be as similar as the target vectors (or "hash centers"). Our model also employs a hash target, but uniquely it is used in a single cosine similarity based single objective.
Cosine similarity. According to [6], which is a theoretical analysis for Locality-sensitive Hashing (LsH) [16, 13], if two samples have high angular similarity, then we have high probability of obtaining the same hash codes as well. Hence, while most works focus on hashing images with various constraints, we reformulate the problem of deep hashing in the lens of cosine similarity. By following the same principle, a similar work is done by [2] which described the hashing problem under pairwise constraint while our work describes the problem under pointwise constraint. As inspired by [51, 14] which utilize cosine similarity to find closest approximate binary or ternary representation, we also interpret the quantization error in terms of cosine similarity. Moreover, deep hypersphere embedding learning methods (e.g., SphereFace [35], CosFace [42] and ArcFace [10]) imposed discriminative constraints on a hypersphere manifold and proposed to improve decision boundary by cosine or angular margin. Inspired by them, we also leverage the benefit of margin to improve intra-class variance.

heading: 3 OrthoHash: One Loss for All
text: In Section 3.1, we reformulate the problem of deep hashing in the lens of cosine similarity, i.e., interpreting both Hamming distance retrieval and quantization error in cosine similarity. In Section 3.2, we propose to maximize cosine similarity between the continuous codes and binary orthogonal target under a single classification objective (for both single-label and multi-labels classification).
Finally, we describe why adding a batch normalization layer after the continuous codes will achieve code balance in Section 3.2.3. Our method is illustrated in Figure 2.
Let us first formally define the deep hashing problem. Let d-dimensional data, X = {xn}Nn=1 ∈ RN×d where N is the number of training samples, and Y = {yn}Nn=1 ∈ {0, 1}N×C as onehot training labels of C classes (for multi-labels, yn , yni = [yn1, · · · , ynC ], whose yni = 1 if any i-th class are assigned to the n-th sample and 0 otherwise). Our objective is to learn a set of K-bit binary codes B = {bn}Nn=1 ∈ {−1, 1}N×K for each training point xn, which is converted from the continuous codes vn through a sgn function. v can be computed by a latent layer H(x) = Wφ(x) ∈ RK , φ(·) is a deep neural network (backbone network) to compute q-dimensional nonlinear feature representation f = φ(x) ∈ Rq, W ∈ RK×q is the weights of the latent layer and sgn(vnk) = 1 if k-th bit of vn ≥ 0 and −1 otherwise. In our work, binary orthogonal targets oyn ∈ [o1, · · · ,oC ]ᵀ = O ∈ {−1,+1}C×K , where oi denotes a column vector belongs to i-th class. Ideally, for any two rows, 1 ≤ i, j ≤ C, oi and oj are orthogonal to each other. We use a or A to represent scalar, a to represent column vector, and A to represent matrix. Both i and j are often used as index.

heading: 3.1 Reformulating Deep Hashing in the Lens of Cosine Similarity
text: Interpreting Hamming Distance as Cosine Similarity. Typically, Hamming distance can be computed using logical xor operation between binary codes bi and bj , followed by popcount. If b is represented by {−1,+1}K , then Hamming distance can also be computed mathematically as:
D(bi,bj) = K − bᵀi bj
2 . (1)
Geometrically, the dot product bᵀi bj can be interpreted as:
bᵀi bj = ‖bi‖ ‖bj‖ cos θij , (2)
in which ‖·‖ is the Euclidean norm and θij is the angle between bi and bj . As both ‖bi‖ and ‖bj‖ are constant (i.e., ‖b‖ = √ K), equation (1) can then be viewed as:
D(bi,bj) = K −K cos θij 2 = K 2 (1− cos θij). (3)
Since K2 is a constant, we can see that the retrieval is now will be only based on the angle between two hash codes i.e., similar hash codes will have a similar direction, yield a lower angle between them, and hence a lower hamming distance.
Interpreting Quantization Error as Cosine Similarity. Typically, converting continuous codes v to binary codes b will lead to information loss, which is also known as quantization error. Therefore,
most of the existing hashing methods have included quantization error minimization in their learning objective such as L1-norm, L2-norm and p-norm (e.g., p = 3 in Greedy Hash [41]), usually in the form of:
minL+ λQ, (4)
where L is the supervised learning objective such as Cross Entropy and Q is the quantization error between v and b. However, it is difficult to control the scale λ, i.e. a low λ might not be effective, while a high λ might lead to underfitting. As a result of this, careful tuning is needed and yet the tuned λ may varies in different tasks. To overcome this cumbersome practise, let us first interpret quantization error geometrically:
min ‖v − b‖2 s.t. b ∈ {−1, 1}K , (5)
in which v is in continuous space, b = sgn(v) is in binary space. We expand equation (5) to get:
‖v − b‖2 = ‖v‖2 + ‖b‖2 − 2 ‖v‖ ‖b‖ cos θvb. (6)
According to equation (3), retrieval is only based on the similarity in the direction of two hash codes. Hence, we can ignore the magnitude of v by normalizing it to have the same norm with b, i.e., ‖v‖ = √ K and interpret the quantization error as to only the angle θvb between v and b3:
‖v − b‖2 = 2K − 2K cos θvb = 2K(1− cos θvb). (7)
Since 2K is a constant, we can then conclude that maximize the cosine similarity between v and b will lead to a low quantization error, leading to a better approximation in the hash codes.

heading: 3.2 Discriminative Hash Codes with Orthogonal Target
text: According to [6], the probability of two samples xi and xj to have the same hash code under a family F of hash functions using random hyperplane technique can be described as Prh∈F [h(xi) = h(xj)] = 1− θijπ , where h(·) is a hash function and θij is the angle between xi and xj . Therefore, based on the same principle, it can be derived that if the two continuous codes vi and vj from latent layerH have high cosine similarity, then the hash codes bi and bj should also have high chance of obtaining the same hash codes. Beside that, as described in Section 3.1, cosine similarity can also be used to justify the retrieval performance using both the hash codes and quantization error between the continuous codes and hash codes. Given these two circumstances, we therefore propose to maximize the cosine similarity of the continuous codes vn and its corresponding binary orthogonal target, oyn ∈ [o1, · · · ,oC ]ᵀ = O ∈ {−1,+1}C×K , where this can be achieved by maximizing the posterior probability of the ground-truth class using softmax (cross-entropy) loss:
L = − 1 N N∑ n=1 log exp (oᵀynvn)∑C i=1 exp (o ᵀ i vn) , (8)
where vn denotes the deep continuous codes of the n-th samples from DNN φ and both oyn , oi ∈ O denote the ground-truth class yn and the i-th class of the binary orthogonal targets. For simplicity, we omit the bias term from equation (8). It follows that under the framework of deep hypersphere embedding [35, 42, 10], we can transform the logit oᵀi vn = ‖oi‖ ‖vn‖ cos θni where θni is the angle between the continuous codes vn and the binary orthogonal target oi. Next, we perform L2 normalization on vn so that ‖vn‖ = 1, and ‖oi‖ = √ K since it is in binary form. Now our loss function can be rewritten as:
L = − 1 N N∑ n=1 log exp ( √ K cos θyn) exp ( √ K cos θyn) + ∑C i=1,i6=yn exp ( √ K cos θni) . (9)
As such, instead of introducing the quantization error minimization in the learning objective (equation (4)), our proposed method unifies both the learning objective and quantization error minimization together under a single classification objective as shown in the loss function (equation (9)). Furthermore, since the binary orthogonal targets attain maximal inter-class Hamming distance and that our loss function also aims to minimize the intra-class variance, we can leverage on cosine or angular
3See Appendix B in supplementary material for proof.
margin4 that have been proven to be beneficial in CosFace [42] and ArcFace [10], to further improve the minimization of intra-class variance (we set m = 0.2 in all of our experiments unless mentioned explicitly). With this, our method is able to perform end-to-end training to learn highly discriminative hash codes without both the sophisticated training objectives and computational graph modifications.

heading: 3.2.1 Binary Orthogonal Target
text: The maximization of the expectation of inter-class Hamming distance will help to increase the recall rate during retrieval as there will be lesser chance to retrieve incorrect items, because the aim is to retrieve more similar items (intra-class), and avoid to retrieve incorrect items (interclass). That is, given a K-bit Hamming space HK ∈ {−1,+1}K , for any two binary vectors bi,bj sampled with probability p for +1 on each bit, the expectation of Hamming distance is E[D(bi,bj)] = 2 ·K · p(1− p) and it achieves the upper bound of K2 with p = 0.5 [12, 50] (See Appendix B in supplementary material for details.). Hence, hash codes bi and bj must be orthogonal so that we can get D(bi,bj) = K2 in equation (3).
Orthogonal Targets Generation. Hadamard matrix naturally contains orthogonal rows and columns, which guarantees the maximum Hamming distance of K2 between any two rows [50, 29]. However, it is restricted when K is not 1, 2, or a multiple of 4. Hence, a simple solution is to sample the targets from Bern(0.5) which every sampled bit has the probability p = 0.5 to be +1. The result is the expectation of Hamming distance between any two rows equals to K2 which indicates orthogonality. One limitation is that if 2K < C, the nearest rows in the sampled targets will be identical, which causes performance degrade. Hence, a simple solution is to increase K. In supplementary material (Appendix D.3), we show that the two nearest rows has Hamming distance closed to K2 as K is higher. We also generate the targets with the objective of maximum inter-class Hamming distance heuristically, it indeed improved the performance at lower K, but the improvement in higher K are negligible.

heading: 3.2.2 Multi-labels Hash Codes Learning
text: As conventional cross-entropy loss only works for single-label classification, we leverage the concept of Label Smoothing [38] to generate labels for multi-labels classification. A standard cross entropy (CE) loss is mathematically formulated as:
L = − 1 N N∑ n=1 C∑ i=1 yni log(p(yni|xn)), (10)
in which yni = 1 if i-th class is assigned to the n-th sample in a single label multiclass classification task. In [38], the target label becomes soft-target such that non-target class has a small "smoothing" value to regularize overconfident samples and we leverage this concept for multi-labels. To adopt CE for multi labels classification, we set yni = z > 0 if any i-th class are assigned to n-th sample. The constant z is determined such that ∑C i=1 yni = 1, e.g., z = 0.5 and yn = [0, 0.5, 0, 0.5] when the 2nd and the 4th classes are the assigned classes. Our motivation is that the model should maximize the probabilities of the target classes, which can optimize the hash codes to be as similar as the binary targets from assigned classes5. In our experiments, we found out empirically that replacing softmax with sigmoid for multi-labels are not effective6. A likely explanation is that softmax will intrinsically suppress the lower activated class unit (i.e., scaled cosine similarity) with lower probability and increase the highly activated class unit with higher probability, while sigmoid will treat each class unit as an individual unit. As a result, maximizing probability of a class might not lead to minimizing the probability of other classes. Therefore, we propose to leverage the concept of Label Smoothing to generate labels so that we can use cross entropy loss for learning.
4Cosine margin will transform exp ( √ K cos θyn) to exp ( √ K(cos(θyn)−m)) and angular margin will
transform the same to exp ( √ K cos(θyn +m)).
5Note, we cannot guarantee that the final hash codes are the center of hash codes of the target classes. Instead we let the optimization algorithm to find the best hash codes.
6See Appendix D.4 in supplementary material for details.

heading: 3.2.3 Code Balance
text: Although binary orthogonal target helps in code balancing, since every bit has 50% of chance being +1 or −1, there is no guarantee that the model will learn to output a balanced code. Therefore, we propose to add a batch normalization (BN) layer after the continuous codes v to ensure the code balance. If ∑ n vnk = 0, then we can see that ∑ n bnk = 0 for the k-th bit. Because the distribution of v has been normalized to have zero-mean and variance of 1, with b = sgn(v), the hash codes b will follow a uniform binary distribution with 50% chances on both +1 and −1. Empirically, we found that it improves the retrieval performance on ImageNet100 by about 17-20% as compared with a model with normal cross entropy loss (see Table 1). Note that the Bi-half method [27] shifts the continuous codes by their median, followed by converting the continuous codes to binary codes for optimization. However, it will have to modify the computational graph in order to have a proxy derivative to the solve vanishing gradient problem. In contrast, appending BN layer will not modify the computational graph, therefore enabling straightforward end-to-end training.

heading: 4 Experiment
text: Training Setup. We select 7 different deep hashing methods for comparison (5 point-wise, 1 pairwise and 1 triplet-wise). For a fair comparison, we use the same learning rate of 0.0001, Adam optimizer [18] and 100 epochs for all methods. For SDH-C [31], we have modified it from pair-wise objective to point-wise objective, while all penalty terms are kept (i.e., quantization loss, bit variance loss and orthogonality on projection weights).
Datasets. We follow prior works [5, 12, 41, 33, 40, 48, 23, 34] and choose ImageNet100 [9], NUSWIDE [8] and MS-COCO [30] for category-level retrieval experiments. For a more practical yet challenging large-scale instance-level retrieval task (i.e., tremendous number of classes), we evaluate on the popular GLDv2 [47],ROxf andRPar [39]. Architecture. For category-level retrieval, following the settings in [5, 41, 40, 12], we use pre-trained AlexNet [20] as the network backbone initialization. The output from last fully-connected with ReLU (4096-dimension vector) acts as input to the latent layer; various supervised deep hashing methods are then applied to generate binary codes. The image size is 224 × 224. For instance-level retrieval, due to the expensive cost of training from scratch, we use pre-trained model7 (R50-DELG-GLDv2clean) from DELG [3] to compute the 2048-dimension global descriptors. We then train a latent layer H to compute hash codes where inputs are the global descriptors. For GLDv2, the images input are 512 × 512. ForROxf andRPar, we use 3 scales { 1√
2 , 1, √ 2} to produce multi-scale representations.
These are subject to L2 normalization, and then average-pooled to obtain a single descriptor as done by [3]. A GLDv2-trained latent layer is used to compute hash codes for the evaluations.
Details of training setups, datasets and architecture can be found in the supplementary material (Appendix C).

heading: 4.1 Results on Category-level Retrieval
text: For performance evaluation8, we use mean average precision (mAP@R) which is the mean of average precision scores of the top R retrieved items. Table 1 offers performance comparison amongst all selected hashing methods and our methods (+variants). CE denotes model trained with cross entropy only, the hash codes are computed from sign of continuous codes. CE+BN denotes CE model with BN layer [17] appended after the latent layer. CE+Bihalf denotes CE model with Bihalf9 [27] layer appended after the latent layer. OrthoCos denotes model trained with cosine margin and binary orthogonal target. OrthoCos+Bihalf denotes a variant of OrthoCos, and with Bihalf layer appended. OrthoCos+BN denotes a variant of OrthoCos, and with BN layer appended. OrthoArc+BN denotes a variant of OrthoCos+BN, trained with angular margin.
Overall. It can be observed that both our OrthoCos+BN and OrthoArc+BN perform better than recent state-of-the-art, DPN [12] and CSQ [50]. On multi-labeled datasets (i.e., NUS-WIDE and MS
7https://github.com/tensorflow/models/tree/master/research/delf 8See Appendix C.3 in supplementary material for evaluation detail. 9Note that Bihalf was originally proposed for unsupervised hashing, however, we think it is worth to compare
as Bihalf is a layer for code balancing instead of a training objective.
COCO), DTSH [45] (triplet based method) performed the best with 0.851 and 0.862 with 64 and 128-bits hash codes in NUS-WIDE followed by our method (e.g., OrthoCos+BN achieves 0.850 and 0.856 in the same settings), while OrthoCos+BN and OrthoArc+BN performed the best on MS-COCO with at most 1% improvement over previous deep hashing methods.
Code Balance. Although retrieval performance of CE models performed the worst, but by appending BN layer after the latent layer (CE+BN), we were able to observe 5-20% improvement over all settings (dataset and number of bits). Bihalf [27] layer (zero-median features) has a proxy derivative to learn hash features, hence getting 0.1-4.9% improvement than CE+BN. This indicates that without sophisticated training objectives, code balance itself is a very important factor in improving Hamming distance based retrieval. However, OrthoCos+Bihalf does not show significant improvement over OrthoCos+BN, but is comparable with OrthoCos. We thus conclude that our method can achieve code balance without explicitly engineering the computational graph.
Cosine and Angular Margin. In our experiments, we observed that cosine margin (OrthoCos+BN) slightly outperform angular margin (OrthoArc+BN) by about 0.2% on average.

heading: 4.2 Results on Instance-Level Retrieval
text: For evaluation metrics, we adapt the evaluation protocol of [3, 39]. The baseline performance of GLDv2, ROxf-Hard and RPar-Hard from the pre-trained R50-DELG-GLDv2-clean are 0.138,
0.510 and 0.715 respectively. Table 2 summarizes the performance of different deep hashing methods and our method. For all the 3 datasets, our method outperforms all previous deep hashing methods on all bits. This suggests that our method has a better generalization ability on unseen instances than previous deep hashing methods. In particular, our model significantly outperforms previous deep hashing models by 0.6%, 9.1% and 17.1% respectively on the 3 datasets with 128-bits hash codes.
Orthogonal Transformation. For GLDv2 2048-bits hash codes, surprisingly it can achieve a much better performance than the pre-trained 2048-dimensions descriptors (by 1.1% improvement over R50-DELG-C). We then analyze the separability in cosine distances, i.e., the difference in the mean of intra-class cosine distance and the mean of inter-class cosine distance before and after the transformation (similar to Figure 3). We observe that the separability in cosine distances increases after the orthogonal transformation, i.e., before it is 0.142 and after it increases to 0.167. The results thus show that learning orthogonal hash codes can transform the inputs to be more discriminative.
Domain shifting with BN. As the model is trained with GLDv2, the running mean and variance in the BN layer might experience domain shifting problem [26] when testing directly on different datasets (e.g.,ROxf andRPar). We empirically found that using running mean and variance from GLDv2 will lead to a large performance drop in Hamming distance retrieval10. One simple solution is to recompute the mean and variance from all continuous codes in the database, then update the running mean and variance with the computed mean and variance. The performances ofROxf and RPar in Table 2 are obtained with running mean and variance of the respective database.

heading: 4.3 Further Analysis
text: Histogram of Hamming Distances. Figure 3 summarizes the histogram of intra-class and interclass distances. We compare our method OrthoCos+BN with pair-wise method HashNet [5] and point-wise classification based GreedyHash [41]. Although the distribution of inter-class distances are about the same for all the 3 methods (close to Hamming distance of K/2 = 32), we can see that the larger the separability i.e., the difference in the mean of intra-class distance (the blue dotted line) with the mean of inter-class distance (the orange dotted line), the better the performance.
10See Appendix D.5 in supplementary material for details.
Performance Improvement Analysis. We further analyze the reasons behind performance improvements of different deep hashing methods, and summarizes the results in Figure 4. We conclude 3 main reasons that contribute to the improvement in deep hashing methods: i) quantization error; ii) the separability in Hamming distances; and iii) orthogonality in hash centers. For quantization error, we measure the angle θvb between the continuous codes v and the hash codes b, i.e., θvb = deg(arccos ( v ᵀb ‖v‖‖b‖ )). For separability, we measure the difference in the mean of inter-class distances and the mean of intra-class distances, i.e., E[Dinter]− E[Dintra]. For orthogonality, we first compute the hash centers H ∈ {−1,+1}C×K for every class (by taking the sign of average hash codes in every class), then we measure the orthogonality with ∥∥ 1 KHH ᵀ − I ∥∥ (lower is better). When the quantization error reduces, the separability increases and the hash centers has better orthogonality, resulting in better performance.

heading: 5 Conclusion & Future Work
text: We propose to unify training objectives of deep hashing under a single classification objective. We show this can be achieved by maximizing the cosine similarity between the continuous codes and binary orthogonal target under a cross entropy loss. For that, we first reformulated the problem of deep hashing in the lens of cosine similarity. We then demonstrated that if we perform L2-normalization on the continuous codes, then end-to-end training of deep hashing is possible without any extra sophisticated constraints. Moreover, we leverage the concept of Label Smoothing to train multilabels classification with cross-entropy loss and batch normalization for code balancing. Extensive experiments validated the efficiency of our method in both category-level and instance-level retrieval benchmarks.
Nonetheless, the proposed method might fail when the number of bits is too small (<8 bits), especially when number of classes is much greater than the number of bits. In this case, there will be overlapping in the generated target code (i.e., the number of maximum unique codes is equal to 2K where K is number of bits). In such condition, the target code will also not guarantee to be orthogonal. Overcoming this limitation is part of the future work. Also, we are exploring how to learn better feature representations to improve the retrieval performance by using hash codes through unsupervised learning.
Broader Impact
Hashing remains a key bottleneck in practical deployments of large-scale retrieval systems. Recent deep hashing frameworks have shown great promise in learning code that are both compact and discriminative. Yet state-of-the-art frameworks are known to be difficult to train and to reproduce – largely owing to their complex loss designs that dictates hyperparameter tuning and multi-stage training. In this work, we set out to change that – we attempt to unify deep hashing under a single objective, therefore simplifying training and help reproducibility. Our key intuition lies with reformulating hashing in the lens of cosine similarity. We report competitive hashing performance on all common datasets, and significant improvements over state-of-the-arts on the more challenging task of instance-level retrieval.
Acknowledgments and Disclosure of Funding
This research is partly supported by the Fundamental Research Grant Scheme (FRGS) MoHE Grant FP021-2018A, from the Ministry of Education Malaysia. We also thank Kilho Shin for helpful discussions and recommendations.

Review summaries:
The paper proposes a deep hashing model with only a single learning objective. It avoids the problem that existing deep hashing models are difficult to train due to a large number (>4) of losses. Specifically, it maximizes the cosine similarity between the continuous codes and their corresponding binary orthogonal codes to ensure both the discriminative capability of hash codes and the quantization error minimization. Besides, it adopts a Batch Normalization layer to ensure code balance and leverages the Label Smoothing strategy to modify the Cross-Entropy loss to tackle multi-labels classification. Extensive experiments show that the proposed method achieves better performance compared with the state-of-the-art multi-loss hashing methods on several benchmark datasets.

In this paper, authors propose a novel deep hashing model with only a single learning objective and claim that state of the art papers generally use a large number (>4) of losses. Specifically, authors show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hashcode discriminativeness and quantization error minimization. Authors show significance of their method using extensive experiments.  

This paper presents OrthoHash, a new deep hashing model that is trained by maximizing the cosine similarity between the continuous codes and the corresponding binary orthogonal target. In addition, code balancing is achieved by using batch normalization. Different from prior works that typically involves multiple loss functions, the training objective in this paper is simpler and easy to optimize.