LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes

Learning binary representations of instances and classes is a classical problem with several high potential applications . In modern settings , the compression of high-dimensional neural representations to low-dimensional binary codes is a challenging task and often require large bit-codes to be accurate . In this work , we propose a novel method for Learning Low-dimensional binary Codes ( LLC ) for instances as well as classes . Our method does not require any side-information , like annotated attributes or label meta-data , and learns extremely low-dimensional binary codes ( ≈ 20 bits for ImageNet-1K ) . The learnt codes are super-efficient while still ensuring nearly optimal classification accuracy for ResNet50 on ImageNet-1K . We demonstrate that the learnt codes capture intrinsically important features in the data , by discovering an intuitive taxonomy over classes . We further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution ( OOD ) detection problems . For ImageNet-100 retrieval problem , our learnt binary codes outperform 16 bit HashNet using only 10 bits and also are as accurate as 10 dimensional real representations . Finally , our learnt binary codes can perform OOD detection , out-of-the-box , as accurately as a baseline that needs ≈ 3000 samples to tune its threshold , while we require none . Code is open-sourced at https : //github.com/RAIVNLab/LLC . 1 Introduction . Embedding data in low-dimensional binary space is a long-standing machine learning problem [ 56 ] . The problem has received a lot of interest in the computer vision ( CV ) domain , where the goal is to find binary codes that capture the key semantics of the image , like , objects present in the image or interpretable attributes . Section 2 covers the literature on learning binary codes and their applications . In addition to learning semantically meaningful representations of the instances , low-dimensional binary codes allow efficiency in a variety of large-scale machine learning ( ML ) applications . Lowdimensional codes are crucial in extreme classification with millions of classes [ 8 , 30 , 15 ] and also critical in efficient large-scale retrieval settings [ 38 , 16 , 59 ] . Compressing information into binary codes is challenging due to its highly non-smooth nature while requiring the preservation of relevant information in an instance/class . This might explain the lack of good classification accuracy for existing classification-based embedding techniques [ 28 , 13 ] . To address that , traditional methods often relied on side-information like attributes to construct class codes and then use that to learn the instance codes [ 18 , 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Learning binary embeddings can be posed in a variety of formulations like pairwise optimization [ 34 ] or unsupervised learning [ 11 , 50 ] , in this work we focus on learning binary codes using a given labeled multi-class dataset , e.g. , ImageNet-1K . This allows us to couple the representation ( code ) learning of both instances and classes thus enabling us to capture the underlying semantic structure efficiently to assist in downstream tasks like classification , retrieval etc . We propose LLC , a method to learn both class and instance codes via the standard classification task and its setup without any side-information . Our Learning Low-dimensional binary Codes ( LLC ) technique , formulates the embedding ( code ) learning problem as that of learning a low-dimensional binary embedding of a standard deep neural “ backbone ” . Instead of directly training for the lowdimensional binary instance codes , we propose a two-phase approach . In the first phase , LLC learns low-dimensional ( k-bit ) binary codes for classes that capture semantic information through a surrogate classification task . Then in the second phase , LLC uses these learnt class codes as an efficient alternative to learning instance codes in sub-linear cost ( in the number of classes , L ) using the Error-Correcting Output Codes ( ECOC ) approach [ 19 ] . This two-phase pipeline helps in the effective distillation of required semantic similarity between instances through the learnt class codes . For example , on ImageNet-1K with ResNet50 , LLC is able to learn tight 20-bit codes that can be used for efficient classification and achieve 74.5 % accuracy compared to the standard baseline 77 % on ImageNet-1K ( Section 4.1 ) . Furthermore , we observe that the learnt 20-bit class codes capture intuitive taxonomy over classes ( Figure 1 ) while the instance codes retain the distilled class similarity information useful in efficient retrieval and OOD detection . Retrieval . To further study , the effectiveness of our learnt binary codes , we apply them to hashingbased efficient retrieval , where the goal is to retrieve a large number of similar instances with the same class label in top retrieved samples . Deep supervised hashing is a widely studied problem with several recent results [ 9 , 53 ] which are designed specifically for the learnt hashing-based retrieval . Interestingly , our learnt instance codes through the LLC routine provide strikingly better performance while not being learnt explicitly for hashing . For eg. , using AlexNet , with just 32-bit codes we are can provide 5.4 % more accurate retrieval than HashNet ’ s 64-bit codes on ImageNet-100 ( Section 4.2 ) . OOD Detection . We similarly apply LLC based learnt binary codes to detect OOD instances [ 26 ] . We adopt a simple approach based on our binary codes : if an instance is not within a Hamming distance of 1 to any class codes , we classify it as OOD . That is , we do not fine-tune our OOD detector for the new domain , which is critical in practical settings . In contrast , baseline techniques for OOD detection require a few samples ( eg. , ≈ 3000 for ImageNet-750 ) from the OOD domain to fine-tune thresholds , while we require no samples yet reaching similar OOD detection ( Section 4.3 ) . In this work , we make the following key contributions : • LLC method to learn semantically-rich low-dimensional binary codes for both classes & instances . • Show that the learnt codes enable accurate & efficient classification : ImageNet-1K with 20-bits . • Apply LLC to image retrieval task , and demonstrate that it comfortably outperforms the instance code learning methods for hashing-based retrieval on ImageNet-100 . • Finally , use codes from LLC for strong & sample efficient OOD detection in practical settings . 2 Related Work . Binary class codes were originally aimed at sub-linear training and prediction for multi-class classification . The Error-Correcting Output Codes ( ECOC ) framework [ 19 , 3 , 20 ] reformulated multi-class classification as multi-label classification using k-bit codes per class ( codebook ) . The learning of optimal codebook is NP-complete [ 14 ] which lead to use of random codebooks [ 28 , 13 ] in traditional ML . However , there were a few codebook learning [ 5 , 64 , 58 , 4 ] and construction schemes using side-information from other modalities [ 1 ] . The lack of a strong learnable feature extractor often deterred the gains these codebooks provide for the classification and effective learning of instance codes . Attribute annotations can also help in constructing class codes [ 2 ] . These binary codes are either explicitly annotated [ 21 ] or discovered [ 47 , 22 ] . Attributed-based learning also ties into leveraging the class codes for zero/few-shot learning [ 36 , 37 , 1 , 44 ] expecting some form of interpretability . Most methods that use class codes as supervision can produce instance codes [ 19 ] . However , the standalone literature of instance codes comes from requirements in large-scale application like retrieval ( hashing ) . In the past , most hashing techniques that created instance codes were based on random projections [ 16 , 13 , 12 ] , semantics [ 51 , 18 ] or learnt through metric learning [ 34 , 33 , 43 , 32 ] , clustering [ 57 , 50 ] and quantization [ 23 ] . Deep learning further helped in learning more accurate hashing functions to generate instance codes either in an unsupervised [ 11 , 52 ] or supervised [ 38 , 9 , 53 , 62 ] fashion . We refer to [ 39 , 63 , 56 ] for a more thorough review on deep hashing methods . Finally , embedding-based classification [ 13 , 61 , 8 , 24 ] enables joint low-dimensional representation learning for both classes and instances with an eye on sub-linear training and prediction costs . After distilling the key ideas from the literature , we aim to a ) learn semantically rich low-dimensional representations for both classes and instances together , b ) have these representations in the binary space , and c ) do this with minimal dependence on side-information or metadata . LLC , to the best of our knowledge - for the first time , jointly learns low-dimensional binary codes for both classes and instances using a surrogate classification task , without any side-information ( Section 3 ) . The learnt class codes capture intrinsic information at the semantic level that helps in discovering an intuitive taxonomy over classes ( Figure 1 ) . The learnt class codes then anchor the instance code learning which results in tight and accurate low-dimensional instance codes further used in retrieval ( Section 4.2 ) . Finally , both the learnt class and instance codes power extremely efficient yet accurate classification ( Section 4.1 ) and out-of-distribution detection ( Section 4.3 ) . 3 Learning Low-dimensional Binary Codes . The goal is to learn a binary embedding ( code ) function g : X → { −1 , 1 } k where X is the input domain and k is the dimensionality of the code . We focus on learning embeddings using a labelled multi-class data [ 28 ] . That is , suppose we are given a labelled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } where xi ∈ X is an input point and yi ∈ [ L ] is the label of xi for all i ∈ [ n ] . Then , the goal is to learn an instance embedding function g : X → { +1 , −1 } k and class embeddings hq ∈ { +1 , −1 } k for all q ∈ [ L ] such that g ( xi ) = hyi and g ( xi ) = g ( xj ) if and only if yi = yj . Intuitively , for large-scale datasets , g ( x ) and hq should capture key semantic information to provide accurate classification , thus allowing their use in application domains like retrieval or OOD detection . Note that while we present our technique for learning embeddings using multi-class datasets , it applies more generally to multi-labeled datasets as well . Instance and Class Code Parameterization . For learning such embedding function , we assume access to a deep neural architecture F ( · ; θF ) : X → Rd that maps the input x ∈ X to a d-dimensional real-valued representation . θF is a learnable parameterization of the network ; we drop θF from F wherever the meaning is clear from the context . For example , ResNet50 is one such network that encodes 224× 224 RGB images into d = 2048 dimensions . Now , given a network F and x ∈ X , we formulate embedding function of x and the corresponding multiclass prediction scores ŷ ∈ ZL as : g ( x ) : = B ( P · F ( x ; θF ) ) , ŷ : = B ( C ) · g ( x ) , ( 1 ) where P ∈ Rk×d maps F ( x ) into k-dimensions and B ( a ) = sign ( a ) ∈ { +1 , −1 } is the standard binarization/sign operator applied elementwise ( with the assumption sign ( 0 ) = +1 ) . Finally , C ∈ RL×k , and ŷ = B ( C ) · g ( x ) represents the scores of each class for an input x . Note that for a class ` ∈ [ L ] , B ( C ` ) ( where C ` represents the ` -th row of C ) is the learnt binary class embedding ( code ) of class ` ∈ [ L ] , and g ( x ) = B ( P · F ( x ; θF ) ) is the learnt instance embedding ( code ) of instance x . Note that ( 1 ) is a general purpose formulation for the problem of learning class and instance codes . 3.1 The LLC Method Phase 1 : Codebook Learning – B ( C ) . Given labelled examples D , we use standard empirical risk minimization to learn a multi-class classifier , i.e. , we solve min C , P , θF ∑ ( xi , yi ) ∈D L ( B ( C ) · ( P · F ( xi ; θF ) ) ; yi ) , ( 2 ) where L : RL × [ L ] → R+ is the standard multi-class softmax cross-entropy loss function . This is a standard optimization problem that can be solved using standard gradient descent methods or other sub-gradient based optimizers . However , one challenge is that B ( C ) is a binary matrix and B is a binary function , so the gradients are 0 almost everywhere . Instead , we use the Straight-Through Estimator ( STE ) [ 6 ] technique popular in binary neural networks domain [ 48 ] , to optimize for C through the binarization . Intuitively , STE uses binarization/sign function in the forward pass , but in the backpropagation phase , it allows the gradients to flow straight-through as if it were real-valued . The codebook , B ( C ) refers to the collection of k-bit class codes learnt in this process . For ImageNet-1K , we learnt unique binary codes , B ( C ` ) , for every class ` ∈ [ L ] of the 1000 classes using only 20-bits , only twice the information-theoretic limit . As with the class representations from a linear classifier , these class codes do capture intrinsically important features that help in discovering intuitive taxonomy over classes ( Section 3.2 ) among various applications ( Section 4 ) . Phase 2 : Instance Code Learning – B ( P · F ( x ; θF ) ) . Several existing techniques model C and P in different ways to learn an embedding function similar to ( 1 ) . However , these methods often try to only learn instance codes and have challenges in maintaining high accuracy [ 9 , 11 ] in a variety of applications because optimization problem ( 2 ) is challenging and might lead to significantly sub-optimal classification error . For example , for ImageNet-1K classification with ResNet50 , the accuracy for our trained model ( 20-bits ) at this stage is 72.5 % compared to the standard 77 % . To remedy this , we further optimize our embeddings using the ECOC framework [ 19 ] for multi-class classification , which essentially transforms the multi-class problem into a multi-label problem , which in turn is k independent binary classification problems . That is , we use the k-bit codes learnt for each class as the supervision to further train F ( · ; θF ) and P : min θF , P ∑ ( xi , yi ) ∈D k∑ j=1 BCE ( σ ( Pj · F ( xi ; θF ) ) ; ( B ( Cyi , j ) + 1 ) /2 ) , ( 3 ) where σ is the sigmoid/logistic function , BCE is the binary cross-entropy loss between the j-th bit of instance xi ’ s embedding , and the j-th bit extracted from the class embedding of it ’ s label yi ( the function z 7→ ( z + 1 ) /2 is used to map { +1 , −1 } to { 1 , 0 } to make it a simple binary classification problem per each bit ) . We use gradient based optimization to learn θF and P. As mentioned earlier , ECOC framework allows us to correct errors in classification . For example , with just 20 bits on ImageNet-1K dataset , the method now achieves 74.5 % accuracy with ResNet50 backbone . The advantage of this two-phase pipeline where we start with a codebook learning for classes is that the cost of learning instances codes reduces to a bottleneck of k-dims ( L ) instead of the usual L . Furthermore , these learnt low-dimensional binary codes for both classes and instances help in large-scale applications via efficient classification and retrieval ( see Section 4 ) . Note that , unlike attribute-based methods [ 37 ] , we do not require additional meta-data , but learn binary codes by only using the standard classification task . This also circumvents the potential instabilities of pairwise optimization in instance binary code learning which often leads to poor class codes due to codebook collapse . At the end of LLC routine , we have learnt the instance codes , B ( P · F ( x ; θF ) ) , and class codes , B ( C ) to be used for downstream applications . Algorithm 1 presents LLC in full . Algorithm 1 The LLC Method Input : D , F and B Output : C , P and θF 1 : Codebook Learning – B ( C ) : Solve ( 2 ) using ERM and STE to get C , P and θF - C , P , θF ← argmin C , P , θF ∑ ( xi , yi ) ∈D L ( B ( C ) · ( P · F ( xi ; θF ) ) ; yi ) . 2 : Instance Code Learning – B ( P · F ( x ; θf ) ) : Further optimize P and θF by solving ( 3 ) using ECOC framework and ERM by fixing C - θF , P ← argmin θF , P ∑ ( xi , yi ) ∈D k∑ j=1 BCE ( σ ( Pj · F ( xi ; θF ) ) ; ( B ( Cyi , j ) + 1 ) /2 ) . Overall , we present a simple yet scalable method to learn low-dimensional ( exact ) binary codes for both classes and instances which in turn could power multi-class classification with sub-linear costs ( in terms of L ) and efficient retrieval for large-scale applications . Using our method , we can consistently learn unique low-dimensional binary codes for all 1000 classes in ImageNet-1K using only 20-bits ( which is twice the information-theoretic limit of dlog2 ( 1000 ) e ) . Next , we discuss the learnt codebook ’ s intrinsic information about the classes and their structure .

Summary: This paper studies the problem of learning binary vector representations of instances and classes. We pick a small dimension k (the paper uses k=20) and aim to learn a k-dimensional binary vector for each input and each label. Compressing the data into small-dimensional binary codes like this is challenging, but provides many benefits in terms of the space usage (the representations are very small) and the time to do lookups and find nearby neighbors.  The paper presents a new method called LLC which simultaneously finds codewords for both inputs and labels without using any side-information. They claim (and I have no reason to doubt this) that this is the first time a method for this has been designed. LLC consists of two phases:  In the first phase, the LLC method learns a codebook (low-dimensional bianry code for each label) by using the popular Straight-Through Estimator technique to do empirical risk minimization.   In the second phase, the ECOC framework is used to classify the inputs according to this codebook. The algorithm treats each of the k bits of the code completely separately, and hence solves k disjoint binary classification problems instead of one multi-class (with L or 2^k classes) problem. This gives an additive 2% improvement to accuracy over just solving the optimization problem to classify all k bits together, since out of the box optimization finds fairly suboptimal solutions when trying to classify a length k vector all at once. (In other words, the second phase uses a heuristic where sepaartely learning each of the k bits works better than learning all k at once for the current optimization software which is well-suited to doing binary classification.)  The resulting classification accuracy seems quite good: 68.82% of inputs are assigned exactly the same binary code as their label, and 74.57% are assigned a binary code which is closer to the code of their label than any other label.  The result of this binary code classification can now be used in a number of applications, including: - the binary codebook can be naturally used to make a taxonomy visualization for the labels.  - the system can be used for efficient (in terms of representation size and running time of retrieval) multi-class classification. - out-of-distribution detection can be done by seeing whether an input is mapped to the same codeword as any label. (I'm confused about how effective this can be when the original inputs are only correctly mapped to the codeword of their label 69% of the time.)