LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes

Learning binary representations of instances and classes is a classical problem with several high potential applications . In modern settings , the compression of high-dimensional neural representations to low-dimensional binary codes is a challenging task and often require large bit-codes to be accurate . In this work , we propose a novel method for Learning Low-dimensional binary Codes ( LLC ) for instances as well as classes . Our method does not require any side-information , like annotated attributes or label meta-data , and learns extremely low-dimensional binary codes ( ≈ 20 bits for ImageNet-1K ) . The learnt codes are super-efficient while still ensuring nearly optimal classification accuracy for ResNet50 on ImageNet-1K . We demonstrate that the learnt codes capture intrinsically important features in the data , by discovering an intuitive taxonomy over classes . We further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution ( OOD ) detection problems . For ImageNet-100 retrieval problem , our learnt binary codes outperform 16 bit HashNet using only 10 bits and also are as accurate as 10 dimensional real representations . Finally , our learnt binary codes can perform OOD detection , out-of-the-box , as accurately as a baseline that needs ≈ 3000 samples to tune its threshold , while we require none . Code is open-sourced at https : //github.com/RAIVNLab/LLC . 1 Introduction . Embedding data in low-dimensional binary space is a long-standing machine learning problem [ 56 ] . The problem has received a lot of interest in the computer vision ( CV ) domain , where the goal is to find binary codes that capture the key semantics of the image , like , objects present in the image or interpretable attributes . Section 2 covers the literature on learning binary codes and their applications . In addition to learning semantically meaningful representations of the instances , low-dimensional binary codes allow efficiency in a variety of large-scale machine learning ( ML ) applications . Lowdimensional codes are crucial in extreme classification with millions of classes [ 8 , 30 , 15 ] and also critical in efficient large-scale retrieval settings [ 38 , 16 , 59 ] . Compressing information into binary codes is challenging due to its highly non-smooth nature while requiring the preservation of relevant information in an instance/class . This might explain the lack of good classification accuracy for existing classification-based embedding techniques [ 28 , 13 ] . To address that , traditional methods often relied on side-information like attributes to construct class codes and then use that to learn the instance codes [ 18 , 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Learning binary embeddings can be posed in a variety of formulations like pairwise optimization [ 34 ] or unsupervised learning [ 11 , 50 ] , in this work we focus on learning binary codes using a given labeled multi-class dataset , e.g. , ImageNet-1K . This allows us to couple the representation ( code ) learning of both instances and classes thus enabling us to capture the underlying semantic structure efficiently to assist in downstream tasks like classification , retrieval etc . We propose LLC , a method to learn both class and instance codes via the standard classification task and its setup without any side-information . Our Learning Low-dimensional binary Codes ( LLC ) technique , formulates the embedding ( code ) learning problem as that of learning a low-dimensional binary embedding of a standard deep neural “ backbone ” . Instead of directly training for the lowdimensional binary instance codes , we propose a two-phase approach . In the first phase , LLC learns low-dimensional ( k-bit ) binary codes for classes that capture semantic information through a surrogate classification task . Then in the second phase , LLC uses these learnt class codes as an efficient alternative to learning instance codes in sub-linear cost ( in the number of classes , L ) using the Error-Correcting Output Codes ( ECOC ) approach [ 19 ] . This two-phase pipeline helps in the effective distillation of required semantic similarity between instances through the learnt class codes . For example , on ImageNet-1K with ResNet50 , LLC is able to learn tight 20-bit codes that can be used for efficient classification and achieve 74.5 % accuracy compared to the standard baseline 77 % on ImageNet-1K ( Section 4.1 ) . Furthermore , we observe that the learnt 20-bit class codes capture intuitive taxonomy over classes ( Figure 1 ) while the instance codes retain the distilled class similarity information useful in efficient retrieval and OOD detection . Retrieval . To further study , the effectiveness of our learnt binary codes , we apply them to hashingbased efficient retrieval , where the goal is to retrieve a large number of similar instances with the same class label in top retrieved samples . Deep supervised hashing is a widely studied problem with several recent results [ 9 , 53 ] which are designed specifically for the learnt hashing-based retrieval . Interestingly , our learnt instance codes through the LLC routine provide strikingly better performance while not being learnt explicitly for hashing . For eg. , using AlexNet , with just 32-bit codes we are can provide 5.4 % more accurate retrieval than HashNet ’ s 64-bit codes on ImageNet-100 ( Section 4.2 ) . OOD Detection . We similarly apply LLC based learnt binary codes to detect OOD instances [ 26 ] . We adopt a simple approach based on our binary codes : if an instance is not within a Hamming distance of 1 to any class codes , we classify it as OOD . That is , we do not fine-tune our OOD detector for the new domain , which is critical in practical settings . In contrast , baseline techniques for OOD detection require a few samples ( eg. , ≈ 3000 for ImageNet-750 ) from the OOD domain to fine-tune thresholds , while we require no samples yet reaching similar OOD detection ( Section 4.3 ) . In this work , we make the following key contributions : • LLC method to learn semantically-rich low-dimensional binary codes for both classes & instances . • Show that the learnt codes enable accurate & efficient classification : ImageNet-1K with 20-bits . • Apply LLC to image retrieval task , and demonstrate that it comfortably outperforms the instance code learning methods for hashing-based retrieval on ImageNet-100 . • Finally , use codes from LLC for strong & sample efficient OOD detection in practical settings . 2 Related Work . Binary class codes were originally aimed at sub-linear training and prediction for multi-class classification . The Error-Correcting Output Codes ( ECOC ) framework [ 19 , 3 , 20 ] reformulated multi-class classification as multi-label classification using k-bit codes per class ( codebook ) . The learning of optimal codebook is NP-complete [ 14 ] which lead to use of random codebooks [ 28 , 13 ] in traditional ML . However , there were a few codebook learning [ 5 , 64 , 58 , 4 ] and construction schemes using side-information from other modalities [ 1 ] . The lack of a strong learnable feature extractor often deterred the gains these codebooks provide for the classification and effective learning of instance codes . Attribute annotations can also help in constructing class codes [ 2 ] . These binary codes are either explicitly annotated [ 21 ] or discovered [ 47 , 22 ] . Attributed-based learning also ties into leveraging the class codes for zero/few-shot learning [ 36 , 37 , 1 , 44 ] expecting some form of interpretability . Most methods that use class codes as supervision can produce instance codes [ 19 ] . However , the standalone literature of instance codes comes from requirements in large-scale application like retrieval ( hashing ) . In the past , most hashing techniques that created instance codes were based on random projections [ 16 , 13 , 12 ] , semantics [ 51 , 18 ] or learnt through metric learning [ 34 , 33 , 43 , 32 ] , clustering [ 57 , 50 ] and quantization [ 23 ] . Deep learning further helped in learning more accurate hashing functions to generate instance codes either in an unsupervised [ 11 , 52 ] or supervised [ 38 , 9 , 53 , 62 ] fashion . We refer to [ 39 , 63 , 56 ] for a more thorough review on deep hashing methods . Finally , embedding-based classification [ 13 , 61 , 8 , 24 ] enables joint low-dimensional representation learning for both classes and instances with an eye on sub-linear training and prediction costs . After distilling the key ideas from the literature , we aim to a ) learn semantically rich low-dimensional representations for both classes and instances together , b ) have these representations in the binary space , and c ) do this with minimal dependence on side-information or metadata . LLC , to the best of our knowledge - for the first time , jointly learns low-dimensional binary codes for both classes and instances using a surrogate classification task , without any side-information ( Section 3 ) . The learnt class codes capture intrinsic information at the semantic level that helps in discovering an intuitive taxonomy over classes ( Figure 1 ) . The learnt class codes then anchor the instance code learning which results in tight and accurate low-dimensional instance codes further used in retrieval ( Section 4.2 ) . Finally , both the learnt class and instance codes power extremely efficient yet accurate classification ( Section 4.1 ) and out-of-distribution detection ( Section 4.3 ) . 3 Learning Low-dimensional Binary Codes . The goal is to learn a binary embedding ( code ) function g : X → { −1 , 1 } k where X is the input domain and k is the dimensionality of the code . We focus on learning embeddings using a labelled multi-class data [ 28 ] . That is , suppose we are given a labelled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } where xi ∈ X is an input point and yi ∈ [ L ] is the label of xi for all i ∈ [ n ] . Then , the goal is to learn an instance embedding function g : X → { +1 , −1 } k and class embeddings hq ∈ { +1 , −1 } k for all q ∈ [ L ] such that g ( xi ) = hyi and g ( xi ) = g ( xj ) if and only if yi = yj . Intuitively , for large-scale datasets , g ( x ) and hq should capture key semantic information to provide accurate classification , thus allowing their use in application domains like retrieval or OOD detection . Note that while we present our technique for learning embeddings using multi-class datasets , it applies more generally to multi-labeled datasets as well . Instance and Class Code Parameterization . For learning such embedding function , we assume access to a deep neural architecture F ( · ; θF ) : X → Rd that maps the input x ∈ X to a d-dimensional real-valued representation . θF is a learnable parameterization of the network ; we drop θF from F wherever the meaning is clear from the context . For example , ResNet50 is one such network that encodes 224× 224 RGB images into d = 2048 dimensions . Now , given a network F and x ∈ X , we formulate embedding function of x and the corresponding multiclass prediction scores ŷ ∈ ZL as : g ( x ) : = B ( P · F ( x ; θF ) ) , ŷ : = B ( C ) · g ( x ) , ( 1 ) where P ∈ Rk×d maps F ( x ) into k-dimensions and B ( a ) = sign ( a ) ∈ { +1 , −1 } is the standard binarization/sign operator applied elementwise ( with the assumption sign ( 0 ) = +1 ) . Finally , C ∈ RL×k , and ŷ = B ( C ) · g ( x ) represents the scores of each class for an input x . Note that for a class ` ∈ [ L ] , B ( C ` ) ( where C ` represents the ` -th row of C ) is the learnt binary class embedding ( code ) of class ` ∈ [ L ] , and g ( x ) = B ( P · F ( x ; θF ) ) is the learnt instance embedding ( code ) of instance x . Note that ( 1 ) is a general purpose formulation for the problem of learning class and instance codes . 3.1 The LLC Method Phase 1 : Codebook Learning – B ( C ) . Given labelled examples D , we use standard empirical risk minimization to learn a multi-class classifier , i.e. , we solve min C , P , θF ∑ ( xi , yi ) ∈D L ( B ( C ) · ( P · F ( xi ; θF ) ) ; yi ) , ( 2 ) where L : RL × [ L ] → R+ is the standard multi-class softmax cross-entropy loss function . This is a standard optimization problem that can be solved using standard gradient descent methods or other sub-gradient based optimizers . However , one challenge is that B ( C ) is a binary matrix and B is a binary function , so the gradients are 0 almost everywhere . Instead , we use the Straight-Through Estimator ( STE ) [ 6 ] technique popular in binary neural networks domain [ 48 ] , to optimize for C through the binarization . Intuitively , STE uses binarization/sign function in the forward pass , but in the backpropagation phase , it allows the gradients to flow straight-through as if it were real-valued . The codebook , B ( C ) refers to the collection of k-bit class codes learnt in this process . For ImageNet-1K , we learnt unique binary codes , B ( C ` ) , for every class ` ∈ [ L ] of the 1000 classes using only 20-bits , only twice the information-theoretic limit . As with the class representations from a linear classifier , these class codes do capture intrinsically important features that help in discovering intuitive taxonomy over classes ( Section 3.2 ) among various applications ( Section 4 ) . Phase 2 : Instance Code Learning – B ( P · F ( x ; θF ) ) . Several existing techniques model C and P in different ways to learn an embedding function similar to ( 1 ) . However , these methods often try to only learn instance codes and have challenges in maintaining high accuracy [ 9 , 11 ] in a variety of applications because optimization problem ( 2 ) is challenging and might lead to significantly sub-optimal classification error . For example , for ImageNet-1K classification with ResNet50 , the accuracy for our trained model ( 20-bits ) at this stage is 72.5 % compared to the standard 77 % . To remedy this , we further optimize our embeddings using the ECOC framework [ 19 ] for multi-class classification , which essentially transforms the multi-class problem into a multi-label problem , which in turn is k independent binary classification problems . That is , we use the k-bit codes learnt for each class as the supervision to further train F ( · ; θF ) and P : min θF , P ∑ ( xi , yi ) ∈D k∑ j=1 BCE ( σ ( Pj · F ( xi ; θF ) ) ; ( B ( Cyi , j ) + 1 ) /2 ) , ( 3 ) where σ is the sigmoid/logistic function , BCE is the binary cross-entropy loss between the j-th bit of instance xi ’ s embedding , and the j-th bit extracted from the class embedding of it ’ s label yi ( the function z 7→ ( z + 1 ) /2 is used to map { +1 , −1 } to { 1 , 0 } to make it a simple binary classification problem per each bit ) . We use gradient based optimization to learn θF and P. As mentioned earlier , ECOC framework allows us to correct errors in classification . For example , with just 20 bits on ImageNet-1K dataset , the method now achieves 74.5 % accuracy with ResNet50 backbone . The advantage of this two-phase pipeline where we start with a codebook learning for classes is that the cost of learning instances codes reduces to a bottleneck of k-dims ( L ) instead of the usual L . Furthermore , these learnt low-dimensional binary codes for both classes and instances help in large-scale applications via efficient classification and retrieval ( see Section 4 ) . Note that , unlike attribute-based methods [ 37 ] , we do not require additional meta-data , but learn binary codes by only using the standard classification task . This also circumvents the potential instabilities of pairwise optimization in instance binary code learning which often leads to poor class codes due to codebook collapse . At the end of LLC routine , we have learnt the instance codes , B ( P · F ( x ; θF ) ) , and class codes , B ( C ) to be used for downstream applications . Algorithm 1 presents LLC in full . Algorithm 1 The LLC Method Input : D , F and B Output : C , P and θF 1 : Codebook Learning – B ( C ) : Solve ( 2 ) using ERM and STE to get C , P and θF - C , P , θF ← argmin C , P , θF ∑ ( xi , yi ) ∈D L ( B ( C ) · ( P · F ( xi ; θF ) ) ; yi ) . 2 : Instance Code Learning – B ( P · F ( x ; θf ) ) : Further optimize P and θF by solving ( 3 ) using ECOC framework and ERM by fixing C - θF , P ← argmin θF , P ∑ ( xi , yi ) ∈D k∑ j=1 BCE ( σ ( Pj · F ( xi ; θF ) ) ; ( B ( Cyi , j ) + 1 ) /2 ) . Overall , we present a simple yet scalable method to learn low-dimensional ( exact ) binary codes for both classes and instances which in turn could power multi-class classification with sub-linear costs ( in terms of L ) and efficient retrieval for large-scale applications . Using our method , we can consistently learn unique low-dimensional binary codes for all 1000 classes in ImageNet-1K using only 20-bits ( which is twice the information-theoretic limit of dlog2 ( 1000 ) e ) . Next , we discuss the learnt codebook ’ s intrinsic information about the classes and their structure .

Summary: This work proposes a two-phase method for learning low-dimensional binary codes via the standard classification task without side-information. Specifically, binary codes for classes are learned via a surrogate classification task in the first phase. On top of that, instance codes are learned using the Error-Correcting Output Codes approach.