Revisiting Multi-Codebook Quantization

Multi-Codebook Quantization ( MCQ ) is a generalized version of existing codebook-1 based quantizations for Approximate Nearest Neighbor ( ANN ) search . Therefore,2 MCQ theoretically has the potential to achieve the best performance because so-3 lutions of other codebook-based quantization methods are all covered by MCQ ’ s4 solution space under the same codebook size setting . However , finding the opti-5 mal solution to MCQ is proved to be NP-hard due to its encoding process , i.e.,6 converting an input vector to a binary code . To tackle this , researchers apply7 constraints to it to find near-optimal solutions , or employ heuristic algorithms8 which are still time-consuming for encoding . Different from previous approaches,9 this paper takes the first attempt to find a deep solution to MCQ . The encoding10 network is designed to be as simple as possible , so the very complex encoding11 problem becomes simply a feed-forward . Compared with other methods on three12 datasets , our method shows state-of-the-art performance . Notably , our method13 is 11×-38× faster than heuristic algorithms for encoding , which makes it more14 practical for real scenery of large-scale retrieval . Our code is publicly available:15 https : //github.com/DeepMCQ/DeepQ.16 1 Introduction17 Rapidly increasing multimedia contents in recent years raise an urgent request for retrieval in a18 short time . Unlike the exhaustive routine [ 31 , 20 ] , Approximate Nearest Neighbor ( ANN ) search19 significantly reduces retrieval time while preserving high recall . It has been widely applied to various20 scenarios , such as database indexing , fast image retrieval , and recommender systems.21 As a typical approach , vector quantization ( VQ ) [ 7 ] is at first developed as a compression technique,22 which uses a codebook to approximate vectors . People further find the power of VQ to preserve23 similarities between quantized features and enable VQ to perform ANN search . In order to achieve24 low quantization errors with limited codebook size , a multi-codebook structure is introduced . The25 proposal of the Multi-Codebook Quantization ( MCQ ) [ 2 ] describes the approach as a combination26 of one codeword for each sub-codebook , and previous methods [ 9 , 6 , 19 , 30 , 10 , 3 ] are summarized27 as exceptional cases of MCQ or constrained MCQs . The quantization codes are designed to be28 compacted , which results in negligible storage cost and high-quality results.29 However , the optimization of MCQ without any constraints is formally NP-hard . [ 14 ] models30 it as the minimization on several fully-connected Markov Random Fields ( MRFs ) . As a result,31 current researches aim at solving MCQ under acceptable computational costs . Other than applying32 constraints on it [ 34 , 4 , 15 ] , another approach designs algorithms in a heuristic way [ 2 , 14 , 16 ] . The33 latter achieves better performance but suffers from slow encoding.34 There are chances to employ neural networks ’ power to solve MCQ , where people expect to obtain35 higher performance and encoding efficiency than previous methods . [ 11 , 5 , 28 , 33 , 27 ] already give36 the way to treat codebook as network parameter and update it by gradient-descent , but they are37 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . all still under constraints that hinder performance . Morozov and Babenko [ 18 ] and Sablayrolles et38 al . [ 22 ] map datapoints to learned space , which are not flexible , especially when performing the39 reconstruction . Therefore in this paper , we give our first attempt to solve MCQ in a deep learning40 approach , without constraints and work-arounds . Our contributions can be summarized as three-folds:41 • Our novel approach , Deep Multi-Codebook Quantization ( DeepQ ) , fully considers encoding42 difficulty and time complexity in MCQ . With the high efficient and parallelized encoding networks,43 our method significantly reduces encoding time.44 • To tackle the NP-hard encoding problem and non-differentiable gradient estimation , we employ and45 further revise a policy gradient method . Value-Corrected Proximal Policy Optimization ( VC-PPO ) 46 is proposed to speed up convergence in the training phase.47 • Experiments conducted on a benchmark dataset validate our proposed method . Furthermore , to48 evaluate the scalability of the method , it is tested on million-scale datasets to show the effectiveness49 of our proposed algorithm.50 2 Related Works51 Vector quantization is a routine to approximate vectors by a codebook . Typical applications include52 clustering , compression , and Approximate Nearest Neighbor ( ANN ) search . The famous proposal53 k-means [ 7 ] , also known as Lloyd ’ s algorithm [ 13 ] , clusters the dataset into uniformly sized convex54 cells . When it is applied to ANN search , datapoints from the base set are quantized into their55 nearest centriods and represented by indices . The distance from a given query to any datapoint56 is approximated by the distance from the query to the datapoint ’ s centriod , which is effectively57 pre-computed and stored in a lookup table . To perform fine-grained clustering as well as reducing the58 space and time complexity , they [ 9 , 6 , 19 , 10 , 30 ] divide the feature space orthogonally by performing59 k-means in each subspace concurrently . Meanwhile , the introduced sub-codebook structure reveals60 the prototype of MCQ . Formally , [ 2 ] gives a well definition of MCQ , and previous works are all61 summarized into constrained MCQs . Specifically , subspace k-means must keep orthogonality among62 sub-codebooks . Zhang et al . [ 34 ] loosens the orthogonality constraint , but sub-codebooks are still63 weakly-orthogonal . Chen et al . [ 4 ] and Martinez et al . [ 15 ] propose hierarchical k-means , where64 vectors are quantized coarse-to-fine . If constraints are moved , MCQ is not easy to solve . Current65 state-of-the-art methods develop heuristic algorithms to help to encode . Specifically , Babenko and66 Lempitsky [ 2 ] employs beam search , Martinez et al . [ 14 , 16 ] give algorithm based on Iterated67 Conditional Modes ( ICM ) . However , the above methods do not achieve satisfied time complexity in68 encoding yet.69 When neural networks and gradient descent become a fashion , a few attempts to integrate quantization70 into deep retrieval networks are proposed . Klein and Wolf [ 11 ] and Song et al . [ 5 ] propose Deep71 Product Quantization ( DPQ ) and Deep Progressive Quantization ( DPgQ ) which update codebook by72 soft relaxation , but they are still under the same constraints as [ 9 , 15 ] . Sablayrolles et al . [ 22 ] and73 Morozov and Babenko [ 18 ] give pipelines to encode compact representations for compressed-domain74 search , but they do not strictly follow the paradigm of MCQ.75 3 Preliminaries76 Given a vector x ∈ RD , its quantized vector x̃ are composed by several codewords in a codebook77 C. More Specifically , C = ( Cm ) , Cm ∈ RK×D , 1 ≤ m ≤M contains M sub-codebooks and K78 codewords for each . Quantization codes are formed by b = ( bm ) , bm ∈ { 1 , 2 , · · · , K } , 1 ≤ m ≤79 M , which indicates the picked codeword in each sub-codebook . For the whole training setX = { x } 80 with N datapoints , MCQ aims at finding the optimal quantization codesB = { b } and codebook C81 to minimize following objective:82 min C , B E x∈X b∈B Q ( x , b , C ) = min C , B E x∈X b∈B ∥∥∥∥∥x− M∑ m=1 Cmbm ∥∥∥∥∥ 2 ( 1 ) whereCmbm ∈ RD is the bm-th codeword of the m-th sub-codebook . The sum of picked codewords83 ∑ Cmbm tries to approximate x . C and b are stored for further retrieval . Some of the previously84 mentioned methods [ 9 , 6 , 4 , 15 , 34 ] are treated as constrained MCQs , as they are all represented85 as special cases of ( 1 ) . Specifically , when M = 1 , ( 1 ) becomes VQ . Or if any two sub-codebooks86 Ci , Cj are orthogonal , it will be PQ or OPQ.87 The optimization of ( 1 ) without any constraints is proved to be NP-hard [ 14 ] . To tackle this , we88 propose a Expectation-Maximization style solution . Following sections will explain the deep neural89 network for encoding b ( Section 4.1 ) , the way to solve C ( Section 4.2 ) , and how to conduct retrieval90 ( Section 4.3 ) , respectively.91 4 Methodology92 4.1 Expectation : EncodingB with neural networks93 Our first step , is to find a potential code b by given x and94 a fixed C. A policy π parameterized by θ is employed to95 take possible solution of b by feeding x:96 π = ( πm ) = π ( x | θm ) , 1 ≤ m ≤M . ( 2 ) More specifically , π produces M Categorical distribu-97 tions Categorical ( K , pm1 , · · · , pmK ) , where pmj is98 the probability to pick the j-th codeword in the m-th99 sub-codebook . A potential encoding bm is generated by100 drawing samples from πm , which then helps us to pick101 codeword Cmbm . Therefore:102 bm ∼ πm ( x | θm ) = Categorical ( K , pm ) . ( 3 ) Since the independence among different sub-codebooks103 is a prerequisite of MCQ , bm should be drawn from πm104 independently . Intuitively , the probability of b to be a105 specific b ? is derived by conditional independence:106 Pr ( b=b ? ) = M∏ m=1 Pr ( bm=b ? m ) = M∏ m=1 pmb ? m . ( 4 ) We adopt the power of neural networks to model107 πm . Specifically , θm produces K unnormalized log-108 probabilities ` m and pmj is obtained by Softmax . To109 keep the independence , θm will not share parameters with110 each other.111 Therefore , θ , or our proposed IndepNet is illustrated in112 Figure 1 . We first build a basic structure called IndepBlock and duplicate this block for M times as113 θ1 , θ2 , · · · , θM . We try to keep the basic structure really simple to achieve high efficiency during114 training and encoding . As the figure shows , IndepBlock is an hourglass network contains 6 layer-115 groups ( consists of a linear layer with ReLU activation and layer-normalization ) with skip-connections.116 The last three outputs are concatenated and further fed into a final linear layer with K outputs as117 ` m = ( ` m1 , · · · , ` mK ) , and therefore:118 pmj = Softmax ( ` m ) j , where ` m = θm ( x ) . ( 5 ) 4.1.1 Gradient estimation119 The objective of training θ is formed as:120 min π E x∈X b∼π ( x|θ ) Q ( x , b , C ) . ( 6 ) However , the optimization faces two problems : 1 ) The encoding of b involves sampling from discrete121 distributions , which is non-differentiable , 2 ) All possible encoding of b is O ( KM ) . Exhaustive122 search becomes impracticable.123 Therefore , gradient estimation over discrete , stochastic computation graph is required to train θ.124 Mainstream methods [ 23 , 32 , 17 ] include score function gradient estimator , pathwise gradient125 estimator , etc . Meanwhile , minimizing ( 6 ) is also faced with the high-variance problem during126 gradient estimation . To tackle this , the advantage function is introduced [ 12 , 25 ] . Specifically in127 our work , a value network called QENet parameterized by τ is proposed to model a value function128 v = V ( · | τ ) . It performs a regression task to minimize the following objectives:129 min τ E x∈X b∼π ( x|θ ) ‖Q ( x , b , C ) −V ( x , b , C | τ ) ‖2 . ( 7 ) Advantages Â is then estimated by130 Â = Q ( x , b , C ) −V ( x , b , C | τ ) . ( 8 ) The detailed architecture of QENet is shown in Figure 2.131 We reuse the IndepBlock to generate v by M + 1 blocks:132 τ = ( τ1 , · · · , τM , τx ) . Specifically , latent representation133 for each selected-codeword Cmbm is obtained by:134 ιm = τm ( Cmbm ) . ( 9 ) The last IndepBlock τx is introduced to transform x. Then,135 all the outputs from IndepBlocks are summed up to get136 scalar value v ( denoted as “ reduce-sum ” ) :137 v = sum ( ι1 , · · · , ιM , ιx ) . ( 10 ) Value-corrected proximal policy optimization We138 propose a variant of score function gradient estimator139 called Value Corrected Proximal Policy Optimization ( VC-140 PPO ) based on PPO to get simple but efficient Trust Re-141 gion updates [ 26 , 24 ] . In the real scenario of large-scale142 ANN search , the training size N is usually larger than 10k . Conventional PPO still does not satisfy143 us due to the speed of convergence . Therefore , we revise and propose the Value-Corrected PPO144 ( VC-PPO ) to achieve fast training . Firstly in the sampling stage , bo and vo is produced from datapoint145 x over whole training setX by freezing current policy network and value network as θo , τo:146 bo ∼ π ( x | θo ) , vo = V ( x , bo , C | τo ) . ( 11 ) The probability of producing the sampled bo is denoted as po = Pr ( bo | x , θo ) , calculated by147 equation ( 4 ) . Finally , our surrogate objectives of VC-PPO is defined as [ 8 ] :148 Lθ = min ( Pr ( bo | x , θ ) Pr ( bo | x , θo ) Â , clip1+ 1− ( Pr ( bo | x , θ ) Pr ( bo | x , θo ) ) Â ) , ( 12 ) 149 Lτ = max ( ( Q ( x , bo , C ) −V ( x , bo , C | τ ) ) 2 , ( Q ( x , bo , C ) − vo − clip+ − ( V ( x , bo , C | τ ) − vo ) ) 2 ) . ( 13 ) Here , The clip ( · ) forces the policy and value to be not too far from old ones and is the clip-range.150 In both equations , it prevents a large update ratio leading to an unstable policy . The key difference151 between the original PPO and our VC-PPO is , we use V ( x , bo , C | τ ) other than the recorded old152 value vo from sampling stage to estimate advantage . This modification is treated as a value-correction153 process . Correcting value leads to a precise estimation on advantage , which is based on two reasons:154 a ) Biases are introduced into advantage estimation if we use vo , since the policy is getting better and155 better during training but τo is froze , and 2 ) The calculation of V ( x , bo , C | τ ) can be done instantly156 without introducing significant computational overhead . To further encourage the network choose157 codewords uniformly , a regularization is applied to θ to maximize the entropy of π:158 eθ = − M∑ m=1 K∑ j=1 pmj log pmj ( 14 ) which forces network to try more codeword combinations.159 4.2 Maximization : Solve C by least-squares160 To give the closed-form derivation of solving C by givenX andB , We will firstly rewrite Equation161 ( 1 ) to a matrix formulation . Since b = ( b1 , b2 , · · · , bM ) and bm ∈ { 1 , 2 , · · ·K } is the index of162 selected codeword in the i-th sub-codebook , a one-hot encoding and a concatenation on each bm:163 b′m = one-hot ( bm ) , b ′ = ( b′1 , · · · , b ′ m ) will convert the quantization code to a M -hot vector i.e . a164 vector that contains M segments , and each segment contains exactly one 1 and remaining 0 , where165 1 is the entry of picked codeword . Correspondingly , a reshape is applied to C : C ′ = C1 C2 ... CM ∈166 R ( M×K ) ×D . ( 1 ) will become:167 min C′ ∥∥X −B′C ′∥∥2 2 . ( 15 ) This equation is formally a linear least-squares regression , where B′ ∈ { 0 , 1 } N× ( M×K ) is known168 and X is target . Although there is a bunch of algorithms to solve it , we finally choose gelsy [ 1 ] ,169 which in our experiments shows the best results . The solution is to first apply a QR factorization with170 column permutation onB′ :171 B′ = Q ( R11 R12 0 R22 ) P ᵀ ( 16 ) where Q and R = ( R11 R12 0 R22 ) is the factorization matrix and P is an orthogonal matrix that172 permutes columns ofB′ untilR11 is well-conditioned ( its estimated condition number approaches173 0 ) . With the permutation , R22 becomes negligible . Moreover , R12 is erased by another orthogonal174 transformation:175 ( R11 R12 0 R22 ) → ( R11 R12 0 0 ) = ( T 11 0 0 0 ) Z ( 17 ) where T and Z are from the orthogonal transformation ofR . Then , C ′ is derived by:176 B′ = Q ( T 11 0 0 0 ) ZP ᵀ , C ← C ′ ← PZᵀ ( T−111 Q ᵀ 1X 0 ) ( 18 ) whereQ1 is the top rank ( B ′ ) columns ofQ.177 In brief , our overall training approach is summarized into algorithm 1.178 4.3 Fast retrieval179 After training , we are able to encode the base set for retrieval . Other than sampling from π , codewords180 are simply rolled out by greedy assignments:181 bgm = arg max θm ( x ) . ( 19 ) We firstly use the greedy roll-out strategy to obtain B in the training set in order to solve the final182 codebook . Then , we employ the same strategy to encode the base set.183 To further refine assignments , we add an extra step that randomly selects and alters bi while fixing184 others:185 bgi ← arg min bgi Q ( x , bg , C ) , i ∼ U [ 1 , M ] . ( 20 ) Since this refinement only causes negligible overhead referred to the implementation by [ 14 ] , in186 practice , we benefit from it not only to get lower quantization error but also to obtain acceptable187 performance from a fast training , i.e. , training within a very few steps before the network is converged.188 The encoded and refined base set , combined with the codebook , is finally employed for retrieval . The189 LSQ-style lookup table [ 14 ] is utilized to speed up similarity search.190 4.4 Discussion191 Algorithm 1 : VC-PPO for Training Inputs : Training setX , max step T , hyperparameters α , , learning rates η1 , η2 . Outputs : Policy π. Initialize codebook C , parameters θ and τ ; i← 0 ; while i < T do / * Training loop * / for x inX do / * Sampling stage * / Sample bo ∼ π ( x | θo ) intoB ; Compute vo , po into V , Pr ; end for x , bo , vo , po inX , B , V , Pr do / * Updating stage * / τ ← τ − η1∇τLτ ; Compute Â by ( 8 ) ; θ ← θ + η2∇θ ( Lθ + α · eθ ) ; end C ← Solved by ( 15 ) ∼ ( 18 ) ; i← i+ 1 ; end return π ( · | θ ) Our work aims at solving Multi-Codebook Quanti-192 zation via neural networks . Similar works include193 Unsupervised Neural Quantization ( UNQ ) [ 18 ] and194 Spreading Vectors [ 22 ] . But ours has several key195 advantages compared to previous works : 1 ) Unlike196 UNQ , which reconstructs features by an encoder-197 decoder structure , we follow the paradigm of MCQ198 to directly give binary codes and codebooks for the199 benefit of speed and storage , for UNQ needs an200 extra decoding stage during retrieval . 2 ) UNQ and201 Spreading Vectors both project original features202 into a learned space . Although similarities between203 features are preserved , they still have biases in quan-204 tized results . This causes several issues , especially205 when we want to perform a reconstruction to ap-206 proximate original features , e.g . data compression.207 Compared to LSQ [ 14 ] , the state-of-the-art heuris-208 tic algorithm , our work is the first to tackle MCQ in209 a deep learning fashion . The policy network is de-210 signed to be very simple to get fast encoding speed211 and comparable retrieval performance.212 5 Experiments213 Our proposed Deep Multi-Codebook Quantization ( DeepQ ) is compared against the state-of-the-arts214 on a visual-feature dataset ( LabelMe22K ) to evaluate retrieval performance and encoding speed.215 Then , we scale up to make comparisons on commonly used large-scale datasets ( SIFT1M and216 DEEP1M ) , whose base sets include 1 million vectors for retrieval . Furthermore , ablation study on217 SIFT1M investigates the effectiveness of each component in our proposed pipeline.218 5.1 Datasets and evaluation metrics219 LabelMe22K [ 29 ] : This dataset collects images by the LabelMe annotation tool1 and uses Convolu-220 tional Neural Network ( CNN ) to extract them into 512-d features . It has 22 , 019 vectors for training221 and 2 , 000 vectors for test.222 SIFT1M2 and DEEP1M3 : Both datasets contain 104 , 105 , 106 vectors in query , training and base223 set , respectively . Vectors from SIFT1M is extracted by Scale-Invariant Feature Transform ( 128-d ) 224 while DEEP1M contains 96-d vectors from outputs of a CNN.225 1https : //github.com/CSAILVision/LabelMeAnnotationTool 2http : //corpus-texmex.irisa.fr/ 3http : //sites.skoltech.ru/compvision/noimi/ Recall @ { 1 , 10 , 100 } and quantization error are adopted as evaluation metrics . These two metrics226 indicate not only the retrieval performance but also the reconstruction accuracy . Because LabelMe22K227 does not have a base set , its training set is adopted as a base set . We train on the training set , and then228 encode the base set for evaluations with queries . When calculating recall , groundtruth is defined as229 the nearest neighbor of each query in the base set ( sorted by l2 distance ) . As for quantization error,230 the average value of Q ( x , b , C ) is reported over all x in the base set.231 We compare our proposal with both shallow and deep methods , including three classic quantiza-232 tion : OPQ [ 6 ] , SQ [ 15 ] and LSQ++ [ 14 , 16 ] ( denoted as LSQ for simplicity . Also , these two in233 our experiments have similar performance ) , as well as three graident-based methods : DPQ [ 11 ] ,234 DPgQ [ 5 ] and DRQ [ 28 ] . DPQ and PQNet [ 33 ] have basically the same architecture that extend235 PQ with gradient-descent , so we only report the performance of DPQ . Additionally , UNQ [ 18 ] is236 also included , although they introduce an extra decoder and re-ranking trick for retrieval.237 5.2 Implementation details238 Our method is implemented with Py-239 Torch,4 the popular deep learning240 package in Python . Codebook C is241 solved by Intel MKL that has been242 fully optimized for speed . As for net-243 work training , we adopt Adam opti-244 mizer with AMSGrad [ 21 ] and hyper-245 parameters are tuned by grid search.246 Specifically , learning rates η1 = η2 =247 2× 10−4 , with an exponetial learning248 rate decay γ = 0.9999 . Batch-size249 in updating stage is 2000 , while other250 hyper-parameters = 0.2 , α = 0.05.251 Additionally , during training , we in-252 sert dropout layers after every layer-253 normalization in all layer-groups to254 tackle overfitting . More detailed set-255 tings as well as specifications of In-256 depNet θ and QENet τ on each dataset257 ( LabelMe22K , SIFT1M , DEEP1M ) can be found in supplementary material.258 As for quantization code-lengths , K = 256 codewords for each sub-codebook and M = { 4 , 8 } 259 sub-codebooks are employed in total . We follow [ 2 ] to report “ effective ” code-lengths ( additional260 code-length for storing ‖x‖ for lookup table is ignored ) . Therefore code-lengths become { 32 , 64 } 261 bits , respectively.262 For a fair comparison , experiments are conducted on a single machine , equipped with Intel Xeon263 E5-2678v3 CPU , 256 GiB RAM , and NVIDIA RTX 3090 GPU . For other methods , we re-run on all264 datasets under unified settings with implementations provided by the authors.265 5.3 Comparisons with state-of-the-arts266 Under the small training set and base set settings on LabelMe22K , we get the results placed in267 Table 1 . Our method takes the highest recall on this dataset , outperforming the state-of-the-art268 by 2.20 % , 7.85 % , 3.30 % on 32 bits for R @ 1 , R @ 10 and R @ 100 . It also outperforms the best269 competitor by 2.70 % , 1.45 % , 0.65 % on 64 bits . In brief , All methods except for UNQ are generally270 split into three styles : 1 ) PQ-like : OPQ and DPQ . 2 ) SQ-like : SQ , DPgQ and DRQ . 3 ) MCQ : LSQ271 and ours . Generally , DPQ , DPgQ , and DRQ achieve similar results compared to their shallow272 versions . However , since they are still constrained MCQs , they show worse performances than 3 ) .273 The performance of LSQ is worse than ours , shows the effectiveness of neural networks for modeling274 the MCQ encoding problem . As for UNQ , it takes several extra tricks i.e. , another network for275 decoding and re-ranking in retrieval . Although it beats LSQ , our network still shows the power of276 MCQ to win the competition.277 4https : //pytorch.org/ 5.3.1 Large-scale retrieval performance278 Our evaluations on SIFT1M and DEEP1M datasets is presented in Table 2 . The training set and279 base set are scaled up , and retrievals on these datasets become more difficult . We observe expected280 results on two datasets . Compared to our main competitor , LSQ , our method achieves comparable281 performance on SIFT1M , and outperforms LSQ on DEEP1M in most cases . Our method achieves282 higher recall on DEEP1M than SIFT1M . A potential reason is that DEEP1M is under a nearly normal283 distribution that , in practice , is easier to converge than SIFT1M , which has a larger variance between284 datapoints . The performance of UNQ in our experiments is lower than expected , possibly due to285 different dataset settings.286 Another key advantage of our method is that , different from shallow methods , which are hand-crafted287 algorithms that find possible solutions manually or with constraints , our DeepQ encodes vectors by288 only a feed-forward.289 5.3.2 Encoding efficiency290 In order to verify the encoding efficiency of291 our method , evaluations of encoding time on292 SIFT1M with the 106 base set are conducted293 by checking the total time spent . All of them294 are run under GPU-acceleration . Additionally,295 we evaluate the time with and without the extra296 codewords refinement that introduced in section297 4.3 ( 128 bits results are simulated ) . As Figure298 3 shows , our network is significantly faster than299 LSQ since it needs to perform local search it-300 eratively for 25 or even 100 rounds . Specifi-301 cally , to encode SIFT1M base set , LSQ takes302 52.84s , 96.99s , 256.86s and 639.18s for 16 , 32,303 64 and 128 bits respectively . By contrast , our304 method takes 4.46s , 5.46s , 8.26s and 16.64s,305 which is 11.8× , 17.8× , 31.1× and 38.4× faster306 than LSQ . Moreover , our method is even faster307 than most of the constrained MCQs . We also308 notice that the refinement takes negligible over-309 head . Although UNQ takes the fastest encoding310 speed , it still needs to decode and re-rank during311 retrieval , which slows down its retrieval speed.312 5.3.3 Reconstruction accuracy313 The comparisons of quantization error on three314 datasets are stated in Table 3 . Basically , when the quantization error gets lower , recall will be higher.315 Ours get the 2nd place on SIFT1M , and the lowest on remaining datasets in most cases . Quantization316 error indicates reconstruction accuracy and further shows the quality of codebook generation and317 quantization codes selection . Notably , ours significantly outperforms UNQ , which has a strong bias318 on the reconstruction task . This is because they focus more on ranking , not the quantization error.319 The result shows that our method can be applied to other areas , e.g . vector compression.320 5.4 Ablation study321 Method SIFT1M @ 32 bitsQE R @ 1 R @ 10 R @ 100 w/o regularization 3.38× 104 7.60 29.96 68.73 w/o return-norm 3.06× 104 10.57 36.44 76.04 w/o correction 3.10× 104 10.09 35.30 75.16 w/o refinement 3.17× 104 9.91 30.39 68.28 DeepQ 2.92× 104 11.02 37.73 76.79 Table 4 : Ablation study conducted on SIFT1M with 32 bits code-length . Entropy regularization forces network to try more codeword combinations , which help to jump out of local-optima . Return normalization and value correction help for fast convergence . The extra refinement leads to low quantization error and high recall with negligible costs . Our ablation study is conducted on the322 SIFT1M dataset , with the code-length of323 32 bits , which in our experiments is suf-324 ficient to show how does each component325 affects our model . We choose the following326 variants to perform ablation:327 w/o regularization : which removes eθ in328 the losses , and the output distributions will329 not be forced to be uniform.330 w/o return-norm : which does not normal-331 ize R , and therefor advantage is computed332 by R other than R̄.333 w/o correction : which removes value cor-334 rection . So our VC-PPO falls back to the original PPO.335 w/o refinement : which directly encode the base set without extra refinement.336 Quantization error and recall are evaluated and placed in Table 4 . We report the best value they337 ever met during the training procedure . Specifically , when regularization is removed , it seems that338 the network is trapped in local-optima and the performance drops . Meanwhile , although return339 normalization and value correction give us only subtle improvements , we find they help the network340 to converge quickly . The extra refinement gives us lower quantization error and higher recall , specially341 when we want to perform fast training before the network is converged.342 6 Conclusion and Future Work343 In this paper , we first review previous works of constrained MCQs , and investigate solutions to344 unconstrained ones . Since finding the global-optima of MCQ is NP-hard , researchers apply constraints345 to find near-optimal solutions or employ heuristic algorithms that are still time-consuming . This346 paper takes the first attempt to find a deep solution to MCQ . The proposed IndepNet is designed to be347 simple enough to encode vectors extremely fast . Furthermore , our network shows state-of-the-art348 performance in retrieval and reconstruction tasks . Our method is slow to converge in a large dataset,349 which hinders our performance . So , our future work will focus on training speedup.350 References351 [ 1 ] Anderson , E. , Bai , Z. , Dongarra , J. J. , Greenbaum , A. , McKenney , A. , Croz , J. D. , Hammarling , S.,352 Demmel , J. , Bischof , C. H. , and Sorensen , D. C. ( 1990 ) . LAPACK : a portable linear algebra library for353 high-performance computers . In SC , pages 2–11.354 [ 2 ] Babenko , A. and Lempitsky , V. ( 2014 ) . Additive quantization for extreme vector compression . In CVPR,355 pages 931–938.356 [ 3 ] Babenko , A. and Lempitsky , V. ( 2015 ) . Tree quantization for large-scale similarity search and classification.357 In CVPR , pages 4240–4248.358 [ 4 ] Chen , Y. , Guan , T. , and Wang , C. ( 2010 ) . Approximate nearest neighbor search by residual vector359 quantization . Sensors , 10 ( 12 ) :11259–11273.360 [ 5 ] Gao , L. , Zhu , X. , Song , J. , Zhao , Z. , and Shen , H. T. ( 2019 ) . Beyond product quantization : Deep progressive361 quantization for image retrieval . In IJCAI , pages 723–729.362 [ 6 ] Ge , T. , He , K. , Ke , Q. , and Sun , J . ( 2013 ) . Optimized product quantization for approximate nearest neighbor363 search . In CVPR , pages 2946–2953.364 [ 7 ] Gray , R. ( 1984 ) . Vector quantization . IEEE Assp Magazine , 1 ( 2 ) :4–29.365 [ 8 ] Ilyas , A. , Engstrom , L. , Santurkar , S. , Tsipras , D. , Janoos , F. , Rudolph , L. , and Madry , A . ( 2020 ) . A closer366 look at deep policy gradients . In ICLR.367 [ 9 ] Jégou , H. , Douze , M. , and Schmid , C. ( 2010 ) . Product quantization for nearest neighbor search . IEEE Trans.368 Pattern Anal . Mach . Intell. , 33 ( 1 ) :117–128.369 [ 10 ] Kalantidis , Y. and Avrithis , Y . ( 2014 ) . Locally optimized product quantization for approximate nearest370 neighbor search . In CVPR , pages 2329–2336.371 [ 11 ] Klein , B. and Wolf , L. ( 2019 ) . End-to-end supervised product quantization for image search and retrieval.372 In CVPR , pages 5041–5050.373 [ 12 ] Konda , V. R. and Tsitsiklis , J. N. ( 2000 ) . Actor-critic algorithms . In NeurIPS , pages 1008–1014.374 [ 13 ] Lloyd , S. ( 1982 ) . Least squares quantization in pcm . IEEE transactions on information theory , 28 ( 2 ) :129–375 137.376 [ 14 ] Martinez , J. , Clement , J. , Hoos , H. H. , and Little , J. J . ( 2016 ) . Revisiting additive quantization . In ECCV,377 pages 137–153 . Springer.378 [ 15 ] Martinez , J. , Hoos , H. H. , and Little , J. J . ( 2014 ) . Stacked quantizers for compositional vector compression.379 arXiv preprint arXiv:1411.2173.380 [ 16 ] Martinez , J. , Zakhmi , S. , Hoos , H. H. , and Little , J. J . ( 2018 ) . Lsq++ : Lower running time and higher381 recall in multi-codebook quantization . In ECCV , pages 491–506.382 [ 17 ] Mohamed , S. , Rosca , M. , Figurnov , M. , and Mnih , A . ( 2020 ) . Monte carlo gradient estimation in machine383 learning . J. Mach . Learn . Res. , 21:132:1–132:62.384 [ 18 ] Morozov , S. and Babenko , A . ( 2019 ) . Unsupervised neural quantization for compressed-domain similarity385 search . In ICCV , pages 3036–3045.386 [ 19 ] Norouzi , M. and Fleet , D. J . ( 2013 ) . Cartesian k-means . In CVPR , pages 3017–3024.387 [ 20 ] Radenovic , F. , Iscen , A. , Tolias , G. , Avrithis , Y. , and Chum , O . ( 2018 ) . Revisiting oxford and paris:388 Large-scale image retrieval benchmarking . In CVPR , pages 5706–5715.389 [ 21 ] Reddi , S. J. , Kale , S. , and Kumar , S. ( 2018 ) . On the convergence of adam and beyond . In ICLR.390 [ 22 ] Sablayrolles , A. , Douze , M. , Schmid , C. , and Jégou , H. ( 2019 ) . Spreading vectors for similarity search . In391 ICLR.392 [ 23 ] Schulman , J. , Heess , N. , Weber , T. , and Abbeel , P. ( 2015a ) . Gradient estimation using stochastic computa-393 tion graphs . In NeurIPS , pages 3528–3536.394 [ 24 ] Schulman , J. , Levine , S. , Abbeel , P. , Jordan , M. I. , and Moritz , P. ( 2015b ) . Trust region policy optimization.395 In ICML , pages 1889–1897.396 [ 25 ] Schulman , J. , Moritz , P. , Levine , S. , Jordan , M. I. , and Abbeel , P. ( 2016 ) . High-dimensional continuous397 control using generalized advantage estimation . In ICLR.398 [ 26 ] Schulman , J. , Wolski , F. , Dhariwal , P. , Radford , A. , and Klimov , O . ( 2017 ) . Proximal policy optimization399 algorithms . arXiv preprint arXiv:1707.06347.400 [ 27 ] Song , J. , Lang , R. , Zhu , X. , Xu , X. , Gao , L. , and Shen , H. T. ( 2020 ) . 3d self-attention for unsupervised401 video quantization . In ACM SIGIR , pages 1061–1070.402 [ 28 ] Song , J. , Zhu , X. , Gao , L. , Xu , X.-S. , Liu , W. , and Shen , H. T. ( 2019 ) . Deep recurrent quantization for403 generating sequential binary codes . In IJCAI , pages 912–918.404 [ 29 ] Torralba , A. , Fergus , R. , and Weiss , Y . ( 2008 ) . Small codes and large image databases for recognition . In405 CVPR , pages 1–8 . IEEE.406 [ 30 ] Wang , J. , Wang , J. , Song , J. , Xu , X. , Shen , H. T. , and Li , S. ( 2015 ) . Optimized cartesian k-means . IEEE407 Transactions on Knowledge and Data Engineering , 27 ( 1 ) :180–192.408 [ 31 ] Weyand , T. , Araujo , A. , Cao , B. , and Sim , J . ( 2020 ) . Google landmarks dataset v2 - A large-scale409 benchmark for instance-level recognition and retrieval . In CVPR , pages 2572–2581.410 [ 32 ] Williams , R. J . ( 1992 ) . Simple statistical gradient-following algorithms for connectionist reinforcement411 learning . Machine learning , 8 ( 3-4 ) :229–256.412 [ 33 ] Yu , T. , Yuan , J. , Fang , C. , and Jin , H. ( 2018 ) . Product quantization network for fast image retrieval . In413 ECCV , pages 186–201.414 [ 34 ] Zhang , T. , Du , C. , and Wang , J . ( 2014 ) . Composite quantization for approximate nearest neighbor search.415 In ICML , volume 2 , page 3.416 Checklist417 The checklist follows the references . Please read the checklist guidelines carefully for information on418 how to answer these questions . For each question , change the default [ TODO ] to [ Yes ] , [ No ] , or419 [ N/A ] . You are strongly encouraged to include a justification to your answer , either by referencing420 the appropriate section of your paper or providing a brief inline description . For example:421 • Did you include the license to the code and datasets ? [ Yes ] See Section ? ? .422 • Did you include the license to the code and datasets ? [ No ] The code and the data are proprietary.423 • Did you include the license to the code and datasets ? [ N/A ] 424 Please do not modify the questions and only use the provided macros for your answers . Note that the425 Checklist section does not count towards the page limit . In your paper , please delete this instructions426 block and only keep the Checklist section heading above along with the questions/answers below.427 1 . For all authors ... 428 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s429 contributions and scope ? [ Yes ] 430 ( b ) Did you describe the limitations of your work ? [ Yes ] See Section 6.431 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 432 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to them ? 433 [ Yes ] 434 2 . If you are including theoretical results ... 435 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 436 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 437 3 . If you ran experiments ... 438 ( a ) Did you include the code , data , and instructions needed to reproduce the main experimental439 results ( either in the supplemental material or as a URL ) ? [ Yes ] See https : //github.com/440 DeepMCQ/DeepQ.441 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were442 chosen ) ? [ Yes ] See Section 5 and supplementary materials.443 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experiments444 multiple times ) ? [ No ] 445 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type of446 GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Section 5.447 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 448 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See Section 5.449 ( b ) Did you mention the license of the assets ? [ N/A ] 450 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] See451 https : //github.com/DeepMCQ/DeepQ.452 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re453 using/curating ? [ N/A ] 454 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable455 information or offensive content ? [ N/A ] 456 5 . If you used crowdsourcing or conducted research with human subjects ... 457 ( a ) Did you include the full text of instructions given to participants and screenshots , if applica-458 ble ? [ N/A ] 459 ( b ) Did you describe any potential participant risks , with links to Institutional Review Board460 ( IRB ) approvals , if applicable ? [ N/A ] 461 ( c ) Did you include the estimated hourly wage paid to participants and the total amount spent on462 participant compensation ? [ N/A ] 463

Summary: This paper introduces DeepQ, a multi-codebook quantization framework with NN-based methods.   In DeepQ, code b is encoded by neural networks. To tackle the non-differentiable gradient estimation issue, a revised policy gradient method has been adapted here to accelerate convergence.   The effectiveness of the proposed methods is well demonstrated on multiple datasets.