SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu Liang $^{1,4*}$
Shiliang Zhang
Kenli  $\mathbf{L}\mathbf{i}^{1\dagger}$
Xiaoyu Wang $^{3\dagger}$
$^{1}$ College of Computer Science and Electronic Engineering, Hunan University  $^{2}$ National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
<sup>3</sup>The Hong Kong University of Science and Technology(Guangzhou) <sup>4</sup>Intellifusion Inc.
{leungyu, lkl}@hnu.edu.cn, slzhang.jdl@pku.edu.cn, fanghuaxue@gmail.com
Due to its promising performance, deep hashing has become a prevalent method for approximate nearest neighbors search (ANNs). However, most of current deep hashing methods are validated on relatively small-scale datasets, leaving potential threats when are applied to large-scale real-world scenarios. Specifically, they can be constrained either by the computational cost due to the large number of training categories and samples, or unsatisfactory accuracy. To tackle those issues, we propose a novel deep hashing framework based on product quantization (PQ). It uses a softmax-based differentiable PQ branch to learn a set of predefined PQ codes of the classes. Our method is easy to implement, does not involve large-scale matrix operations, and learns highly discriminate compact codes. We validate our method on multiple large-scaled datasets, including ImageNet100, ImageNet1K, and Glint360K, where the category size scales from 100 to 360K and sample number scales from 10K to 17 million, respectively. Extensive experiments demonstrate the superiority of our method. Code is available at https://github.com/yuleung/FPPQ.
Approximate Nearest Neighbors Search (ANNs) aims to quickly find necessary information within vast amounts of data. It is commonly used in image and video search [49, 37, 4], recommendation systems [44, 8], and anomaly detection [34, 1] tasks. Hashing [18, 7, 16, 37], which is a key component of ANNs, has been continuously and rapidly developed over time. Hashing enables the efficient mapping of high-dimensional floating-point data into binary code, with the main objective of minimizing storage space and retrieval time while maintaining satisfactory retrieval accuracy.
In the past decade, researchers have been exploring the use of deep learning to optimize hash technology for improved retrieval performance [18, 23, 7, 27, 12, 48]. Compared to traditional methods, deep hashing has shown significant improvement. Nevertheless, to the best of our knowledge, existing deep hash or deep quantization methods have not been comprehensively tested in large-scale data retrieval scenarios, which are commonly encountered in real-world applications. For example, in face recognition [9, 2, 52, 39, 29], where the underlying databases may contain millions of categories and hundreds of millions of samples. The large-scale datasets pose higher requirements to the
*Work done as an intern at Intellifusion.
efficiency of learning and inference, as well as the accuracy of deep hashing algorithms. As shown in our experiments, when tested on large-scale dataset, the classic product quantization (PQ) [20] outperforms many deep hashing methods.
As the one of the most popular methods for quantization retrieval, PQ, however, as shown in Fig. 2, its retrieval performance rapidly declines as the encoding length decreases, rendering it impractical in some applications. Shorter PQ codes can improve retrieval speed and reduce storage overhead, greatly expanding its use, for example, making it a more viable option for edge devices. Due to PQ's widespread use in various industrial applications and its advantages in large-scale data compared to other hash methods, we hope to further unlock the potential of PQ and use it for real-world large-scale image retrieval scenarios. Our objective is to address the challenge that PQ's performance rapidly decays with code length by obtaining a feature representation of the database that is better suited for cluster assignment in multiple sub-spaces. We hold the view that the poor performance of short PQ codes stems from insufficient clustering conditions in the subspace of the original feature vector, resulting in excessively large quantization errors and high error rates during retrieval.
Building on the aforementioned discussion, we propose an end-to-end deep product quantization method called FPPQ(Full Potential Product Quantization). Our approach is simple and intuitive: for the large-scale dataset, we generate a PQ code for each class, which is then utilized as a label for learning via a differentiable multi-segment fully connected branch. Upon completing training, the retrieval process is precisely the same as that of traditional PQ, allowing for easy integration into current mature retrieval modules.
The contribution of this paper can be summarized as three fold. Firstly, for the first time, we investigate the retrieval performance of current popular or advanced deep hash methods under extremely large datasets with an extensive number of categories and demonstrate that they may not be suitable for such scenarios. Secondly, we propose a simple yet effective end-to-end deep feature quantization compression method, based on class-level product quantization coding supervision. Our method achieves a low-slope decay of retrieval performance with decreasing code lengths and can be used for retrieval in the same way as the traditional PQ. Finally, we conducted extensive experiments on Glint360K [2], a large-scale datasets with 360k classes and 17 million images, as well as two relatively smaller scale datasets, Imagenet1K [35] and Imagenet100 [7, 35]. The results demonstrated the effectiveness of our method.
Hash technology is widely used for compressing data and speeding up retrieval processes in computer vision. The goal is typically to ensure that samples that are similar in their original space remain similar when they are mapped to the new hash space. Researchers have conducted extensive research on hash technology in recent years, and hash methods can be classified in several ways.
For example, hash methods can be categorized according to the calculation method of hash distance. Hamming distance-based methods aim to learn a binary code and calculate the distance between samples using the Hamming distance metric [16, 19, 32, 42, 46, 18, 37, 12, 25, 24, 40, 7, 36, 26]. Conversely, dictionary-based methods calculate distance by looking up a table through a codebook [20, 51, 15, 28, 13, 27, 23, 49, 45].
In an alternative way, hash methods can be categorized into non-deep and deep methods. Non-deep methods such as LSH [16], Composite Quantization (CQ) [50], Additive Quantization (AQ) [3], PQ [20], OPQ [14], LOPQ [21] are widely used for efficient retrieval. LSH is a data-independent random locality sensitive hash algorithm that can use a variety of methods such as random projection or buckets to directly perform hashing. PQ clusters the sub-vectors and represents original vectors by the cluster categories. OPQ and LOPQ improve upon PQ by transforming the original data to reduce quantization error and considering the local relationships of the vectors, respectively. Non-deep hashing methods have higher efficiency but lower performance, while deep hashing methods leverage the powerful representation learning abilities of neural networks to learn hash codes and feature representations simultaneously, reducing quantization error for better performance [18, 49, 7, 37, 23, 45, 27, 47, 48, 51]. Since the quantization process is not differentiable, researchers often employ the sigmoid function or tanh function to fit the sign function [7], or use the straight-through (ST) estimator [5] to avoid gradient non-transfer, as in the case of DPQ [23] and GreedyHash [37]. For more detailed information on deep hashing, please refer to [30].
Limitation: Despite the existence of many hashing methods with their own unique advantages, they unfortunately have limitations when applied to large-scale datasets. For example, in the face of large-scale data, the Gini Impurity related penalty constraint item designed by DPQ [23] is difficult to apply and can easily cause network collapse, leading to a large number of data being mapped to the same code. In terms of computing resources, PQN [45] and DTQ [27] are computationally expensive due to the construction of triples, despite DTQ proposal to reduce the amount of calculation by dividing the dataset into multiple groups. DQN [47] requires frequent clustering of the entire training data to update the hash code center. Recent works attempt to apply deep hashing to large-scale datasets with thousands of classes, such as OPQN [49], ADSVQ [51], DCDH [48], etc. However, ADSVQ and DCDH need to calculate a square matrix of the number of categories in the training process, which limits the further expansion of the number of categories in the dataset. The number of bits in OPQN is limited to the feature dimension, resulting in significant training overhead when a longer bit number is needed to ensure retrieval performance. In contrast, our method does not involve large-scale matrix operations, converges quickly, and overcomes the limitations of previous methods for large-scale datasets.
Denoting the input images set as  $\mathcal{I}$  and the deep features set extracted by the backbone as  $F = \{\mathcal{F} \mid \mathcal{F} \in \mathbb{R}^{D}\}$ , Product Quantization divides the feature  $\mathcal{F}$  into  $M$  segments, i.e.,  $\mathcal{F} = [\mathcal{F}_1, \dots, \mathcal{F}_m, \dots, \mathcal{F}_M]$ , where  $\mathcal{F}_m \in \mathbb{R}^{D / M}$ . Within each segment, all sub-vectors  $\{\mathcal{F}_m\}$  are clustered into  $K$  categories using the k-means algorithm. Consequently, each feature can be represented as an  $M$ -tuple:  $PQ(\mathcal{F}) = [k_1, \dots, k_m, \dots, k_M]$ , where each  $k_m$  represents the clustering category(index) of the  $m$ -th sub-vector and ranges from 0 to  $2^b - 1$ . The codebook  $\mathcal{C} \in \mathbb{R}^{M \times K \times (D / M)}$  consists of the clustering centers in each of the  $M$  segments, which are used to reconstruct the original features from the  $M$ -tuple representation  $PQ(\mathcal{F})$ :
$$
\operatorname {q u a n t} (\mathcal {F}) = \left[ c _ {k _ {1}} ^ {1}, \dots , c _ {k _ {m}} ^ {m}, \dots , c _ {k _ {M}} ^ {M} \right], \tag {1}
$$
where  $\text{quant}(\mathcal{F})$  is the quantized vector that approximates the original feature  $\mathcal{F}$ , and  $c_{k_m}^m$  denotes the  $k_m$ -th codeword in the  $m$ -th part of the codebook. Finally, the feature is compressed into  $M \times b$  bits.
In retrieval phase, a Lookup Tables (LUTs) will be calculated, which stroed the distances between the query and the codebook elements. Then, the distances between the query and all samples in the gallery set can be efficiently computed by looking up the LUTs.
The original PQ method is designed to minimize quantization error using clustering algorithms. However, when we increase the compression strength to reduce the encoding bits in PQ, the number of segments  $M$  decreases, and the sub-vector dimension increases, which brings problems: 1) The impact of misallocation of segment increases: the quantization error will increase with the sub-vector dimension increases, and also the importance of a single segment will increase with the number of segments decreases. 2) As the number of segments decreases and the sub-vector dimension increases, the sub-vectors become more like complete features, usually with stronger discriminative. Therefore, the sub-vector distribution becomes more uniform. Since we need to assign  $N$  classes to  $K$  clustering centers, where  $N \gg K = 2^b$ , this makes it more likely for features of different classes to be assigned to the same PQ codes, or for the same class to be assigned to different PQ codes. These problems will significantly affect PQ performance.
To mitigate those issues, we train a backbone to generate features that are better suited for a short PQ encoding. Our aim is to assign features of the same class to the same PQ code and cluster the sub-vectors that belong to the same encoding within each sub-space. To achieve these goals, we undertook these two actions: 1) learning a backbone to generate features that better suited for short PQ encoding based on a set of pre-defined class-level PQ code supervision, and 2) using the learned adaptable codebook to guide the PQ encoding process.
Figure 1: The overall flow of our framework. It consists two branches during the training phase: a one-hot classification branch for the entire feature and another PQ classification branch that constrains sub-vectors of the feature. We use the predefined class-level PQ labels to guide the learning of the PQ branch and improve feature separation in sub-spaces. During the retrieval phase, the only difference with traditional PQ is that our framework does not require clustering for the codebook. Instead, we directly use the learned codebook obtained from the PQ branch, which comprises the fully connected weights of multiple sub-branches in the PQ branch.
Fig. 1 illustrates the overall flow of our framework. We incorporate a PQ branch at the end of the backbone and run it in parallel with the classifier. To learn the PQ code targets for each class, we divide the features  $\mathcal{F}$  into  $M$  segments refer to the PQ encoding process. For each PQ sub-branch, we maximize the posterior probability of the ground-truth:
$$
L _ {p q} = - \frac {1}{B M} \sum_ {b = 1} ^ {B} \sum_ {m = 1} ^ {M} \log \frac {e ^ {z _ {m k l}}}{\sum_ {k = 1} ^ {K} e ^ {z _ {m k}}}, \tag {2}
$$
where  $B$  is the batch size,  $M$  is the number of PQ code segments, and  $K$  is the number of clusters in each segment.  $z_{mk}$  is the output of the  $k$ -th unit in the  $m$ -th PQ sub-branch.  $k^l$  is the ground-truth value, which is the code value of a unit in the PQ label.
Optimizing Sub-branches with Angle: Let us consider the Euclidean distance between the sub-vector  $\mathcal{F}_m$  in the  $m$ -th segment and  $K$  classification weights  $\{w_{mk}; k \in [1, K]\}$  belonging to the  $m$ -th PQ sub-branch:
$$
D _ {E u c l i d e a n} \left\langle \mathcal {F} _ {m}, w _ {m k} \right\rangle = \sqrt {\left\| \mathcal {F} _ {m} \right\| ^ {2} + \left\| w _ {m k} \right\| ^ {2} - 2 \left\| \mathcal {F} _ {m} \right\| \left\| w _ {m k} \right\| \cos \theta_ {m k}}, \tag {3}
$$
where  $\theta_{mk}$  is the angle between  $w_{mk}$  and the sub-vector  $\mathcal{F}_m$ . We normalize  $\{w_{mk}\}$  by setting  $\| w_{mk}\| = 1$ , Then, Eq. (3) can be interpreted as:
$$
D _ {E u c l i d e a n} \left\langle \mathcal {F} _ {m}, w _ {m k} \right\rangle = \sqrt {\left\| \mathcal {F} _ {m} \right\| ^ {2} + 1 - 2 \left\| \mathcal {F} _ {m} \right\| \cos \theta_ {m k}}. \tag {4}
$$
Now, the ranking of the Euclidean distance between the sub-vector  $\mathcal{F}_m$  in the  $m$ -th segment and all the weights  $\{w_{mk}\}$  in the corresponding PQ sub-branch depends solely on the angles  $\{\theta_{mk}; \theta \in [0, \pi]\}$ . This is because the norm of  $\mathcal{F}_m$ , denoted  $\| \mathcal{F}_m \|$ , is invariant to changes in the magnitude of the weights  $\{w_{mk}\}$ . Exploiting this property, we can optimize M softmax sub-branches based on the angles. To further stabilize the training process and accelerate model convergence, we omit the bias term in the FC layer. Additionally, we normalize  $\mathcal{F}_m$  and set  $\| \mathcal{F}_m \| = 1$ . This gets:
$$
z _ {m k} = \left\| \mathcal {F} _ {m} \right\| \left\| w _ {m k} \right\| \cos \theta_ {m k} = \cos \theta_ {m k}. \tag {5}
$$
Retrieval tasks typically demand a higher degree of discriminability than classification tasks. A simple softmax cross-entropy loss may fall short of achieving the desired results. To enhance the discriminability of each sub-branch, a margin can be added to the loss. Fortunately, we can utilize existing metric learning losses [39, 9, 29] such as CosFace [39] to successfully achieve our objectives:
$$
L _ {p q} = - \frac {1}{B M} \sum_ {b = 1} ^ {B} \sum_ {m = 1} ^ {M} \log \frac {e ^ {s \left(\cos \left(\theta_ {m k l} + m a r g i n\right)\right)}}{e ^ {s \left(\cos \left(\theta_ {m k l} + m a r g i n\right)\right)} + \sum_ {k = 1 , k \neq k ^ {l}} ^ {K} e ^ {s \cos \left(\theta_ {m k}\right)}}, \tag {6}
$$
where  $\text{margin} \geq 0$  is a fixed parameter to control the magnitude of the cosine margin, and  $s$  is a scaling factor.
Apart from the PQ branch, the classification branch is also utilized to enhance the discriminability of the full features. The final optimization objective is defined as:
$$
L = L _ {c l s} + L _ {p q}, \tag {7}
$$
where  $L_{cls}$  denotes any classification loss used to learn the backbone, and we can achieve the optimization objective in an end-to-end manner.
4.2 Predefined Class-Level PQ Label
Earlier works have utilized predefined hash codes as labels for guiding model learning, such as DPN [12], CSQ [46], and OrthoHash [18]. They constructed hash codes via Hadamard matrix [41] or Bernoulli sampling to uniformly distribute the hash codes in the target space and maximize the distance between the hash codes. However, as the number of data categories increases, the large dimensions of Hadamard matrix make it impractical to use, and achieving near-uniformity of compressed hash codes based on Bernoulli sampling as a supervised target can be difficult to converge.
In our approach, we get the class average featurew  $\{\mathcal{F}_{avg}\}$  for each class by averaging the output features of a pre-trained model. Here, we assume that the classifier of the pre-trained model has been discarded and is not usable, as this is a common scenario, particularly in retrieval tasks. We then execute the PQ encoding process on  $\{\mathcal{F}_{avg}\}$  by performing k-means [31] clustering in each sub-space to generate the corresponding class-level PQ encoding. These PQ codes implicitly capture inter-class relationships:
$$
\begin{array}{l} P Q _ {l a b e l} = P Q \left(\left\{\mathcal {F} _ {a v g} \right\}\right) \\ = \left\{\left[ k _ {1}, \dots , k _ {m}, \dots , k _ {M} \right] _ {\text {c l s}}; \text {c l s} \in [ 1, \text {c l a s s} _ {-} n u m ] \right\}. \tag {8} \\ \end{array}
$$
In situations with a lot of data categories and low encoding bits, there is a risk of PQ code duplication. To remove duplication and ensure meaningful inter-class relationships, we remove such duplicates by assigning them to the position with the lowest quantization error, provided that the position is not already occupied by another PQ code.
4.3 Encode and Retrieval
After completing the training phase of our framework, we utilize the backbone to extract the feature representation of the input images  $\mathcal{I}$ . The PQ branch weights, which consist of  $M$  FC layers, are used as the codebook in the PQ algorithm to obtain the PQ code of the gallery set and calculate the quantized distance. Specifically, we extract the weights of the PQ branch and perform normalization on them as follows:  $\widetilde{w}_{mk} \gets w_{mk} / \| w_{mk} \|$ ;  $\widetilde{w}_{mk} \in \mathbb{R}^{D / M}$ . All  $\widetilde{w}_{mk}$  of the  $M \times K$  items in the  $M$  branches are combined to a codebook, denoted as  $\mathcal{C} \in \mathbb{R}^{M \times K \times (D / M)}$ , where  $\mathcal{C}_{mk} = \widetilde{w}_{mk}$ . At this point, the PQ algorithm no longer requires training, and the codebook  $\mathcal{C}$  learned during the training phase will be used to initialize the PQ algorithm. We can encode the samples of the gallery set as  $PQ(\mathcal{F}_{\mathcal{G}}) = \{[k_1, \dots, k_m, \dots, k_M]_i; i \in [1, num\_images]\}$ .
There are two methods of utilizing PQ for retrieval: symmetric retrieval and asymmetric retrieval. When apply symmetric retrieval, the distance between samples in the query set and the gallery set is calculated as follows:
$$
\mathcal {D} _ {s y m} = \sum_ {m = 1} ^ {M} \left\langle \mathcal {C} _ {m k ^ {*}}, \mathcal {C} _ {m k ^ {i}} \right\rangle , \tag {9}
$$
where  $k^*$  is the index of the closest codeword to sub-vector  $\mathcal{F}_m$  among the  $K$  codewords in the  $m$ -th part of the codebook, and  $\mathcal{C}_{mk^*}$  and  $\mathcal{C}_{mk^*}$  represent the quantized representation of the  $m$ -th segment of the query sample and the gallery sample, respectively. When apply asymmetric retrieval, the distance between samples in the query set and gallery set is calculated as follows:
$$
\mathcal {D} _ {\text {a s y m}} = \sum_ {m = 1} ^ {M} \left\langle \mathcal {F} _ {m} ^ {\text {q u e r y}}, \mathcal {C} _ {m k ^ {i}} \right\rangle , \tag {10}
$$
where  $\mathcal{F}_m^{query}$  is the sub-vector of the  $m$ -th segment of the query sample. Considering the discussion about Eq. (4), we have:
$$
\begin{array}{l} k^{*} = \operatorname *{arg  min}_{k\in [0,K - 1]}\big\langle \mathcal{F}_{m},\mathcal{C}_{mk}\big\rangle  \\ = \underset {k \in [ 0, K - 1 ]} {\arg \min } \left\langle \widetilde {\mathcal {F}} _ {m}, \mathcal {C} _ {m k} \right\rangle , \tag {11} \\ \end{array}
$$
where  $\widetilde{\mathcal{F}}_m$  is the normalized vector of  $\mathcal{F}_m$  and  $\langle \bullet \rangle$  can be either the cosine distance or the Euclidean distance. Referring to Eq. (11), for symmetric retrieval, no additional feature operations are necessary for encoding, even though we perform segment normalization on the features during training phase. When apply asymmetric retrieval,  $\sum_{m = 1}^{M}\left\langle \mathcal{F}_{m}^{\text{query}},\mathcal{C}_{mk^{i}}\right\rangle \neq \sum_{m = 1}^{M}\left\langle \widetilde{\mathcal{F}}_{m}^{\text{query}},\mathcal{C}_{mk^{i}}\right\rangle$ , the retrieved results may be different. However, as  $\mathcal{C}_{mk^i}$  is a certain value and the classification loss with margin of PQ branch increases the compactness of the encoding, we found in our experiments that whether applying segment normalization operations for the feature had little impact on the ultimate retrieval performance. The same conclusion holds for the normalization of full features, despite the intuition that normalizing the complete features can enhance the stability of asymmetric retrieval results by fixing the magnitude of  $\| \mathcal{F}_m\|$ . Overall, whether applying symmetric or asymmetric retrieval, we do not need to perform any additional processing on the features compared with PQ algorithm.
5.1 Datasets and Evaluation
Although our primary focus is to evaluate the performance of the proposed hash method on large-scale datasets, we will validate our approach on datasets of various scales, including a large-scale datasets Glint360K, as well as two relatively smaller scale datasets, ImageNet100 and ImageNet1K.
Glint360K [2]: Glint360K is one of the largest publicly available face datasets, with over 360K IDs and 17 million images. We use all available data to train hash methods and test for convergence by evaluating the unseen retrieval performance on two other datasets, MegaFace [22] and FaceScrub [33], which are non-intersecting with Glint360K.
MageFace and FaceScrub: MegaFace is a dataset of one million images that capture over 690K different individuals and will be used as the gallery set. FaceScrub comprises over 100K photos of 530 celebrities. Following [22], the evaluation was performed on a subset of FaceScrub that includes 80 individuals (40 females and 40 males) from individuals with more than 50 images each. We followed the recommendations from insightface project<sup>1</sup> to remove noise and duplicate samples. The final query set consists of 3529 face images from 80 different individuals.