SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ImageNet1K [35] and ImageNet100: ImageNet1K is a widely-used large-scale visual recognition challenge dataset, which contains 1000 classes and over 1.2 million images. We employed the training set of ImageNet1K to train the hash methods and used the validation set, which consists of 50,000 images, to evaluate hash retrieval performance. Specifically, we selected the top 5 images from each class in the validation set, totalling 5000 images, to construct the query set, while the remaining 45,000 images comprised the gallery set. ImageNet100 has been popularly used in previous deep hashing methods [46, 18, 7], and it is the subset of ImageNet1K, consisting of 100 classes.
Evaluation: In the following experiments, we will use asymmetric retrieval and our method will not perform any feature preprocessing during retrieval. We will assess the retrieval performance using Top-1, Top-5, and Top-20 accuracies on the Glint360K datasets. Specifically, the query set consists of N individuals, and for each individual, we have M images. We will test each image by incorporating it into the gallery set and employing each of the other M-1 images as the query set. Regarding ImageNet1K and ImageNet100, we will assess the retrieval performance by using the mean average precision(mAP) of the top 1000 result: mAP@1000.
All experiments were conducted using the PyTorch framework and 8 NVIDIA 2080Ti or 3090Ti GPUs for training. We employed a pre-trained model to get features so that we could acquire
$^{1}$ https://github.com/deepinsight/insightface
Table 1: On the large-scale dataset Glint360K, the performance comparison of multiple methods under multiple code lengths, it should be noted that some methods were not listed due to ineffective performance or requiring unrealistic computational resources.
Glint360K iResnet50
Method	32 bits	64 bits	128 bits
Top-1	Top-5	Top-20	Top-1	Top-5	Top-20	Top-1	Top-5	Top-20
PQ [20]	0.1724	0.2999	0.4254	0.6665	0.7749	0.8404	0.9184	0.9492	0.9610
HHF [42]	0.0001	0.0014	0.0054	0.0773	0.1646	0.2661	0	0	0
CSQ [46]	0.0061	0.0113	0.0183	0.0124	0.0197	0.0269	0.0283	0.0372	0.0451
OrthoHash [18]	0.3098	0.3706	0.4171	0.5526	0.6007	0.6351	0.6626	0.7104	0.7454
GreedyHash [37]	0.3688	0.5220	0.6323	0.6102	0.7452	0.8251	0.8259	0.9003	0.9338
FPPQ(Ours)	0.9343	0.9601	0.9652	0.9467	0.9571	0.9628	0.9568	0.9690	0.9731
class-level PQ labels. It is worth noting that this operation is fair since other comparative methods also use this pre-trained model to initialize their model parameters. In all experiments, we set the hyperparameters  $s$  to 64 and margin to 0.2. Moreover, we set 8 bits per segment, which is the commonly used setting in practical applications. This implies that the number of categories  $K$  per sub-branch was set to 256, while the number of segments varied depending on the target bits. For example, the bits [16-bits, 32-bits, 64-bits, 128-bits] corresponded to the segments [2, 4, 8, 16]. We compared various hashing methods [20, 7, 46, 12, 25, 48, 36, 37, 18, 23, 42, 43], and we implemented these methods based on the cisip-FIRE project<sup>2</sup> and integrated some public projects provided by the specific author. For a fair comparison, we use the default settings in the cisip-FIRE project or used the default hyperparameters specified in their papers, and we trained all methods with the same backbone, same epochs, and same batch sizes, among others. In general, unless otherwise specified, these methods we compare are trained using fine-tuning based on the pre-trained model, and the hash layer had an initial learning rate of 0.0001 that decreased to 0.00001 after training 0.8 times number of epochs. The backbone learning rate was always 0.1 times that of the hash layer. In the following experiments, we will provide more specific explanations regarding implementation details.
5.3 Results on Large-Scale Dataset Glint360K
Implementation Details Supplement: We employed iResnet [11] as the backbone and adopted the default training settings from the insightface project. The dimension of features output by the backbone was set to 512. We employed SGD as the optimizer with a weight decay of 0.0001. The batch size was set to  $128 \times 8$ , and the model was trained for a total of 20 epochs. The initial learning rate was set to 0.1 and linearly decreased it to 0 based on the number of training steps.
Comparison: We explored and tested various advanced methods, including PQ [20], HHF [42], CSQ [46], OrthoHash [18], GreedyHash [37], JMLH [36], DPQ [23], DCDH [48], DWDM [10], etc. These methods can be considered as encompassing the current most competitive hashing methods. However, DCDH and DWDM were abandoned due to exceeding computational resource limitations with large-scale matrix multiplication, JMLH and DPQ did not achieve effective performance. The further restricted methods are explained in Sec. 2. It should be noted that OrthoHash also applies a similar angular loss function for model optimization. To enhance its feasibility on large-scale data, we set its hyperparameters  $s$  to 64 and margin to 0.2, the reason please refer to [39]. For methods that used predefined hash centers [46, 18], we used Bernoulli sampling and ensure that hash centers were non-redundant. This approach is chosen due to the inefficiency and infeasibility of generating hundreds of thousands, or even millions, of hash centers using the technique proposed in OrthoHash. We only demonstrate methods that achieved a certain level of performance in Tab. 1.
As it is apparent, currently available methods fail to achieve satisfactory performance on large-scale datasets under multiple bit settings. The hyperparameters in HHF change depending on the minimum distance bound tables proposed in [6] for different code lengths, which hinders its efficacy to converge in 128 bits. While OrthoHash and GreedyHash have achieved a certain level of performance, they have exhibited inferiority to the original PQ method in both 64 bits and 128 bits. In contrast, our proposed method has displayed significant performance improvement under multiple bit settings.
$^{2}$ https://github.com/CISiPLab/cisip-FIRE
Table 2: Under different network structure and bits settings, our method achieved consistent and significant improvements.
Glint360K
Backbone	iResnet18	iResnet50	iResnet100
Method	Metric	128 bits	64 bits	32 bits	128 bits	64 bits	32 bits	128 bits	64 bits	32 bits
PQ [20]	Top-1	0.8005	0.4588	0.0973	0.9184	0.6665	0.1724	0.9465	0.7380	0.2074
Top-5	0.8659	0.5958	0.1940	0.9492	0.7749	0.2999	0.9658	0.8378	0.3569
Top-20	0.9025	0.6903	0.3031	0.9610	0.8404	0.4254	0.9720	0.8878	0.4941
FPPQ(Ours)	Top-1	0.8231	0.7019	0.6945	0.9568	0.9467	0.9342	0.9679	0.9578	0.9305
Top-5	0.8824	0.7890	0.7576	0.9690	0.9571	0.9601	0.9746	0.9728	0.9700
Top-20	0.9133	0.8414	0.8006	0.9731	0.9628	0.9652	0.9772	0.9753	0.9734
Figure 2: In Glint360K, comparison of the asymmetric retrieval performance trend of PQ with compression efficiency and the results of our methods for improving quality. 'L2' represents using the original features for L2 retrieval, while  $M = 256$  represents dividing each feature into 256 segments for PQ retrieval, and 8 bits per segment.
(a) PQ
(b) FPPQ
Figure 3: The visualization of the sub-vectors and clustering centers for the 10k classes in the Glint360K dataset of two approaches: (a) classification loss with PQ and (b) our proposed method. The black dots represent sub-features, while the red dots in (a) depict the clustering centers obtained by performing k-means clustering on the sub-features of all classes. In contrast, in (b), the red dots represent the codewords trained by our proposed method.
Specifically, in the 32-bit setting, our method outperformed GreedyHash with a nearly  $57\%$  increase in Top-1 performance. Additionally, we were able to achieve approximately  $28\%$  and nearly  $4\%$  performance growth in the 64 bits and 128 bits settings, respectively.
Performance with Different Backbone Settings: We also tested the effectiveness of our method under multiple network structures, and our method achieved significant advantages in all results, as shown in Tab. 2. Noting that the large-scale datasets provided favorable conditions for PQ clustering, making PQ a strong baseline, as evident in Tab. 1. Therefore, in this comparison, we focused on comparing our method with PQ (additional results can be found in the supplementary material). We can observe that larger backbones typically result in better performance. However, iResnet100 did not consistent improvement in comparison to iResnet50 when encoded to 32 bits (Top-1↓, Top-5↑, Top-20↑). we suspect that the performance limits of the 32 bits setting may be due to the intrinsic constraint of compression strength.
Low-Slope Decay of Retrieval Performance: As the encoding bit-length decreases, the performance of naive product quantization encoding experiences a significant decline. However, after being trained using our proposed method, the performance degradation gradually occurs at a lower slope with decreasing bit-lengths due to our approach's performance improvement at low bit-lengths. This trend is illustrated in Fig. 2.
t-SNE Visualization: To better illustrate the effectiveness of our proposed method, we utilize t-SNE [38] to visualize changes in sub-vectors when the feature are compressed to 32 bits. For visualization purposes, We select the first segment of the average features of the first 10,000 classes in Glint360K, denoted as  $\{\mathcal{F}_1\} \in \mathbb{R}^{10000\times (D / M)}$ , and their corresponding codewords for the first part, represented by  $\mathcal{C}_1\in \mathbb{R}^{K\times (D / M)}$ . As we can see in Fig. 3, the sub-vectors of the original features tend to be
Figure 4: Visualization of the Top-20 retrieval samples using FaceScrub as the query set and MegaFace as the gallery set.
evenly distributed throughout the subspace, hindering sub-vectors compression. In contrast, our method effectively learns class-level PQ labels, leading the sub-vectors to cluster into  $K$  clusters in the subspace.
Retrieval Result Visualization: Moreover, we provide visualizations of the Top-20 retrieval results on the MegaFace and FaceScrub datasets in Fig. 4. This encompasses comparisons between various ways: 1) our method, which includes L2 retrieval and PQ4 retrieval (without feature processing, normalization, or segment normalization), and 2) the original L2 retrieval and naive PQ4 retrieval (with or without feature normalization). It can be observed that our method generates similar top-20 results under different feature preprocessing settings, further validating the insensitivity of our method to feature preprocessing.
5.4 Results on Smaller Scale Datasets
In addition to conducting experiments on large-scale datasets, we evaluated our proposed method on relatively small datasets such as ImageNet1K and ImageNet100. We utilized ResNet50 [17] as the backbone and employed parameters pre-trained on ImageNet1K to initialize the model. The dimension of the features output by the backbone is 2048. Our experiments on these datasets not only demonstrated the effectiveness of our proposed method but also allowed us to make preliminary attempts at addressing two special scenarios.
Results on ImageNet100: The experiment on ImageNet100 highlights a scenario where the data scale is small and has fewer categories compared to the number of clusters required for k-means clustering. As k-means clustering cannot be executed in this scenario, we overcame this limitation by training the PQ using all instance features and subsequently encoding the class average features. To prevent overfitting due to the limited data, we fixed the backbone and added a linear layer of 2048 to 128 at the end of the backbone for learning. We fine-tuned our model for 100 epochs with a learning rate of 0.0001. The results presented in Tab. 3, and the performances of other methods were obtained either by replicating the results using the cisip-FIRE project or citing their respective papers. Our method achieved the best performance on ImageNet100, indicating its efficacy in scenarios involving limited dataset scale.
Results on ImageNet1K: The experiments on ImageNet1K reveal another scenario: the inability to obtain a good class-level PQ label due to limitations in the number of training samples or low feature discrimination. With only 1000 classes in ImageNet1K, training PQ effectively becomes challenging given the limited number of categories available. Moreover, the low discriminability between classes in the features generated by a pre-trained model leads to a significant amount of duplicate PQ codes
Table 3: mAP@R1000 on ImageNet100
Method	ImageNet100 mAP@R1000
16 bits	32 bits	64 bits
HashNet [7]	0.5101	0.7059	0.7997
CSQ [46]	0.8379	0.8750	0.8874
DPN [12]	0.8543	0.8799	0.8927
DFH [25]	0.8352	0.8781	0.8849
DCDH [48]	0.7856	0.8158	0.8400
JMLH [36]	0.8366	0.8671	0.8799
GreedyHash [37]	0.8544	0.8796	0.8886
HBS-RL [43]	0.8465	0.8702	0.8851
DPQ [23]	0.8860	0.8770	0.8660
HHF [42]	0.8710	0.8910	0.8960
OrthoHash [18]	0.8693	0.8869	0.8994
FPPQ(Ours)	0.8956	0.9128	0.9154
after PQ training and encoding. For instance, duplicate rates can exceed  $50\%$  for the PQ codes of 1000 class average features when compressed to 32 bits. To tackle this issue, we adopt the method proposed by OrthoHash, which involves generating hash codes using repeated Bernoulli sampling to achieve binary encodings with sufficient hamming distance. These binary encodings are then segmented and converted into decimal codes to serve as PQ labels. We adopts the default training strategy from the cisip-FIRE project, fine-tuning for 90 epochs. We compared several of the most competitive methods and the experimental results are presented in Tab. 4. As shown in the results, our method achieved competitive results, particularly when the encoding length was relatively short.
Table 4: mAP@R1000 on ImageNet1K
Method	ImageNet1K mAP@R1000
16 bits	32 bits	64 bits
HHF [42]	0.2961	0.5979	0.6121
DCDH [48]	0.3666	0.4764	0.5299
CSQ [46]	0.5040	0.6061	0.6093
JMLH [36]	0.5286	0.5876	0.6098
GreedyHash [37]	0.5424	0.5896	0.5952
OrthoHash [18]	0.5936	0.6514	0.6761
FPPQ(Ours)	0.6207	0.6543	0.6649
This study explored and evaluated several advanced deep hashing methods for large-scale image retrieval, identifying their limitations. In addition, we proposed a product quantization-based framework that employs multiple softmax branches to learn specific class-level PQ labels. Our proposed method effectively performs during the retrieval stage and aligns well with the traditional product quantization methods. Notably, our method is independent of the number of classes and data samples required, requiring only an additional parameter quantity of  $D \times K$  and a relatively small amount of computational resources. Therefore, our proposed method offers promising prospects for large-scale image retrieval tasks.
This paper primarily focuses on hash technology, aiming to enhance data compression and retrieval while extending current deep hash methods to large-scale scenarios. The datasets we utilize are publicly available, and we ensure strict adherence to their respective open licenses.
This work is supported in part by the National Key Research and Development Program of China under Grant No. 2021ZD40303; in part by the Natural Science Foundation of China under Grant No. U20B2052, 61936011; in part by the National Key Research and Development Program of China under Grant No. 2018YFE0118400; in part by the National Natural Science Foundation Innovation Research Group Project under Grant No. 62321003.
[38] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 8
[39] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274, 2018. 1, 4, 7
[40] Xiaofang Wang, Yi Shi, and Kris M Kitani. Deep supervised hashing with triplet labels. In Computer Vision-ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I 13, pages 70-84. Springer, 2017. 2
[41] Eric W. Weisstein. Hadamard matrix. https://mathworld.wolfram.com/, 2002. 5
[42] Chengyin Xu, Zenghao Chai, Zhengzhuo Xu, Hongjia Li, Qiruyi Zuo, Lingyu Yang, and Chun Yuan. Hhf: Hashing-guided hinge function for deep hashing retrieval. IEEE Transactions on Multimedia, 2022. 2, 7, 10
[43] Xin Yao, Min Wang, Wengang Zhou, and Houqiang Li. Hash bit selection with reinforcement learning for image retrieval. IEEE Transactions on Multimedia, 2022. 7, 10
[44] Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems, pages 269-277, 2019. 1
[45] Tan Yu, Junsong Yuan, Chen Fang, and Hailin Jin. Product quantization network for fast image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 186-201, 2018. 2, 3
[46] Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng. Central similarity quantization for efficient image and video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3083-3092, 2020. 2, 5, 6, 7, 10
[47] Cao Yue, M Long, J Wang, Zhu Han, and Q Wen. Deep quantization network for efficient image retrieval. In Proc. 13th AAAI Conf. Artif. Intell, pages 3457-3463, 2016. 2, 3
[48] Ming Zhang, Xuefei Zhe, Shifeng Chen, and Hong Yan. Deep center-based dual-constrained hashing for discriminative face image retrieval. Pattern Recognition, 117:107976, 2021. 1, 2, 3, 7, 10
[49] Ming Zhang, Xuefei Zhe, and Hong Yan. Orthonormal product quantization network for scalable face image retrieval. Pattern Recognition, page 109671, 2023. 1, 2, 3
[50] Ting Zhang, Chao Du, and Jingdong Wang. Composite quantization for approximate nearest neighbor search. In International Conference on Machine Learning, pages 838-846. PMLR, 2014. 2
[51] Chang Zhou, Lai Man Po, and Weifeng Ou. Angular deep supervised vector quantization for image retrieval. IEEE Transactions on Neural Networks and Learning Systems, 33(4):1638-1649, 2020. 2, 3
[52] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10492-10502, 2021. 1