SlowGuess/NeurIPS_2023_OCR huggingface dataset:
recognition on PA-100K [47], RAP [39] and PETA [18]. For person ReID, we use the implementation of [50] and report mAP. For 2D pose estimation, the evaluation metrics are PCKh for MPII and AP for COCO and AIC. The codebase is based on [78]. Rank-1 is reported for text-to-image person ReID with the implementation of [59]. mA is reported in the pedestrian attribute recognition task with the codebase of [36]. The evaluation metrics are MPJPE/PA-MPJPE/MPVPE for 3D pose and shape estimation. Since there is no implementation with a plain ViT structure available for this task, we modify the implementation based on [15]. More training details can be found in Appendix.
Table 2: HAP with MAE [30] and CAE [9]. The evaluation metrics are the same as those in Table 1.
MSMT17	Market-1501	PA-100K	MPII	COCO	RSTPReid
256	384	256	384
MAE [30]	76.4	76.8	91.7	91.9	86.54	91.8	75.9	49.35
CAE [9]	76.5	77.0	93.3	93.1	86.33	91.8	75.3	51.70
We compare HAP with both previous task-specific methods and human-centric pre-training methods. Our HAP is simple by using only one dataset of  $\sim 2.1\mathrm{M}$  samples, compared with existing pre-training methods (Table1a), yet achieves superior performance on various human-centric perception tasks.
In Table 1, we report the results of HAP with MAE [30] on 12 datasets across 5 representative human-centric tasks. Compared with previous task-specific methods and human-centric pre-training methods, HAP achieves state-of-the-art results on 11 datasets of 4 tasks. We discuss each task next.
Pedestrian attribute recognition. The goal of pedestrian attribute recognition is to assign fine-grained attributes to pedestrians, such as young or old, long or short hair, etc. Table 1b shows that HAP surpasses previous state-of-the-art on all of the three commonly used datasets, i.e.,  $+0.17\%$  on PA-100K [47],  $+0.57\%$  on RAP [39], and  $+0.36\%$  on PETA [18]. It can be observed that our proposed HAP effectively accomplishes this fine-grained human characteristic understanding tasks.
Person ReID. The aim of person ReID is to retrieve a queried person across different cameras. Table 1c shows that HAP outperforms current state-of-the-art results of MSMT17 [74] by  $+5.0\%$  (HAP  $78.0\%$  vs. MALE [50]  $73.0\%$ ) and  $+1.0\%$  (HAP  $78.1\%$  vs. SOLIDER [8]  $77.1\%$ ) under the input resolution of  $256 \times 128$  and  $384 \times 128$ , respectively. HAP also achieves competitive results on Market-1501 [87], i.e.,  $93.9\%$  mAP with  $384 \times 128$  input size. For a fair comparison, we also report the results of HAP with extra convolution modules following [50], denoted as  $\mathrm{HAP}^{\ddagger}$ . It brings further improvement by  $+1.3\%$  and  $+2.0\%$  on MSMT17 and Market-1501 with resolution of 384. Review that previous ReID pre-training methods [81, 23, 90, 51] usually adopt contrastive learning approach. These results show that our HAP with MIM is more effective on the person ReID task.
Text-to-image person ReID. This task uses textual descriptions to search person images of interest. We use the pre-trained model of HAP as the initialization of the vision encoder in this task. Table1e shows that HAP performs the best on the three popular text-to-image benchmarks. That is, HAP achieves  $68.05\%$ ,  $61.80\%$  and  $49.35\%$  Rank-1 on CUHK-PEDES [42], ICFG-PEDES [20] and RSTPReid [89], respectively. Moreover, HAP is largely superior than MAE [30], e.g.,  $68.05\%$  vs.  $60.19\%$ $(+7.86\%)$  on CUHK-PEDES. It clearly reveals that human structure priors especially human keypoints and structure-invariant alignment indeed provide semantically rich human information, performing well for human identification and generalizing across different domains and modalities.
2D pose estimation. This task requires to localize anatomical keypoints of persons by understanding their structure and movement. For fair comparisons, we also use multiple datasets as in [78] and larger image size as in [82, 8] during model training. Table 1d shows that HAP outperforms previous state-of-the-art results by  $+0.3\%$ ,  $+1.0\%$ ,  $+3.1\%$  on MPII [1], COCO [46], AIC [75], respectively. Meanwhile, we find both multi-dataset training and large resolution are beneficial for our HAP.
3D pose and shape estimation. We then lift HAP to perform 3D pose and shape estimation by recovering human mesh. Considering that there is no existing method that performs this task on a plain ViT backbone, we thus follow the implementation of [15] and just replace the backbone to a plain ViT. The results are listed in Table 1f. It is interesting to find that despite this simple implementation, our HAP achieves competitive results for the 3D pose task even HAP does not see any 3D information. These observations may suggest that HAP improves representation learning ability on human structure from the 2D keypoints information, and transfers its ability from 2D to 3D.
CAE [9] vs MAE [30]. HAP is model-agnostic that can be integrated into different kinds of MIM methods. We also apply HAP to the CAE framework [9] for human-centric pre-training. Table 2 shows that HAP with CAE [9] achieves competitive performance on various human-centric perception tasks, especially on discriminative tasks such as person ReID that HAP with CAE is superior than HAP with MAE [30] by  $1.6\%$  on Market-1501 (with the input resolution of  $256 \times 128$ ), and text-to-image person ReID that HAP with CAE performs better than HAP with MAE by  $2.35\%$  on RSTPReid. These observations demonstrate the versatility and scalability of our proposed HAP framework.
Table 3: Main properties of HAP. "HSP", "HPM", and "SIA" stand for human structure priors, human part prior for mask sampling, and structure-invariant alignment, respectively. We compare HAP with the baseline of MAE ImageNet pretrained model on the representative benchmarks.
method	HSP	HPM	SIA	MSMT17	MPII	CUHK-PEDES	3DPW ↓	PA-100K
baseline	68.9	90.3	66.49	57.4	83.41
HAP-0	✓	74.1	90.9	67.69	56.2	85.50
HAP-1	✓	✓	75.1	91.1	67.95	56.2	85.83
HAP-2	✓	✓	75.6	91.6	68.00	56.1	86.38
HAP	✓	✓	✓	76.4	91.8	68.05	56.0	86.54
MAE [30]	62.0	89.6	60.19	58.0	79.56
Table 4: Ablation on (left) number of samples and (right) training epochs for pre-training of HAP.
samples	MSMT17	MPII	epochs	MSMT17	MPII
~0.5M	66.9	90.4	100	72.2	91.3
~1.0M	71.9	91.2	200	73.9	91.6
~2.1M	76.4	91.8	400	76.4	91.8
4.3 Main properties
We ablate our HAP in Table 3. The analyses of main properties are then discussed in the following. Pre-training on human-centric data. Our baseline, using default settings of [30] and LUPerson [23] as pre-training data, has achieved great improvement than original MAE [30], e.g.,  $+6.9\%$  mAP on MSMT17 and  $+6.30\%$  Rank-1 on CUHK-PEDES. It shows that pre-training on human-centric data benefits a lot for human-centric perception by effectively bridging the domain gap between the pre-training dataset and downstream datasets. This observation is in line with previous works [23, 51].
Human structure priors. We denote HAP-0 to represent human structure priors, i.e., large scaling ratio and block-wise masking with intermediate masking ratio, introduced in Section 3.1. HAP-0 performs better than the baseline method, especially on MSMT17 with  $+5.2\%$  improvement. It clearly reveals that human structure priors can help the model learn more human body information.
Human part prior. We further use human part prior to guide the mask sampling process. Compared with HAP-1 and HAP-0, part-guided mask sampling strategy brings large improvement, e.g.,  $+1.0\%$  on MSMT17 and  $+0.33\%$  on PA-100K. It clearly shows that the proposed mask sampling strategy with the guidance of human part prior has positive impact on the human-centric perception tasks.
Structure-invariant alignment. The structure-invariant alignment loss is applied on different masked views to align them together. Compared with HAP-2 vs. HAP-0 and HAP vs. HAP-1, this alignment loss results in  $+1.5\%$  and  $+1.3\%$  improvement on MSMT17, and  $+0.7\%$  and  $+0.7\%$  improvement on MPII, respectively. It reflects that the model successfully captures human characteristics with this loss. We provide more discussions about the alignment choices as well as the weight in Appendix.
Overall, our HAP greatly outperforms the original MAE [30] and the baseline by large margins across various human-centric perception tasks, showing significant benefits of the designs of our HAP.
Number of samples. We compare using different number of data samples during pre-training in Table 4 (left). Only using  $\sim 0.5\mathrm{M}$  LUPerson data already outperforms original MAE [30] pre-trained on ImageNet [17]. Meanwhile, more pre-training data brings larger improvement for our HAP.
Training epochs. Table 4 (right) shows that pre-training only 100 epochs already yields satisfactory performance. It suggests that HAP can effectively reduces training cost. These observations indicate that HAP, as a human-centric pre-training method, effectively reduces annotation, data, and training cost. Therefore, we believe HAP can be useful for real-world human-centric perception applications.
Visualization. To explore the structure information learned in the plain ViT encoder of HAP, we probe its attention map by querying the patch of a random body keypoint. See visualizations in Figure 6. Interestingly, we observe that HAP captures the relation among body parts. For example,
Figure 6: Visualization on attention maps of body structure from the ViT encoder in HAP. For each pair, we show (left) a randomly selected keypoint within a patch marked in red and (right) its corresponding attention weight to other patches learned by HAP. Patches in deeper color indicate more relevant to the selected keypoint region. The left two pairs are from LUPerson [23], the middle two are from COCO [46], and the right two are from AIC [75]. It reveals that HAP is able to learn human body structure information and successfully identifies the correlations among the body parts.
when querying the right hip of the player in the first image pair (from LUPerson), we see that there are strong relations to the left leg, left elbow, and head. And when querying the left knee of the skateboarder in the second image pair (from COCO), it is associated with almost all other keypoints. These observations intuitively illustrate that HAP learns human body structure and generalizes well.
This paper makes the first attempt to use Masked Image Modeling (MIM) approach for human-centric pre-training, and presents a novel method named HAP. Upon revisiting several training strategies, HAP reveals that human structure priors have great benefits. We then introduces human part prior as the guidance of mask sampling and a structure-invariant alignment loss to align two random masked views together. Experiments show that our HAP achieves competitive performance on various human-centric perception tasks. Our HAP framework is simple by unifying the learning of two modalities within a single dataset. HAP is also versatility and scalability by being applied to different MIM structures, modalities, domains, and tasks. We hope our work can be a strong baseline with generative pre-training method for the future researches on this topic. We also hope our experience in this paper can be useful for the general human-centric perception and push this frontier.
Broader impacts. The proposed method is pre-trained and evaluated on several person datasets, thus could reflect biases in these datasets, including those with negative impacts. When using the proposed method and the generated images in this paper, one should pay careful attention to dataset selection and pre-processing to ensure a diverse and representative sample. To avoid unfair or discriminatory outcomes, please analyze the model's behavior with respect to different demographics and consider fairness-aware loss functions or post-hoc bias mitigation techniques. We strongly advocate for responsible use, and encourage the development of useful tools to detect and mitigate the potential misuse of our proposed method and other masked image modeling techniques for unethical purposes.
Limitations. We use human part prior to guide pre-training, which is extracted by existing pose estimation methods. Therefore, the accuracy of these methods could affect the pre-training quality. Besides, due to the limitations in patch size, the masking of body parts may not be precise enough. If there are better ways to improve this, it should further enhance the results. Nonetheless, the proposed HAP method still shows promising performance for a wide range of human-centric perception tasks.
We thank Shixiang Tang and Weihua Chen for discussions about the experiments of human-centric pre-training, and Yufei Xu and Yanzuo Lu for discussions about downstream tasks. This work was supported in part by Zhejiang Provincial Natural Science Foundation of China (LZ22F020012), National Natural Science Foundation of China (62376243, 62037001, U20A20387), Young Elite Scientists Sponsorship Program by CAST (2021QNRC001), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010), Project by Shanghai AI Laboratory (P22KS00111), Program of Zhejiang Province Science and Technology (2022C01044), the Fundamental Research Funds for the Central Universities (226-2022-00142, 226-2022-00051).
[78] Y. Xu, J. Zhang, Q. Zhang, and D. Tao. Vitpose: Simple vision transformer baselines for human pose estimation. arXiv preprint arXiv:2204.12484, 2022.
[79] S. Yang, Z. Quan, M. Nie, and W. Yang. Transpose: Keypoint localization via transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11802-11812, 2021.
[80] Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transactions on pattern analysis and machine intelligence, 35(12):2878-2890, 2012.
[81] Z. Yang, X. Jin, K. Zheng, and F. Zhao. Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14298–14307, 2022.
[82] Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint arXiv:2110.09408, 2021.
[83] S. Zhang, F. Zhu, R. Zhao, and J. Yan. Contextual image masking modeling via synergized contrasting without view augmentation for faster and better visual pretraining. In The Eleventh International Conference on Learning Representations, 2023.
[84] X. Zhang, J. Cao, C. Shen, and M. You. Self-training with progressive augmentation for unsupervised cross-domain person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8222–8231, 2019.
[85] X. Zhang, J. Chen, J. Yuan, Q. Chen, J. Wang, X. Wang, S. Han, X. Chen, J. Pi, K. Yao, J. Han, E. Ding, and J. Wang. CAE v2: Context autoencoder with CLIP latent alignment. Transactions on Machine Learning Research, 2023.
[86] X. Zhang, D. Li, Z. Wang, J. Wang, E. Ding, J. Q. Shi, Z. Zhang, and J. Wang. Implicit sample extension for unsupervised person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7369–7378, 2022.
[87] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pages 1116–1124, 2015.
[88] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
[89] A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua. Dssl: deep surroundings-person separation learning for text-based person retrieval. In Proceedings of the 29th ACM International Conference on Multimedia, pages 209–217, 2021.
[90] K. Zhu, H. Guo, T. Yan, Y. Zhu, J. Wang, and M. Tang. Pass: Part-aware self-supervised pre-training for person re-identification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIV, pages 198–214. Springer, 2022.
[91] K. Zhu, H. Guo, S. Zhang, Y. Wang, G. Huang, H. Qiao, J. Liu, J. Wang, and M. Tang. Aaformer: Auto-aligned transformer for person re-identification. arXiv preprint arXiv:2104.00921, 2021.
A Details of pre-training
Model structure. We adopt MAE-base [30] as the basic structure of HAP. It consists of an encoder and a lightweight decoder. The encoder follows the standard ViT-base [21] architecture, comprising 12 common Transformer blocks [66] with 12 attention heads and a hidden size of 768. The decoder is relatively small, which consists of 8 Transformer blocks with 16 attention heads and a hidden size of 512. Sine-cosine position embeddings are added to representations of both encoder and decoder<sup>3</sup>. We also use CAE-base [9] as the basic structure. Please refer to [9] for more details of the structure.
Training data. LUPerson [23] is a large-scale person dataset with  $\sim 4.2\mathrm{M}$  images (see Figure 8a). Following [51], we use a subset of LUPerson with  $\sim 2.1\mathrm{M}$  images for pre-training. The resolution of the input images is set to  $256\times 128$  following previous works [23, 90, 51, 50]. The data augmentation follows [50], which involves random resizing and cropping, and horizontal flipping. We utilize ViTPose [78], a popular 2D pose estimation method, to extract the keypoints of LUPerson as the human structure prior in our HAP. Given that our HAP does not rely on ground-truth labels, it can be pre-trained on a combination of several large-scale person datasets after extracting their keypoints. We speculate that employing multi-dataset co-training can further improve the performance of HAP, as observed in [78]. We also recognize that our HAP would derive great benefits from more accurate keypoints (e.g., human-annotated keypoints) if available. We will discuss this point in details later.
Table 5: Hyper-parameters of pre-training. HAP follows most of the settings of MAE and CAE.
dataset	method	batch size	epochs	warmup epochs	base learning rate	optimizer	weight decay
LUPerson	MAE [30]	4096	400	40	1.5e-4	AdamW	0.05
LUPerson	CAE [30]	2048	400	10	1.5e-4	AdamW	0.05
Implementations details. We follow most of the training settings of MAE/CAE except using training epochs of 400 (Table 5). Different from MAE/CAE, HAP has a large scaling ratio of  $80\%$ , and an intermediate masking ratio of  $50\%$  with block-wise masking strategy, as introduced in the main paper.
B Details of human-centric perception tasks
We list the training details of human-centric perception tasks in Table 1 of the main paper as follows.
B.1 Person ReID
Table 6: Hyper-parameters of person ReID.
dataset	batch size	epochs	learning rate	optimizer	weight decay	layer decay	drop path
MSMT17	64	100	8e-3	AdamW	0.05	0.4	0.1
Market-1501	64	100	8e-3	AdamW	0.05	0.4	0.1
We consider two ReID datasets: MSMT17 [74] (Figure 8b) and Market-1501 [87] (Figure 8c) with the codebase  $^4$  of [50]. The input image resolutions are set to  $256\times 128$  and  $384\times 128$ . Data augmentations are the same as [50], including resizing, random flipping, padding, and random cropping. Following [32, 64, 8, 16], the model is fine-tuned using a cross-entropy loss and a triplet loss with equal weights of 0.5. We also consider using a convolution-based module [50] to further improve results. The evaluation metric of mAP (\%) is reported. See hyper-parameters in Table 6.
B.2 2D human pose estimation.
HAP is evaluated on MPII [1] (Figure 8d), MS COCO [46] (Figure 8e), and AI Challenger (AIC) [75] (Figure 8f). We use the codebase  $^5$  of [78]. Resolutions of  $256\times 192$  and  $384\times 288$  are employed.
In this paper, we conduct all experiments on base-sized model to verify the effectiveness of our method. In fact, all scales of models are compatible with HAP.
$^{4}$ https://github.com/YanzuoLu/MALE
$^{5}$ https://github.com/ViTAE-Transformer/ViTPose
Table 7: Hyper-parameters of 2D pose estimation.  ${}^{ \dagger  }/{}^{ + }$  : multi-dataset training  $/{384} \times  {288}$  resolution.
dataset	batch size	epochs	learning rate	optimizer	weight decay	layer decay	drop path
MPII/MIPI+	512	210	1.0e-4	Adam	-	-	0.30
COCO/COCO+	512	210	2.0e-4	AdamW	0.1	0.80	0.29
AIC/AIC+	512	210	5.0e-4	Adam	-	-	0.30
MPII†/COCO†/AIC†	1024	210	1.1e-3	AdamW	0.1	0.77	0.30
MPII†+/COCO†+/AIC†+	512	210	5.0e-4	AdamW	0.1	0.75	0.30
Data augmentations are random horizontal flipping, half body transformation, and random scaling and rotation. We follow the common top-down setting, i.e., estimating keypoints of the instances detected [76]. We use mean square error (MSE) [78] as the loss function to minimize the difference between predicted and ground-truth heatmaps. Following [78], we also perform multi-dataset training setting by co-training on MPII, MSCOCO, and AIC. Evaluation metric of PCKH (\%) is reported for MPII, and AP (\%) is reported for MS COCO and AIC. See the details of hyper-parameters in Table 7.
B.3 Text-to-image person ReID
Table 8: Hyper-parameters of text-to-image person ReID.
dataset	batch size	epochs	learning rate	optimizer
CUHK-PEDES	64	60	1e-3	Adam
ICFG-PEDES	64	60	1e-3	Adam
RSTPReid	64	60	1e-3	Adam
We employ three commonly used datasets, i.e., CUHK-PEDES [42] (Figure 8g), ICFG-PEDES [20] (Figure 8h), RSTPReid [89] (Figure 8i). The codebase  $^6$  of [59] is used. The resolution is  $384\times 128$ . We adopt resizing and random horizontal flipping as data augmentations. BERT [19] is used to extract embeddings of texts, which are then fed to Bi-LSTM [27]. Feature dimensions of images and texts are both set to 384. The BERT is frozen while Bi-LSTM and ViT are fine-tuned. We report Rank-1  $(\%)$  metric for the results on all datasets. The details of the hyper-parameters can be found in Table 8.
B.4 3D pose and shape estimation
Table 9: Hyper-parameters of 3D pose and shape estimation.
dataset	batch size	epochs	learning rate	optimizer
3DPW	192	11	1e-4	SGD
We perform 3D pose and shape estimation on 3DPW [67] (Figure 8j) using the codebase  $^7$  of [15]. The resolution is  $256\times 256$ . Data augmentations follow [15]. We minimize a pose loss, i.e.,  $l_{1}$  distance between the predicted 3D joint coordinates and the pseudo ground-truth, and a mesh loss, i.e., loss for predicted SMPL parameters [53, 55]. We report MPJPE, PA-MPJPE, MPVPE. The former two are for 3D pose, while the latter one is for 3D shape. Hyper-parameters are in Table 9.
B.5 Pedestrian attribute recognition
We evaluate HAP on three popular datasets, i.e., PA-100K [47] (Figure 8k), RAP [39] (Figure 8l), and PETA [18] (Figure 8m). We implement it using the codebase  $^8$  of [36]. The resolution is  $256\times 192$ . Data augmentations consist of resizing and random horizontal flipping. The binary cross-entropy loss is employed, and the evaluation metric of mA  $(\%)$  is reported. Find the hyper-parameters in Table 10.
$^{6}$ https://github.com/GalaxyReID-test1
$^{7}$ https://github.com/hongsukchoi/3DCrowdNet_RELEASE
<sup>8</sup>https://github.com/valencebond/Rethinking_of_PAR
Table 10: Hyper-parameters of pedestrian attribute recognition.
dataset	batch size	epochs	learning rate	schedule	optimizer	weight decay
PA-100K	64	55	1.7e-4	cosine	AdamW	5e-4
RAP	64	8	2.4e-4	cosine	AdamW	5e-4
PETA	64	50	1.9e-4	cosine	AdamW	5e-4
C Study of human part prior
Figure 7: Part masking with keypoints. We divide human body structure into 6 parts following [60]. For each input image, we randomly select 0 to 6 parts (following a uniform distribution) and mask patches located on them (according to the sequence of human part selection). For each selected part, we mask the patches within the bounding boxes of the corresponding keypoint pairs, i.e., head: { (nose, left eye), (nose, right eye), (left eye, right eye), (left eye, left ear), (right eye, right ear)}; upper body: { (left shoulder, right hip), (right shoulder, left hip)}; left arm: { (left shoulder, left elbow), (left elbow, left wrist)}; right arm: { (right shoulder, right elbow), (right elbow, right wrist)}; left leg: { (left hip, left knee), (left knee, left ankle)}; right leg: { (right hip, right knee), (right knee, right ankle)}