SlowGuess/NeurIPS_2023_OCR huggingface dataset:
left arm
right arm
left leg
right leg
Details of part masking with keypoints. We incorporate human parts, a human structure prior, into pre-training. Figure 7 illustrates the details of how to mask each of the six body parts with keypoints.
Table 11: Guidance in mask sampling. Baseline uses no guidance, i.e., random block-wise masking.
guidance	MSMT17	MPII
attention map	71.7	90.4
keypoint	75.9	91.6
part	76.4	91.8
baseline	75.6	91.6
Masking guidance in mask sampling strategy. We study different guidance, reported in Table 11. Some MIM works [83, 29, 6, 37] use attention map to identify and mask the informative image patches. However, this design performs poorly in human-centric pre-training, achieving even lower performance than baseline (71.7 vs. 75.6 on MSMT17). We conjecture that the underlying reason is the challenge for the model to obtain precise and meaningful attention maps, especially in the initial stage of the pre-training process. That may be why recent works [40, 37, 6] start to employ strong pretrained models to assist mask sampling. In contrast, we directly employ intuitive and explicit human part prior, i.e., human keypoints, to guide mask sampling, which shows great potential. Besides, we observe that only masking patches located on keypoints, instead of human parts, improves little. It indicates that masking a whole human body works better, similar to the finding that block-wise masking performs better than random masking in human-centric pre-training (see the main paper).
Part selection. Here, we ablate different part selection strategies in Table 12. We observe that selecting a fixed number of parts is not beneficial. Compared to a whole range, selecting within a small range has a little improvement. It shows that more part variations brings more randomness, which is good for pre-training. Similar to the findings in recent studies [83, 29, 6, 37] that a deliberate
Table 12: Part selection for masking. Baseline uses no guidance, i.e., random block-wise masking.
part selection	MSMT17	MPII
3	75.4	91.5
6	74.4	91.0
unif{0,3}	76.1	91.7
unif{0,6}	76.4	91.8
baseline	75.6	91.6
design of mask sampling strategy would have a positive impact, our part selection strategy is beneficial for human-centric pre-training, resulting in great performance on a range of downstream tasks.
Using keypoints as target. Since keypoints can also be used as supervision, we perform this baseline by making the decoder predict keypoints instead of pixels. The obtained results are  $50.3\%$  on MSMT17 and  $89.8\%$  on MPII with 100-epoch pre-training, while our HAP is  $72.2\%$  and  $91.3\%$ . The possible reason is that regressing keypoints may lose informative details provided by pixels, which has negative impact on the tasks. In contrast, HAP incorporates prior knowledge of structure into pre-training, which is beneficial to maintain both semantic details and information of body structure.
Using another keypoint detector. To study the robustness of HAP to the extracted keypoints, we make a simple exploration by employing another off-the-shelf keypoint detector named OpenPose [3]. It achieves  $72.0\%$  on MSMT17 and  $91.2\%$  on MPII with 100-epoch pre-training. The results are comparable to our HAP with ViTPose [78] as the pose detector (72.2% and  $91.3\%$  on MSMT17 and MPII respectively), demonstrating that our HAP is generally robust to the keypoint detector method.
D Study of structure-invariant alignment
Table 13: Analysis of loss weight  $\gamma$ .  $\gamma = 0$  represents our baseline that does not use alignment loss.
γ	MSMT17	MPII
0.5	75.6	91.2
0.1	76.3	91.8
0.05	76.4	91.8
0.01	76.0	91.6
0 (baseline)	75.1	91.1
Loss weight  $\gamma$ . Table 13 demonstrates that the weight  $\gamma$  of Eq.2 is beneficial across a wide range.
Table 14: Type of two views in alignment. The "feature 1" and "feature 2" are aligned via  $\mathcal{L}_{\mathrm{align}}$
feature 1	feature 2	MSMT17	MPII
masked view	global view	74.5	91.0
masked view	visible view	76.1	91.7
masked view	masked view	76.4	91.8
baseline	75.1	91.1
Type of two views in alignment. By default, we use two random masked views for alignment. We further explore other two types of alignment, see Table 14. Aligning the masked view with the global view (i.e., without masking) shows poor performance. We attribute it to the high degree of overlap between the two views, which makes the task too easy. Aligning the masked view and the visible view with no overlap yields improvement, but still inferior than the default. It shows that learning subtle humans characteristics by aligning random views of corrupted body structure is important.
Removing negative samples in the alignment loss. We use the loss of SimSiam [11] instead. The results are  $76.1\%$  (SimSiam) vs.  $76.4\%$  (ours) on MSMT17,  $91.6\%$  (SimSiam) vs.  $91.8\%$  (ours) on MPII. It demonstrates that our HAP still works well if removing the negatives in the alignment loss.
(a) LUPerson [23] (for pre-training).
(b) MSMT17 [74] (for person ReID).
(c) Market-1501 [87] (for person ReID).
(d) MPII [1] (for 2D pose estimation).
(e) MS COCO [46] (for 2D pose estimation).
(f) AI Challenger (AIC) [75] (for 2D pose estimation).
(g) CUHK-PEDES [42] (for text-to-image ReID).
(h) ICFG-PEDES [20] (for text-to-image ReID).
(i) RSTPReid [89] (for text-to-image ReID).
(j) 3DPW [67] (for 3D pose and shape estimation).
(k) PA-100K [47] (for attribute recognition).
(1) RAP [39] (for attribute recognition).
(m) PETA [18] (for attribute recognition).
Figure 8: Example images of the used datasets. Our proposed HAP is pre-trained on (a) LUPerson, and evaluated on a broad range of tasks and datasets, including (b-c) person ReID, (d-f) 2D pose estimation, (g-i) text-to-image person ReID, (j) 3D pose and shape estimation, (k-m) pedestrian attribute recognition. These datasets encompass diverse environments with varying image quality, where persons are varied by scales, occlusions, orientations, truncation, clothing, appearances, and postures, demonstrating the great generalization ability of HAP in human-centric perception.