Title: HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Venue: NeurIPS 2023
Accepted: yes
Keywords: human centric perception

masked image modeling

structural-aware pre-training
Abstract: Model pre-training is essential in human-centric perception. In this paper, we first introduce masked image modeling (MIM) as a pre-training approach for this task. Upon revisiting the MIM training strategy, we reveal that human structure priors offer significant potential. Motivated by this insight, we further incorporate an intuitive human structure prior - human parts - into pre-training. Specifically, we employ this prior to guide the mask sampling process. Image patches, corresponding to human part regions, have high priority to be masked out. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across a range of human-centric perception tasks. To further capture human characteristics, we propose a structure-invariant alignment loss that enforces different masked views, guided by the human part prior, to be closely aligned for the same image. We term the entire method as HAP. HAP simply uses a plain ViT as the encoder yet establishes new state-of-the-art performance on 11 human-centric benchmarks, and on-par result on one dataset. For example, HAP achieves 78.1% mAP on MSMT17 for person re-identification, 86.54% mA on PA-100K for pedestrian attribute recognition, 78.2% AP on MS COCO for 2D pose estimation, and 56.0 PA-MPJPE on 3DPW for 3D pose and shape estimation.

HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception

Abstract

Model pre-training is essential in human-centric perception. In this paper, we first introduce masked image modeling (MIM) as a pre-training approach for this task. Upon revisiting the MIM training strategy, we reveal that human structure priors offer significant potential. Motivated by this insight, we further incorporate an intuitive human structure prior - human parts - into pre-training. Specifically, we employ this prior to guide the mask sampling process. Image patches, corresponding to human part regions, have high priority to be masked out. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across a range of human-centric perception tasks. To further capture human characteristics, we propose a structure-invariant alignment loss that enforces different masked views, guided by the human part prior, to be closely aligned for the same image. We term the entire method as HAP. HAP simply uses a plain ViT as the encoder yet establishes new state-of-the-art performance on 11 human-centric benchmarks, and on-par result on one dataset. For example, HAP achieves 78.1% mAP on MSMT17 for person re-identification, 86.54% mA on PA-100K for pedestrian attribute recognition, 78.2% AP on MS COCO for 2D pose estimation, and 56.0 PA-MPJPE on 3DPW for 3D pose and shape estimation.

1 Introduction

Human-centric perception has emerged as a significant area of focus due to its vital role in realworld applications. It encompasses a broad range of human-related tasks, including person reidentification (ReID) [23, 32, 62, 90], 2D/3D human pose estimation [15, 78, 82], pedestrian attribute recognition [34, 36, 63], etc . Considering their training efficiency and limited performance, recent research efforts [8, 13, 16, 33, 64] have dedicated to developing general human-centric pre-trained models by using large-scale person datasets, serving as foundations for diverse human-centric tasks.

Among these studies, some works [8, 13, 33] employ contrastive learning for self-supervised pretraining, which reduces annotation costs of massive data in supervised pre-training [16, 64]. In this work, we take the first step to introduce masked image modeling (MIM), which is another form of selfsupervised learning, into human-centric perception, since MIM [2, 9, 30, 77] has shown considerable potential as a pre-training scheme by yielding promising performance on vision tasks [44, 65, 78].

pre-trained

encoder

task-

head

images

human parts

encoder

(ViT)

decoder

structure-aware pre-training

human-centric perception tasks

age: 16-30

hair: short

wear: sneaker

upper body: red

lower body: red

attribute recognition

3D pose

text-to-image ReID

A football

player in a

red jersey

with black

gloves.

Market-1501

MPII

COCO

AIC

CUHK-PEDES

ICFG-PEDES

RSTPReid

PA-100K

RAP

PETA

78.1

77.1

62.9

93.9

93.9

82.9

89.6

93.3

93.6

75.7

77.2

78.2

31.3

35.0

38.1

60.19

65.25

68.05

61.80

53.68

59.02

49.35

44.15

47.35

86.54

86.37

79.56

82.91

75.73

82.82

88.36

88.0

80.82

MAE

previous SOTA

HAP

→

?

Figure 1: (a) Overview of our proposed HAP framework. HAP first utilizes human structure priors, e.g. , human body parts, for human-centric pre-training. HAP then applies the pre-trained encoder, along with a task-specific head, for addressing each of the human-centric perception tasks. (b) Our HAP demonstrates state-of-the-art performance across a broad range of human-centric benchmarks.

There is no such thing as a free lunch. Upon initiating with a simple trial, i.e. , directly applying the popular MIM method [30] with the default settings for human-centric pre-training, it is unsurprising to observe poor performance. This motivates us to delve deep into understanding why current MIM methods are less effective in this task. After revisiting the training strategies, we identify three crucial factors that are closely related to human structure priors, including high scaling ratio 2 , block-wise mask sampling strategy, and intermediate masking ratio, have positive impacts on human-centric pre-training. Inspired by this, we further tap the potential of human structure-aware prior knowledge.

In this paper, we incorporate human parts [5, 26, 57, 60, 80], a strong and intuitive human structure prior, into human-centric pre-training. This prior is employed to guide the mask sampling strategy. Specifically, we randomly select several human parts and mask the corresponding image patches located within these selected regions. These part-aware masked patches are then reconstructed based on the clues of the remaining visible patches, which can provide semantically rich body structure information. This approach encourages the model to learn contextual correlations among body parts, thereby enhancing the learning of the overall human body structure. Moreover, we propose a structure-invariant alignment loss to better capture human characteristics. For a given input image, we generate two random views with different human parts being masked out through the proposed mask sampling strategy. We then align the latent representations of the two random views within the same feature space. It preserves the unique structural information of the human body, which plays an important role in improving the discriminative ability of the pre-trained model for downstream tasks.

We term the overall method as HAP, short for H uman structureA ware P re-training with MIM for human-centric perception. HAP is simple in terms of modalities, data, and model structure. We unify the learning of two modalities (images and human keypoints) within a single dataset (LUPerson [23]). The model structure is based on the existing popular MIM methods, e.g., MAE [30] and CAE [9], with a plain ViT [21] as the encoder, making it easy to transfer to a broad range of downstream tasks (see Figure 1a). Specifically, HAP achieves state-of-the-art results on 11 human-centric benchmarks (see Figure 1b), e.g. , 78.1% mAP on MSMT17 for person ReID, 68.05% Rank-1 on CUHK-PEDES for text-to-image person ReID, 86.54% mA on PA-100K for pedestrian attribute recognition, and 78.2% AP on MS COCO for 2D human pose estimation. In summary, our main contributions are:

We are the first to introduce MIM as a human-centric pre-training method, and reveal that integrating the human structure priors are beneficial for MIM in human-centric perception.

We present HAP, a novel method that incorporates human part prior into the guidance of the mask sampling process to better learn human structure information, and design a structure-invariant alignment loss to further capture the discriminative characteristics of human body structure.

The proposed HAP is simple yet effective that achieves superior performance on 11 prominent human benchmarks across 5 representative human-centric perception tasks.

2 Related work

Human-centric perception includes a broad range of human-related tasks. The goal of person Re-ID [24, 32, 51, 50, 81, 90, 91, 84, 86] and text-to-image person ReID [59, 20, 62, 58] is to identify persons of interest based on their overall appearance or behavior. Pedestrian attribute recognition [35, 36] aims to classify the fine-grained attributes and characteristics of persons. 2D [78, 79, 82] and 3D [69, 14, 25] pose estimation learn to predict position and orientation of individual body parts.

Recent works make some efforts to pre-train a human-centric model. HCMoCo [33] learns modalinvariant representations by contrasting features of dense and sparse modalities. LiftedCL [13] extracts 3D information by adversarial learning. UniHCP [16] and PATH [64] design multi-task co-training approaches. SOLIDER [8] tries to balance the learning of semantics and appearances.

Compared with these works, this paper focuses on exploring the advantage of utilizing human structure priors in human-centric pre-training. Besides, instead of training with numerous modalities and datasets using a specially designed structure, we pursue a simple pre-training framework by employing two modalities (images and human keypoints) of one dataset (LUPerson [23]) and vanilla ViT structure with the existing popular MIM methods ( e.g. , BEiT [2], MAE [30], and CAE [9]).

Self-supervised learning has two main branches: contrastive learning (CL) and masked image modeling (MIM). CL [4, 7, 10, 11, 12, 28, 31] is a discriminative approach that learns representations by distinguishing between similar and dissimilar image pairs. Meanwhile, MIM [2, 9, 30, 56, 72, 73, 77, 85, 88] is a generative approach that learns to recover the masked content in the corrupted input images. It is shown that CL learns more discriminative representations [4, 12], while MIM learns finer grained semantics [2, 30]. To obtain transferable representations for the diverse tasks, our work combines both of them to extract discriminative and fine-grained human structure information.

Semantic-guided masking is presented in some recent MIM works [6, 29, 37, 40, 45, 68, 83]. Many methods [83, 29, 6, 37] utilize the attention map to identify and mask informative image patches, which are then reconstructed to facilitate representation learning. In comparison, this paper explores to directly employ the human structure priors to guide mask sampling for human-centric perception.

3 HAP

Inspired by the promising performance of MIM in self-supervised learning, we initially employ MIM as the pre-training scheme for human-centric perception. Compared with MIM, our proposed HAP integrates human structure priors into MIM for human-centric pre-training, and introduces a structure-invariant alignment loss. The overview of our HAP framework is shown in Figure 2.

3.1 Preliminary: great potential of human structure priors

We use MAE [30] with its default settings as an example for human-centric pre-training. LUPerson [23] is used as the pre-training dataset following [8, 23, 51]. We designate this setting as our baseline . The results are shown in Figure 3. The baseline achieves poor performance on two representative human-centric perception tasks, i.e., person ReID and 2D human pose estimation. This phenomenon motivates us to delve deep into understanding the underlying reasons.

We start by visualizing the input images under different training strategies in Figure 4. Surprisingly, we observe that it is hard to learn information of human body structure from the input images of the previous baseline (see Figure 4 (b-c)). We conjecture that: i) the scaling ratio may be too small to preserve the overall human body structure, as shown in Figure 4 (b), and ii) the masking ratio may be too large to preserve sufficient human structure information, which makes the reconstruction task too difficult to address, as shown in Figure 4 (c).

80% scaling + 50% masking

+ 75% masking

20% scaling

original image

(a)

(b)

(c)

(d)

encoder

encoder

share

[CLS]

[CLS]

regressor

&

decoder

regressor

&

decoder

share

input image

𝐱

left arm, right leg,

upper body

head, left leg,

right arm

target

target

masked image

𝐗

masked image

#

𝐗

a

a

b

b

c

mask

token

EMA

HOG

DALL-E

CLIP

…

EMA

HOG

DALL-E

CLIP

…

Driven by this conjecture, we further study the impacts of scaling ratio and masking ratio in depth. Figure 3 illustrates that both i) high scaling ratio (ranging from 60% to 90%) and ii) mediate masking ratio (ranging from 40% to 60%) contribute positively to the performance improvement of humancentric tasks. Especially, an 80% scaling ratio can bring an approximately 3.5% improvement on MSMT17of the person ReID task. This effectively confirms that information of human body structure is crucial during human-centric pre-training. Additionally, we also discover that iii) block-wise mask sampling strategy performs sightly better than the random mask sampling strategy, since the former one masks semantic human bodies appropriately due to block regions. We refer to these three factors of the training strategy as human structure priors since they are associated with human structure information. In the following exploration, we employ scaling ratio of 80%, masking ratio of 50% with block-wise masking strategy by default. See an example of our optimal strategy in Figure 4 (d).

3.2 Human part prior

Inspired by the above analyses, we consider to incorporate human body parts, which are intuitive human structure priors, into human-centric pre-training. Here, we use 2D human pose keypoints [78], which are convenient to obtain, to partition human part regions by following previous works [60]. Section 3.1 reveals that the mask sampling strategy plays an important role in human-centric pretraining. To this end, we directly use human structure information provided by human body parts as robust supervision to guide the mask sampling process of our HAP during human-centric pre-training.

Specifically, we divide human body into six parts, including head, upper body, left/right arm, and left/right leg according to the extracted human keypoints, following [60]. Given a specific input image x , HAP first embeds x into N patches. With a pre-defined masking ratio β , the aim of mask sampling is to mask N m = max { n ∈ Z| n ≤ β × N } patches, where Z represents the set of integers. Specifically, we randomly select P parts, where 0 ≤ P ≤ 6 . Total N p image patches located on these P part regions are masked out. Then we adjust the number of masked patches to N m based on the instructions introduced later. In this way, unmasked (visible) human body part regions provide semantically rich human structure information for the reconstruction of the masked human body parts, encouraging the pre-trained model to capture the latent correlations among the body parts.

There are three situations for the masked patches adjustment: i) if N p = N m , everything is wellbalanced; ii) if N p < N m , we further use block-wise sampling strategy to randomly select additional



































































































































































































































 















































































 





























N m -N p patches; iii) if N p > N m , we delete patches according to the sequence of human part selection. We maintain a total of N m patches, which contain structure information, to be masked.

Discussion. i) We use human parts instead of individual keypoints as guidance for mask sampling. This is based on the experimental result in Section 3.1, i.e. , the block-wise mask sampling strategy outperforms the random one. More experimental comparisons are available in Appendix. ii) HCMoCo [33] also utilizes keypoints during human-centric pre-training, in which sparse keypoints serve as one of the multi-modality inputs. Different from HCMoCo, we integrate two modalities, including images and keypoints, together for mask sampling. It should be noted that the keypoints in our method are pseudo labels; therefore they may not be accurate enough for supervised learning as in HCMoCo. However, these generated keypoints can be beneficial for the self-supervised learning.

3.3 Structure-invariant alignment

Learning discriminative human characteristics is essential in human-centric perception tasks. For example, person ReID requires to distinguish different pedestrians based on subtle details and identify the same person despite large variances. To address this issue, we further propose a structure-invariant alignment loss, to enhance the representation ability of human structural semantic information.

For each person image x , we generate two different views through the proposed mask sampling strategy with guidance of human part prior. These two views, denoted as X and ˜ X , display different human part regions for the same person. After model encoding, each view obtains respective latent feature representations Z and ˜ Z on the normalized [CLS] tokens. Our structure-invariant alignment loss L align is applied to Z and ˜ Z for aligning them. We achieve it by using the InfoNCE loss [54]:

where B is the batch size. τ is a temperature hyper-parameter set to 0.2 [12]. In this way, the encoder is encouraged to learn the invariance of human structure, and its discriminative ability is improved.

3.4 Overall: HAP

To sum up, in order to learn the fine-grained semantic structure information, our HAP incorporates the human part prior into the human-centric pre-training, and introduces a structure-invariant alignment loss to capture discriminative human characteristics. The whole loss function to be optimized is:

where L recon is the MSE loss for reconstruction following [30]. γ is the weight to balance the two loss functions of L recon and L align . We set γ to 0.05 by default, and leave its discussion in Appendix.

4 Experiments

4.1 Settings

Pre-training. LUPerson [23] is a large-scale person dataset, consisting of about 4.2M images of over 200K persons across different environments. Following [51], we use the subset of LUPerson with 2.1M images for pre-training. The resolution of the input image is set to 256 × 128 and the batch size is set to 4096. The human keypoints of LUPerson, i.e. , the human structure prior of HAP, are extracted by ViTPose [78]. The encoder model structure of HAP is based on the ViT-Base [21]. HAP adopts AdamW [49] as the optimizer in which the weight decay is set to 0.05. We use cosine decay learning rate schedule [48], and the base learning rate is set to 1.5e-4. The warmup epochs are set to 40 and the total epochs are set to 400. The model of our HAP is initialized from the official MAE/CAE model pre-trained on ImageNet by following the previous works [16, 78]. After pre-training, the encoder is reserved, along with task-specific heads, for addressing the following human-centric perception tasks, while other modules ( e.g. , regressor and decoder) are discarded.

Human-centric perception tasks. We evaluate our HAP on 12 benchmarks across 5 human-centric perception tasks, including person ReID on Market-1501 [87] and MSMT17 [74], 2D pose estimation on MPII [1], COCO [46] and AIC [75], text-to-image person ReID on CUHK-PEDES [42], ICFGPEDES [20] and RSTPReid [89], 3D pose and shape estimation on 3DPW [67], pedestrian attribute

(e) Text-to-image ReID. Rank-1(%) is reported.

recognition on PA-100K [47], RAP [39] and PETA [18]. For person ReID, we use the implementation of [50] and report mAP. For 2D pose estimation, the evaluation metrics are PCKh for MPII and AP for COCO and AIC. The codebase is based on [78]. Rank-1 is reported for text-to-image person ReID with the implementation of [59]. mA is reported in the pedestrian attribute recognition task with the codebase of [36]. The evaluation metrics are MPJPE/PA-MPJPE/MPVPE for 3D pose and shape estimation. Since there is no implementation with a plain ViT structure available for this task, we modify the implementation based on [15]. More training details can be found in Appendix.

4.2 Main results

We compare HAP with both previous task-specific methods and human-centric pre-training methods. Our HAP is simple by using only one dataset of ∼ 2.1M samples, compared with existing pre-training methods (Table1a), yet achieves superior performance on various human-centric perception tasks.

In Table 1, we report the results of HAP with MAE [30] on 12 datasets across 5 representative human-centric tasks. Compared with previous task-specific methods and human-centric pre-training methods, HAP achieves state-of-the-art results on 11 datasets of 4 tasks. We discuss each task next.

Pedestrian attribute recognition. The goal of pedestrian attribute recognition is to assign finegrained attributes to pedestrians, such as young or old, long or short hair, etc . Table 1b shows that HAP surpasses previous state-of-the-art on all of the three commonly used datasets, i.e. , +0.17% on PA-100K [47], +0.57% on RAP [39], and +0.36% on PETA [18]. It can be observed that our proposed HAP effectively accomplishes this fine-grained human characteristic understanding tasks.

Person ReID. The aim of person ReID is to retrieve a queried person across different cameras. Table 1c shows that HAP outperforms current state-of-the-art results of MSMT17 [74] by +5.0% (HAP 78.0% vs. MALE [50] 73.0%) and +1.0% (HAP 78.1% vs. SOLIDER [8] 77.1%) under the input resolution of 256 × 128 and 384 × 128 , respectively. HAP also achieves competitive results on Market-1501 [87], i.e. , 93.9% mAP with 384 × 128 input size. For a fair comparison, we also report the results of HAP with extra convolution modules following [50], denoted as HAP ‡ . It brings further improvement by +1.3% and +2.0% on MSMT17 and Market-1501 with resolution of 384. Review that previous ReID pre-training methods [81, 23, 90, 51] usually adopt contrastive learning approach. These results show that our HAP with MIM is more effective on the person ReID task.

Text-to-image person ReID. This task uses textual descriptions to search person images of interest. We use the pre-trained model of HAP as the initialization of the vision encoder in this task. Table1e shows that HAP performs the best on the three popular text-to-image benchmarks. That is, HAP achieves 68.05%, 61.80% and 49.35% Rank-1 on CUHK-PEDES [42], ICFG-PEDES [20] and RSTPReid [89], respectively. Moreover, HAP is largely superior than MAE [30], e.g. , 68.05% vs. 60.19% (+7.86%) on CUHK-PEDES. It clearly reveals that human structure priors especially human keypoints and structure-invariant alignment indeed provide semantically rich human information, performing well for human identification and generalizing across different domains and modalities.

2D pose estimation. This task requires to localize anatomical keypoints of persons by understanding their structure and movement. For fair comparisons, we also use multiple datasets as in [78] and larger image size as in [82, 8] during model training. Table 1d shows that HAP outperforms previous state-of-the-art results by +0.3%, +1.0%, +3.1% on MPII [1], COCO [46], AIC [75], respectively. Meanwhile, we find both multi-dataset training and large resolution are beneficial for our HAP.

3D pose and shape estimation. We then lift HAP to perform 3D pose and shape estimation by recovering human mesh. Considering that there is no existing method that performs this task on a plain ViT backbone, we thus follow the implementation of [15] and just replace the backbone to a plain ViT. The results are listed in Table 1f. It is interesting to find that despite this simple implementation, our HAP achieves competitive results for the 3D pose task even HAP does not see any 3D information. These observation may suggest that HAP improves representation learning ability on human structure from the 2D keypoints information, and transfers its ability from 2D to 3D.

CAE [9] vs MAE [30]. HAP is model-agnostic that can be integrated into different kinds of MIM methods. We also apply HAP to the CAE framework [9] for human-centric pre-training. Table 2 shows that HAP with CAE [9] achieves competitive performance on various human-centric perception tasks, especially on discriminative tasks such as person ReID that HAP with CAE is superior than HAP with MAE [30] by 1.6% on Market-1501 (with the input resolution of 256 × 128 ), and text-toimage person ReID that HAP with CAE performs better than HAP with MAE by 2.35% on RSTPReid. These observations demonstrate the versatility and scalability of our proposed HAP framework.

4.3 Main properties

We ablate our HAP in Table 3. The analyses of main properties are then discussed in the following. Pre-training on human-centric data. Our baseline, using default settings of [30] and LUPerson [23] as pre-training data, has achieved great improvement than original MAE [30], e.g. , +6.9% mAP on MSMT17 and +6.30% Rank-1 on CUHK-PEDES. It shows that pre-training on human-centric data benefits a lot for human-centric perception by effectively bridging the domain gap between the pre-training dataset and downstream datasets. This observation is in line with previous works [23, 51].

Human structure priors. We denote HAP-0 to represent human structure priors, i.e. , large scaling ratio and block-wise masking with intermediate masking ratio, introduced in Section 3.1. HAP-0 performs better than the baseline method, especially on MSMT17 with +5.2% improvement. It clearly reveals that human structure priors can help the model learn more human body information.

Human part prior. We further use human part prior to guide the mask sampling process. Compared with HAP-1 and HAP-0, part-guided mask sampling strategy brings large improvement, e.g. , +1.0% on MSMT17 and +0.33% on PA-100K. It clearly shows that the proposed mask sampling strategy with the guidance of human part prior has positive impact on the human-centric perception tasks.

Structure-invariant alignment. The structure-invariant alignment loss is applied on different masked views to align them together. Compared with HAP-2 vs. HAP-0 and HAP vs. HAP-1, this alignment loss results in +1.5% and +1.3% improvement on MSMT17, and +0.7% and +0.7% improvement on MPII, respectively. It reflects that the model successfully captures human characteristics with this loss. We provide more discussions about the alignment choices as well as the weight in Appendix.

Overall, our HAP greatly outperforms the original MAE [30] and the baseline by large margins across various human-centric perception tasks, showing significant benefits of the designs of our HAP.

4.4 Ablation study

Number of samples. We compare using different number of data samples during pre-training in Table 4 (left). Only using ∼ 0.5M LUPerson data already outperforms original MAE [30] pre-trained on ImageNet [17]. Meanwhile, more pre-training data brings larger improvement for our HAP.

Training epochs. Table 4 (right) shows that pre-training only 100 epochs already yields satisfactory performance. It suggests that HAP can effectively reduces training cost. These observations indicate that HAP, as a human-centric pre-training method, effectively reduces annotation, data, and training cost. Therefore, we believe HAP can be useful for real-world human-centric perception applications.

Visualization. To explore the structure information learned in the plain ViT encoder of HAP, we probe its attention map by querying the patch of a random body keypoint. See visualizations in Figure 6. Interestingly, we observe that HAP captures the relation among body parts. For example,

27

when querying the right hip of the player in the first image pair (from LUPerson), we see that there are strong relations to the left leg, left elbow, and head. And when querying the left knee of the skateboarder in the second image pair (from COCO), it is associated with almost all other keypoints. These observations intuitively illustrate that HAP learns human body structure and generalizes well.

5 Conclusion

This paper makes the first attempt to use Masked Image Modeling (MIM) approach for human-centric pre-training, and presents a novel method named HAP. Upon revisiting several training strategies, HAP reveals that human structure priors have great benefits. We then introduces human part prior as the guidance of mask sampling and a structure-invariant alignment loss to align two random masked views together. Experiments show that our HAP achieves competitive performance on various human-centric perception tasks. Our HAP framework is simple by unifying the learning of two modalities within a single dataset. HAP is also versatility and scalability by being applied to different MIM structures, modalities, domains, and tasks. We hope our work can be a strong baseline with generative pre-training method for the future researches on this topic. We also hope our experience in this paper can be useful for the general human-centric perception and push this frontier.

Broader impacts. The proposed method is pre-trained and evaluated on several person datasets, thus could reflect biases in these datasets, including those with negative impacts. When using the proposed method and the generated images in this paper, one should pay careful attention to dataset selection and pre-processing to ensure a diverse and representative sample. To avoid unfair or discriminatory outcomes, please analyze the model's behavior with respect to different demographics and consider fairness-aware loss functions or post-hoc bias mitigation techniques. We strongly advocate for responsible use, and encourage the development of useful tools to detect and mitigate the potential misuse of our proposed method and other masked image modeling techniques for unethical purposes.

Limitations. We use human part prior to guide pre-training, which is extracted by existing pose estimation methods. Therefore, the accuracy of these methods could affect the pre-training quality. Besides, due to the limitations in patch size, the masking of body parts may not be precise enough. If there are better ways to improve this, it should further enhance the results. Nonetheless, the proposed HAP method still shows promising performance for a wide range of human-centric perception tasks.

Acknowledgments and Disclosure of Funding

We thank Shixiang Tang and Weihua Chen for discussions about the experiments of human-centric pre-training, and Yufei Xu and Yanzuo Lu for discussions about downstream tasks. This work was supported in part by Zhejiang Provincial Natural Science Foundation of China (LZ22F020012), National Natural Science Foundation of China (62376243, 62037001, U20A20387), Young Elite Scientists Sponsorship Program by CAST (2021QNRC001), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010), Project by Shanghai AI Laboratory (P22KS00111), Program of Zhejiang Province Science and Technology (2022C01044), the Fundamental Research Funds for the Central Universities (226-2022-00142, 226-2022-00051).

References

M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition , pages 3686-3693, 2014.

H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7291-7299, 2017.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

C. Chen, M. Ye, M. Qi, J. Wu, J. Jiang, and C.-W. Lin. Structure-aware positional transformer for visible-infrared person re-identification. IEEE Transactions on Image Processing , 31:2352-2364, 2022.

H. Chen, W. Zhang, Y. Wang, and X. Yang. Improving masked autoencoders by learning where to mask. arXiv preprint arXiv:2303.06583 , 2023.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

W. Chen, X. Xu, J. Jia, H. Luo, Y. Wang, F. Wang, R. Jin, and X. Sun. Beyond appearance: a semantic controllable self-supervised learning framework for human-centric visual tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

X. Chen, M. Ding, X. Wang, Y. Xin, S. Mo, Y. Wang, S. Han, P. Luo, G. Zeng, and J. Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026 , 2022.

X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Z. Chen, Q. Li, X. Wang, and W. Yang. Liftedcl: Lifting contrastive learning for human-centric perception. In The Eleventh International Conference on Learning Representations , 2023.

H. Choi, G. Moon, and K. M. Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 769-787. Springer, 2020.

H. Choi, G. Moon, J. Park, and K. M. Lee. Learning to estimate robust 3d human mesh from in-the-wild crowded scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1475-1484, 2022.

Y. Ci, Y. Wang, M. Chen, S. Tang, L. Bai, F. Zhu, R. Zhao, F. Yu, D. Qi, and W. Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Y. Deng, P. Luo, C. C. Loy, and X. Tang. Pedestrian attribute recognition at far distance. In Proceedings of the 22nd ACM international conference on Multimedia , pages 789-792, 2014.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Z. Ding, C. Ding, Z. Shao, and D. Tao. Semantically self-aligned network for text-to-image part-aware person re-identification. arXiv preprint arXiv:2107.12666 , 2021.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Q. Fang, Q. Shuai, J. Dong, H. Bao, and X. Zhou. Reconstructing 3d human pose by watching humans in the mirror. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12814-12823, 2021.

D. Fu, D. Chen, J. Bao, H. Yang, L. Yuan, L. Zhang, H. Li, and D. Chen. Unsupervised pre-training for person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14750-14759, 2021.

D. Fu, D. Chen, H. Yang, J. Bao, L. Yuan, L. Zhang, H. Li, F. Wen, and D. Chen. Large-scale pre-training for person re-identification with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2476-2486, 2022.

S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik. Humans in 4d: Reconstructing and tracking humans with transformers. arXiv preprint arXiv:2305.20091 , 2023.

K. Gong, X. Liang, D. Zhang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 932-940, 2017.

A. Graves and A. Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks , pages 37-45, 2012.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

J. Gui, Z. Liu, and H. Luo. Good helper is around you: Attention-driven masked image modeling. arXiv preprint arXiv:2211.15362 , 2022.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

S. He, H. Luo, P. Wang, F. Wang, H. Li, and W. Jiang. Transreid: Transformer-based object re-identification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 15013-15022, 2021.

F. Hong, L. Pan, Z. Cai, and Z. Liu. Versatile multi-modal pre-training for human-centric perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1615616166, 2022.

J. Jia, X. Chen, and K. Huang. Spatial and semantic consistency regularizations for pedestrian attribute recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 962-971, 2021.

J. Jia, N. Gao, F. He, X. Chen, and K. Huang. Learning disentangled attribute representations for robust pedestrian attribute recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 1069-1077, 2022.

J. Jia, H. Huang, X. Chen, and K. Huang. Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting. arXiv preprint arXiv:2107.03576 , 2021.

I. Kakogeorgiou, S. Gidaris, B. Psomas, Y. Avrithis, A. Bursuc, K. Karantzalos, and N. Komodakis. What to hide from your students: Attention-guided masked image modeling. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pages 300-318. Springer, 2022.

N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2252-2261, 2019.

D. Li, Z. Zhang, X. Chen, and K. Huang. A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE transactions on image processing , 28(4):1575-1590, 2018.

G. Li, H. Zheng, D. Liu, C. Wang, B. Su, and C. Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. arXiv preprint arXiv:2206.10207 , 2022.

S. Li, M. Cao, and M. Zhang. Learning semantic-aligned feature representation for text-based person search. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2724-2728. IEEE, 2022.

S. Li, T. Xiao, H. Li, B. Zhou, D. Yue, and X. Wang. Person search with natural language description. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1970-1979, 2017.

W. Li, Z. Cao, J. Feng, J. Zhou, and J. Lu. Label2label: A language modeling framework for multi-attribute learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII , pages 562-579. Springer, 2022.

Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 280-296. Springer, 2022.

Z. Li, Z. Chen, F. Yang, W. Li, Y. Zhu, C. Zhao, R. Deng, L. Wu, R. Zhao, M. Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems , 34:13165-13176, 2021.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, and X. Wang. Hydraplus-net: Attentive deep features for pedestrian analysis. In Proceedings of the IEEE international conference on computer vision , pages 350-359, 2017.

I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR , 2017.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Y. Lu, M. Zhang, Y. Lin, A. J. Ma, X. Xie, and J. Lai. Improving pre-trained masked autoencoder via locality enhancement for person re-identification. In Pattern Recognition and Computer Vision: 5th Chinese Conference, PRCV 2022, Shenzhen, China, November 4-7, 2022, Proceedings, Part II , pages 509-521. Springer, 2022.

H. Luo, P. Wang, Y. Xu, F. Ding, Y. Zhou, F. Wang, H. Li, and R. Jin. Self-supervised pre-training for transformer-based person re-identification. arXiv preprint arXiv:2111.12084 , 2021.

G. Moon and K. M. Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 752-768. Springer, 2020.

G. Moon and K. M. Lee. Neuralannot: Neural annotator for in-the-wild expressive 3d human pose and mesh training sets. arXiv preprint arXiv:2011.11232 , 5, 2020.

A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10975-10985, 2019.

Z. Peng, L. Dong, H. Bao, Q. Ye, and F. Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366 , 2022.

R. Quan, X. Dong, Y. Wu, L. Zhu, and Y. Yang. Auto-reid: Searching for a part-aware convnet for person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3750-3759, 2019.

Z. Shao, X. Zhang, C. Ding, J. Wang, and J. Wang. Unified pre-training with pseudo texts for text-to-image person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11174-11184, 2023.

Z. Shao, X. Zhang, M. Fang, Z. Lin, J. Wang, and C. Ding. Learning granularity-unified representations for text-to-image person re-identification. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5566-5574, 2022.

C. Su, J. Li, S. Zhang, J. Xing, W. Gao, and Q. Tian. Pose-driven deep convolutional model for person reidentification. In Proceedings of the IEEE international conference on computer vision , pages 3960-3969, 2017.

K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5693-5703, 2019.

W. Suo, M. Sun, K. Niu, Y. Gao, P. Wang, Y. Zhang, and Q. Wu. A simple and robust correlation filtering method for text-based person search. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 726-742. Springer, 2022.

C. Tang, L. Sheng, Z. Zhang, and X. Hu. Improving pedestrian attribute recognition with weakly-supervised multi-scale attribute-specific localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4997-5006, 2019.

S. Tang, C. Chen, Q. Xie, M. Chen, Y. Wang, Y. Ci, L. Bai, F. Zhu, H. Yang, L. Yi, et al. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602 , 2022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV) , pages 601-617, 2018.

H. Wang, K. Song, J. Fan, Y. Wang, J. Xie, and Z. Zhang. Hard patches mining for masked image modeling. arXiv preprint arXiv:2304.05919 , 2023.

X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3024-3033, 2021.

Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li. Caibc: Capturing all-round information beyond color for text-based person retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5314-5322, 2022.

Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li. Look before you leap: Improving text-based person retrieval by learning a consistent cross-modal common manifold. In Proceedings of the 30th ACM International Conference on Multimedia , pages 1984-1992, 2022.

C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for selfsupervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14668-14678, 2022.

L. Wei, L. Xie, W. Zhou, H. Li, and Q. Tian. Mvp: Multimodality-guided visual pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pages 337-353. Springer, 2022.

L. Wei, S. Zhang, W. Gao, and Q. Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 79-88, 2018.

J. Wu, H. Zheng, B. Zhao, Y. Li, B. Yan, R. Liang, W. Wang, S. Zhou, G. Lin, Y. Fu, et al. Ai challenger: A large-scale dataset for going deeper in image understanding. arXiv preprint arXiv:1711.06475 , 2017.

B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV) , pages 466-481, 2018.

Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653-9663, 2022.

Y. Xu, J. Zhang, Q. Zhang, and D. Tao. Vitpose: Simple vision transformer baselines for human pose estimation. arXiv preprint arXiv:2204.12484 , 2022.

S. Yang, Z. Quan, M. Nie, and W. Yang. Transpose: Keypoint localization via transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11802-11812, 2021.

Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transactions on pattern analysis and machine intelligence , 35(12):2878-2890, 2012.

Z. Yang, X. Jin, K. Zheng, and F. Zhao. Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14298-14307, 2022.

Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint arXiv:2110.09408 , 2021.

S. Zhang, F. Zhu, R. Zhao, and J. Yan. Contextual image masking modeling via synergized contrasting without view augmentation for faster and better visual pretraining. In The Eleventh International Conference on Learning Representations , 2023.

X. Zhang, J. Cao, C. Shen, and M. You. Self-training with progressive augmentation for unsupervised cross-domain person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8222-8231, 2019.

X. Zhang, J. Chen, J. Yuan, Q. Chen, J. Wang, X. Wang, S. Han, X. Chen, J. Pi, K. Yao, J. Han, E. Ding, and J. Wang. CAE v2: Context autoencoder with CLIP latent alignment. Transactions on Machine Learning Research , 2023.

X. Zhang, D. Li, Z. Wang, J. Wang, E. Ding, J. Q. Shi, Z. Zhang, and J. Wang. Implicit sample extension for unsupervised person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7369-7378, 2022.

L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision , pages 1116-1124, 2015.

J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021.

A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua. Dssl: deep surroundings-person separation learning for text-based person retrieval. In Proceedings of the 29th ACM International Conference on Multimedia , pages 209-217, 2021.

K. Zhu, H. Guo, T. Yan, Y. Zhu, J. Wang, and M. Tang. Pass: Part-aware self-supervised pre-training for person re-identification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV , pages 198-214. Springer, 2022.

K. Zhu, H. Guo, S. Zhang, Y. Wang, G. Huang, H. Qiao, J. Liu, J. Wang, and M. Tang. Aaformer: Auto-aligned transformer for person re-identification. arXiv preprint arXiv:2104.00921 , 2021.

A Details of pre-training

Model structure. We adopt MAE-base [30] as the basic structure of HAP. It consists of an encoder and a lightweight decoder. The encoder follows the standard ViT-base [21] architecture, comprising 12 common Transformer blocks [66] with 12 attention heads and a hidden size of 768. The decoder is relatively small, which consists of 8 Transformer blocks with 16 attention heads and a hidden size of 512. Sine-cosine position embeddings are added to representations of both encoder and decoder 3 . We also use CAE-base [9] as the basic structure. Please refer to [9] for more details of the structure.

Training data. LUPerson [23] is a large-scale person dataset with ∼ 4.2M images (see Figure 8a). Following [51], we use a subset of LUPerson with ∼ 2.1M images for pre-training. The resolution of the input images is set to 256 × 128 following previous works [23, 90, 51, 50]. The data augmentation follows [50], which involves random resizing and cropping, and horizontal flipping. We utilize ViTPose [78], a popular 2D pose estimation method, to extract the keypoints of LUPerson as the human structure prior in our HAP. Given that our HAP does not rely on ground-truth labels, it can be pre-trained on a combination of several large-scale person datasets after extracting their keypoints. We speculate that employing multi-dataset co-training can further improve the performance of HAP, as observed in [78]. We also recognize that our HAP would derive great benefits from more accurate keypoints ( e.g. , human-annotated keypoints) if available. We will discuss this point in details later.

Implementations details. We follow most of the training settings of MAE/CAE except using training epochs of 400 (Table 5). Different from MAE/CAE, HAP has a large scaling ratio of 80%, and an intermediate masking ratio of 50% with block-wise masking strategy, as introduced in the main paper.

B Details of human-centric perception tasks

We list the training details of human-centric perception tasks in Table 1 of the main paper as follows.

B.1 Person ReID

We consider two ReID datasets: MSMT17 [74] (Figure 8b) and Market-1501 [87] (Figure 8c) with the codebase 4 of [50]. The input image resolutions are set to 256 × 128 and 384 × 128 . Data augmentations are the same as [50], including resizing, random flipping, padding, and random cropping. Following [32, 64, 8, 16], the model is fine-tuned using a cross-entropy loss and a triplet loss with equal weights of 0.5. We also consider using a convolution-based module [50] to further improve results. The evaluation metric of mAP (%) is reported. See hyper-parameters in Table 6.

B.2 2D human pose estimation.

HAP is evaluated on MPII [1] (Figure 8d), MS COCO [46] (Figure 8e), and AI Challenger (AIC) [75] (Figure 8f). We use the codebase 5 of [78]. Resolutions of 256 × 192 and 384 × 288 are employed.

Data augmentations are random horizontal flipping, half body transformation, and random scaling and rotation. We follow the common top-down setting, i.e., estimating keypoints of the instances detected [76]. We use mean square error (MSE) [78] as the loss function to minimize the difference between predicted and ground-truth heatmaps. Following [78], we also perform multi-dataset training setting by co-training on MPII, MSCOCO, and AIC. Evaluation metric of PCKH (%) is reported for MPII, and AP (%) is reported for MS COCO and AIC. See the details of hyper-parameters in Table 7.

B.3 Text-to-image person ReID

We employ three commonly used datasets, i.e. , CUHK-PEDES [42] (Figure 8g), ICFG-PEDES [20] (Figure 8h), RSTPReid [89] (Figure 8i). The codebase 6 of [59] is used. The resolution is 384 × 128 . We adopt resizing and random horizontal flipping as data augmentations. BERT [19] is used to extract embeddings of texts, which are then fed to Bi-LSTM [27]. Feature dimensions of images and texts are both set to 384. The BERT is frozen while Bi-LSTM and ViT are fine-tuned. We report Rank-1 (%) metric for the results on all datasets. The details of the hyper-parameters can be found in Table 8.

B.4 3D pose and shape estimation

We perform 3D pose and shape estimation on 3DPW [67] (Figure 8j) using the codebase 7 of [15]. The resolution is 256 × 256 . Data augmentations follow [15]. We minimize a pose loss, i.e. , l 1 distance between the predicted 3D joint coordinates and the pseudo ground-truth, and a mesh loss, i.e. , loss for predicted SMPL parameters [53, 55]. We report MPJPE, PA-MPJPE, MPVPE. The former two are for 3D pose, while the latter one is for 3D shape. Hyper-parameters are in Table 9.

B.5 Pedestrian attribute recognition

We evaluate HAP on three popular datasets, i.e. , PA-100K [47] (Figure 8k), RAP [39] (Figure 8l), and PETA [18] (Figure 8m). We implement it using the codebase 8 of [36]. The resolution is 256 × 192 . Data augmentations consist of resizing and random horizontal flipping. The binary cross-entropy loss is employed, and the evaluation metric of mA (%) is reported. Find the hyper-parameters in Table 10.

C Study of human part prior

head

upper body

left arm

right arm

left leg

right leg

Details of part masking with keypoints. We incorporate human parts, a human structure prior, into pre-training. Figure 7 illustrates the details of how to mask each of the six body parts with keypoints.

Masking guidance in mask sampling strategy. We study different guidance, reported in Table 11. Some MIM works [83, 29, 6, 37] use attention map to identify and mask the informative image patches. However, this design performs poorly in human-centric pre-training, achieving even lower performance than baseline (71.7 vs. 75.6 on MSMT17). We conjecture that the underlying reason is the challenge for the model to obtain precise and meaningful attention maps, especially in the initial stage of the pre-training process. That may be why recent works [40, 37, 6] start to employ strong pretrained models to assist mask sampling. In contrast, we directly employ intuitive and explicit human part prior, i.e. , human keypoints, to guide mask sampling, which shows great potential. Besides, we observe that only masking patches located on keypoints, instead of human parts, improves little. It indicates that masking a whole human body works better, similar to the finding that block-wise masking performs better than random masking in human-centric pre-training (see the main paper).

Part selection. Here, we ablate different part selection strategies in Table 12. We observe that selecting a fixed number of parts is not beneficial. Compared to a whole range, selecting within a small range has a little improvement. It shows that more part variations brings more randomness, which is good for pre-training. Similar to the findings in recent studies [83, 29, 6, 37] that a deliberate

design of mask sampling strategy would have a positive impact, our part selection strategy is beneficial for human-centric pre-training, resulting in great performance on a range of downstream tasks.

Using keypoints as target. Since keypoints can also be used as supervision, we perform this baseline by making the decoder predict keypoints instead of pixels. The obtained results are 50.3% on MSMT17 and 89.8% on MPII with 100-epoch pre-training, while our HAP is 72.2% and 91.3%. The possible reason is that regressing keypoints may lose informative details provided by pixels, which has negative impact on the tasks. In contrast, HAP incorporates prior knowledge of structure into pre-training, which is beneficial to maintain both semantic details and information of body structure.

Using another keypoint detector. To study the robustness of HAP to the extracted keypoints, we make a simple exploration by employing another off-the-shelf keypoint detector named OpenPose [3]. It achieves 72.0% on MSMT17 and 91.2% on MPII with 100-epoch pre-training. The results are comparable to our HAP with ViTPose [78] as the pose detector (72.2% and 91.3% on MSMT17 and MPII respectively), demonstrating that our HAP is generally robust to the keypoint detector method.

D Study of structure-invariant alignment

Loss weight γ . Table 13 demonstrates that the weight γ of Eq.2 is beneficial across a wide range.

Type of two views in alignment. By default, we use two random masked views for alignment. We further explore other two types of alignment, see Table 14. Aligning the masked view with the global view ( i.e. , without masking) shows poor performance. We attribute it to the high degree of overlap between the two views, which makes the task too easy. Aligning the masked view and the visible view with no overlap yields improvement, but still inferior than the default. It shows that learning subtle humans characteristics by aligning random views of corrupted body structure is important.

Removing negative samples in the alignment loss. We use the loss of SimSiam [11] instead. The results are 76.1% (SimSiam) vs. 76.4% (ours) on MSMT17, 91.6% (SimSiam) vs. 91.8% (ours) on MPII. It demonstrates that our HAP still works well if removing the negatives in the alignment loss.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Human structure-Aware Pre-training (HAP) method that incorporates human structure priors into the masked image modeling (MIM) training strategy [26] for tasks related to human-centric perception. The authors have demonstrated the advantages of the proposed method on 5 human-centric perception tasks across 12 benchmark datasets. Overall, the method is simple and intuitive, and extensive experiments have provided solid evidence of its superiority in human-centric perception tasks.\n\nSTRENGTHS:\n1. The author, through analyzing the deficiencies of the current MIM training strategy [26] in human-centric perception tasks, further proposes the introduction of human structure priors to expand the MIM training approach. This method is intuitive and appropriate.\n\n2. Although the author uses the existing InfoNCE loss to implement structure-invariant alignment loss, the application of this structure-invariant alignment concept to different masked views in order to align them, thereby enhancing the feature representation ability of human structure information, is both straightforward and promising.\n\n3. The author not only demonstrated the superiority of the proposed method across twelve benchmark datasets, but also validated the benefits of each proposed training strategy or loss function through ablation experiments.\n\n4. The organization of the article and the presentation of the methods are both clear and well-structured.\n\nWEAKNESSES:\nWhile this method effectively integrates prior knowledge and existing loss functions to enhance the performance of human-centric perception tasks, its technical originality and novelty still leave something to be desired.\n\nQUESTIONS:\n1. The accuracy of pose estimation methods seems to significantly impact the performance of the HAP proposed by the author. However, it appears that the author hasn't conducted sufficient analysis, such as how OpenPose or AlphaPose specifically affect the performance of HAP.\n\n2. In 3D human pose and shape estimation task, the performance of 3D human mesh estimation often decreases due to occlusions. The HAP method proposed by the author seems promising in solving this issue. However, it's unfortunate that the author did not conduct relevant experiments to further highlight the advantages of the proposed HAP. It would be beneficial for the author to incorporate such experiments to enhance the strengths of their HAP.\n\nLIMITATIONS:\nThe author mentions that their proposed method may have potential negative societal impacts when applied to image generation tasks. Personally, I consider this to be a problem that we collectively need to address in the process of advancing artificial intelligence. Therefore, it does not diminish my positive evaluation of the research work. However, I still hope that the author can provide possible solutions to mitigate these potential negative impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce masked image modeling as a pre-training method specifically designed for human-centric perception tasks. To this end, the authors incorporate human structure priors (high scaling ratio, mediate masking ratio, block-wise masking), human part prior (2D keypoints as guidance for mask sampling), and structure-invariant alignment loss (contrastive loss on the [CLS] tokens across views). Extensive experiments demonstrate that the proposed method achieves competitive performance on various human-centric perception tasks.\n\nSTRENGTHS:\n-\tThe proposed method is well motivated. Directly applying MIM in human-centric perception tasks will indeed induce some problems. The authors exploited some priors from the person dataset to make pre-training tailored for specific downstream tasks.\n-\tThe paper is generally well-written and easy to follow.\n-\tThe experiments are extensive and the results are promising.\n\nWEAKNESSES:\n-\tThe technical novelty is limited. The proposed method is essentially a combination of many previous techniques used in masked image modeling. For example, the block-wise masking has been proposed in BEiT [2]. Semantic-guided masking has been proposed in prior works [5, 25, 33, 36, 41, 60, 74]. The only difference is that these works use attention maps while the authors use keypoints. Adding alignment loss is actually combining masked image modeling with contrastive learning (like what has been done in iBOT [77]).\n-\tThe authors use 2D keypoints to guide pre-training, which are extracted by existing pose estimation methods. I am afraid the obtained keypoints are not fully unsupervised.\n\nQUESTIONS:\nI have an additional question. The authors use contrastive loss with negative samples. What about removing negatives? Prior works (e.g., BYOL, SimSiam, DINO) have proven that negative samples are not necessary for better representation learning. Overall, I am leaning towards borderline accept considering that the authors provide a new formulation of previous techniques for a new human-centric perception scenario.\n\nLIMITATIONS:\nThe authors have discussed the limitations and the broader impacts in Sec. 5, which looks good to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies masked image modeling (MIM) in human-centric perception. It first revisits the vanilla MIM and finds that human structure prior (2D pose) helps the downstream human-related tasks. This encourages the authors to incorporate this prior into the classical MAE. Based on this human-centric masking strategy, a structure-invariant alignment loss is developed as a regularization for pre-training. To evaluate the effectiveness of the proposed method, the authors conduct extensive experiments on 11 human-centric benchmarks. The competitive performance is observed when compared to the recent human-centric pre-training counterparts.\n\nSTRENGTHS:\n1) The overall writing and organization are satisfactory. The paper well describes the necessity of human structure prior to human-centric pre-training. The statistics are convincing to convey the motivation to the audience.  \n\n2) The pipeline incorporates MAE with pose-related masking is simple and effective. It can be easily applied to any MIM method. \n\n3) The experiments are sufficient to verify the effectiveness of the proposed method, with informative visualization and ablation studies.\n\n4) Codes are attached, and implementation details are well described for re-implementation.\n\nWEAKNESSES:\nThe performance on multiple benchmarks is overclaimed. For example, the results of Attribute recognition are marginal when comparing HAP with other human-centric pre-training methods. With bells and whistles, HAP (multi-dataset training and larger image size) outperforms other methods significantly. However, the vanilla HAP is inferior to the LiftedCL. PATH, and UniHCP, though they use different pre-training datasets. So, I reckon the superior performance this paper claims should be re-examined, or restated at least.\n\nQUESTIONS:\nAs the technical novelty of this paper is human-related masking, the authors are encouraged to ablate the part selection. For example, the P parts are randomly selected, as denoted in Section 3.2. I am wondering how the performance will be if only making one or two parts, such as the head or upper body. Then that will be six ablation studies (each one masks a part only) to investigate the impact of each part.\n\nLIMITATIONS:\nIn the limitation section, the authors state that the accuracy of pose estimation could affect the pre-training quality. However, the quantitative or qualitative results are not presented.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a pertaining strategy for human-centric vision tasks. They extend the mask image modelling approach by incorporating a prior on the human body parts to guide the mask sampling strategy. In short, they mask parts of the image which contains body part. Authors also propose an alignement loss to make sure that the same image with two different masks have same feature representations. They show fine-tuning performance on several standard benchmarks.\n\nSTRENGTHS:\n1) The paper is well-written and easy to read. The figures are clear and they help to get the main idea of the paper. Since authors do not claim to bring a very novel idea/technical contribution they explain that they start from an existing work (MAE) and try to extend this method for human-centric tasks. They perform experiments to identify issues for this method and they propose patches to fix them. The intuition behind the proposed fixs are based on human prior and are explained with references to prior works (i.e. body parts). Even if we could argue that the novelty is somehow limited, the proposed paper is well-constructed and may have a positive impact in the vision community specially for researchers working on human0centric vision.\n2) The introduction introduced well the research problem, and summarise the main idea and claim of the paper. I appreciate that authors do not claim their method as SSL (even if they do in the Appendix l.13) because they use pseudo-GT (keypoints) for sampling masks in the input images.\n3) Section 3.1 is very much appreciated. I guess that the project started with these experiments so it make sens t.\n4) Authors presented good ablation studies (Table 2-3), as a reader we understand the impact of each components during the pretraing (some important ones are still missing see weaknesses).\n5) Supplementary materials contain all the details for each downstream tasks which make the results reproducible. Furthermore authors also include the code for pretraining using their approach. Authors mentioned that code will be release soon.\n6) Good performance on several downstream benchmarks where they compare against recent works. They reach SoTA on multiple datasets.\n\nWEAKNESSES:\n1) One major weakness of this submission is the lack of a simple baseline which is; pretraining on LUPerson by training to regress the 2d keypoints extracted by VITPose. I understand that they are noisy because they are pseudo-GT but I guess that because there are more than 2M images in LUPerson it should already give a better understand of why we need to pretrain using supervision signal from the pixel instead of using supervision signal for the 2d keypoints. For example for the downstream tasks of 3D human mesh recovery most of the training data used by the research community are pseudo-GT extracted on in the wild images and it works better than using indoor image with perfect ground-truth.\n2) Authors mentioned that 2d keypoints are obtained by VITPose, it would be great to study the impact of the keypoint quality by using another off-the-shell detector such as OpenPose or pretraining on a big dataset with ground-truth 2D pose such as MSCOCO. I think that such ablation is very important for making sure that the key component of your proposed method is: a) to leverage a large-scale dataset for pretrainning — or b) to use a very recent off-the-shell 2d pose detector. At the moment we do not have any answer to this question in the proposed manuscript. I would suggest to pretrain your method on either LUPerson or MSCOCO with the same number of pretraining examples and using either GT 2d pose of 2D pose extracted from VITPose.\n3) I did not find the information in the paper of Supp. Mat. About the initialisation of the weights for the pretraining stage. Do you train from scratch or from MAE-Imagenet?\n4) It is a bit surprising that the fine-tuning on the downstream task of 3D pose does not work as well as [14] since you are just changing the backbone. It is a bit counter-intuitive with the results that you get on the other tasks. Moreover the backbone in [14] is a CNN pertained on 2d pose estimation so the conclusion regarding this task are a bit weird in comparison to the other tasks.\n5) The Structure Alignement Loss is appealing since it bring contrastive learning in the paper but given results in Table 2 it seems that it does not bring a significant gain. The improvements are quite marginal and similar to HAP-2 given a certain variance during the fine-tuning stage (only on MSMT17 results are better than HAP-2).\n6) Other human-centric pretraining method such as PATH are proposing zero-shot or results on downstream task with a frozen encoder. Do you have some results following this fine-tuning strategy (i.e. training only the head). It would be great to see how important it is to fine-tune the encoder for each task. Or maybe using adaptor.\n\nQUESTIONS:\n1) l.44 “Or a given input image, we generate two views. Not sure that we can really say that there are two views. Using the word ‘view’ makes reference to 3D and here the masked images are from the same image not from two different camera viewpoints. So I would suggest to find a different explaination.\n2) Most of my questions are in the weaknesses. Weaknesses 1) and 2) are the most important points to me.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}