HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Authors: Junkun Yuan
Xinyu Zhang
Hao Zhou
Jian Wang
Zhongwei Qiu
Zhiyin Shao
Shaofeng Zhang
Sifan Long
Kun Kuang
Kun Yao
Junyu Han
Errui Ding
Lanfen Lin
Fei Wu
Jingdong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: human centric perception
masked image modeling
structural-aware pre-training
TL;DR: This paper presents a new human-centric pre-training approach called HAP, which incorporates structure priors into training and achieves state-of-the-art results on 11 human-centric benchmarks.

Abstract: Model pre-training is essential in human-centric perception. In this paper, we first introduce masked image modeling (MIM) as a pre-training approach for this task. Upon revisiting the MIM training strategy, we reveal that human structure priors offer significant potential. Motivated by this insight, we further incorporate an intuitive human structure prior - human parts - into pre-training. Specifically, we employ this prior to guide the mask sampling process. Image patches, corresponding to human part regions, have high priority to be masked out. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across a range of human-centric perception tasks. To further capture human characteristics, we propose a structure-invariant alignment loss that enforces different masked views, guided by the human part prior, to be closely aligned for the same image. We term the entire method as HAP. HAP simply uses a plain ViT as the encoder yet establishes new state-of-the-art performance on 11 human-centric benchmarks, and on-par result on one dataset. For example, HAP achieves 78.1% mAP on MSMT17 for person re-identification, 86.54% mA on PA-100K for pedestrian attribute recognition, 78.2% AP on MS COCO for 2D pose estimation, and 56.0 PA-MPJPE on 3DPW for 3D pose and shape estimation.

HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Junkun Yuan 1:2 Xinyu Zhang 2han Hao Zhou?, Jian Wang', Zhongwei Qiu Zhiyin Shao", Shaofeng Zhang", Thang",Site Sifan Long", Kun Kuang' Kun Yao?, Junyu Han", Errui Ding².
Lanfen Lin', Fei Jingdong Wang
'Zhejiang University Baidu VIS Uniiversity of Science and Technology Beijing
4 South China University of Technology Shanghai Jíao Tong University Jilln University {zhangxinyu14 iamgggiinndnngg)baaaa com {yuan kunkuang edu. cn
Project Page: http:///wttttuuoooooo//uuuuuuuaa
Code:
xyz. github..hh...... 1o/hap gi thub.u.... /
Abstract
Model pre-training is essential in human-centric perception. In this paper, we first introduce masked image modeling (MIM) as a pre-training approach for this task. Upon revisiting the MIM training strategy, we reveal that human structure priors offer significant potential. Motivated by this insight, we further incorporate an intuitive human structure prior human parts into pre-training. Specifically, we employ this prior to guide the mask sampling process. Image patches, correspond- ing human part regions, have high priority be masked out. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across range of human-centric perception tasks. To further capture human characteristics, we propose a structure-invariann alignment loss that enforces different masked views, guided by the human part prior, be closely aligned for the same image. We term the entire method as HAP. HAP sim- ply uses plain ViT as the encoder yet establishes new sate-of-the-an performance on 11 numan-centriy benchmarks, and on-par result on one dataset. For example, HAP achieves 78. 8.1% mAP on MSMT17 for person re-identification, 86.54% mA on PA- 100K for pedestrian attribute recognition. 78.2% AP on MS COCO for 2D pose estimation, and 56.0 PA-MPIPE on 3DPW for pose and shape estimation.
Introduction
Human-centric perception has emerged as a significant area of focus due to its vital role in real- world applications. It encompasses a broad range of human-related tasks, including person re-
identification (ReID) 23
2D/3D human pose estimation [15] 78 82 pedestrian attribute 63 etc. Considering their training efficiency and limited performance, recent have dedicated to developing general human- pre-trained models by using large-scale person datasets, serving as foundations for diverse human-centric tasks.
32
62
recognition 34
36
research efforts 8
16
33
64
Among these studies, some works g training, which reduces annotation costs of massive data in supervised pre-training [16] 644. In this work, we take the first step introduce masked image modeling (MIM), which is another form of self- supervised learning, into human-centric perception, since MIM (2 9 30 777 has shown considerable
B3 33 employ contrastive learning for self-supervised pre-
potential as pre-training scheme by yielding promising performance vision tasks 44 65
Equual contribution. This work was done when Junkun Yuan, Zhongwei Qiu, Zhiyin Shao, and Shaofeng Zhang were interns Baidu VIS. :Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

MENU
Mario SM1
FITZA
R2X1
Lecages
(XI))
decoder
pee
tosk- bead
Eolige HADD
4130
poee
paie
mooghtda
strocture- aware pre traininng
buman- Cesttic perceptonn tasks
procioss SOTA
(a) Human-centric perception with HAP.
(b) High performance of HAP.
Figure I: (a) Overview of our proposed HAP framework. HAP first utilizes human structure priors, egg., human body parts, for human-centric pre-training. HAP then applies the pre-trained encoder, along with a task-specific head, for addressing each of the human-centric perception tasks. (b) Our HAP demonstrates state-of-the-art performance across broad range of human-centric benchmarks.
There is no such thing as a free lunch. Upon initiating with simple trial, ie., directly applying the popular MIM method [30 with the default settings for human-centric pre-training, it unsurprising to observe poor performance. This motivates to delve deep into understanding why current MIM methods are less effective in this task. After revisiting the training strategies, we identify three crucial factors that are closely related to human structure priors, including high scaling ratic block-wise mask sampling strategy, and intermediate masking ratio, have positive impacts on human-centric pre-training. Inspired by this, further tap the potential of human structure-awe aware prior knowledge.
In this paper, we incorporate human parts [5 26 57 60 [0). strong and intuitive human structure prior, into human-centric pre-training. This prior is employed to guide the mask sampling strategy. Specifically, we randomly select several human parts and mask the corresponding image patches located within these selected regions. These part-aware masked patches are then reconstructed based on the clues the remaining visible patches, which can provide semantically rich body structure information. This approach encourages the model to learn contextual correlations among body parts, thereby enhancing the learning the overall human body structure. Moreover, we propose a structure-ine tracture-invariant invariant alignment loss to better capture human characteristics. For a given input image, we generate two random views with different human parts being masked out through the proposed mask sampling strategy. We then align the latent representations of the two random views within the same feature space. It preserves the unique structural information the human body, which plays an important role in improving the discriminative ability of the pre-trained model for downstream tasks.
We term the overall method as HAP, short for Human structure- ware Pre-training with MIM for human-centric perception. HAP is simple in terms of modalities, data, and model structure. We unify the learning two modalities (images and human keypoints) within a single dataset (LUPerson 23 The model structure is based on the existing popular MIM methods, e.g., MAE 330 and CAE [9] with plain ViT 21 as the encoder, making it easy to transfer to a broad range of downstream tasks (see Figure Tak. Specifically, HAP achieves state-of ofatee-of-hheaare art results 11 human-centric benchmarks (see Figure lb e.g., 78.1% 1% mAP on MSMTI7 for person ReID, 68.05% Rank-1 on -PEDES for rext-to-image person ReID, 86.54% mA PA-100K for pedestrian attribute recognition, and 78.2% AP on MS COCO for 2D human pose estimation. In summary, our main contributions are:
We are the first to introduce MIM as a human-centric pre-training method, and reveal that integrating the human structure priors are beneficial for MIM in human-centric perception.
We present HAP, a novel method that incorporates human part prior into the guidance of the mask sampling process to better learn human structure information, and design a structure-invariant alignment loss to further capture the discriminative characteristics of human body structure.
The proposed HAP is simple yet effective that achieves superior performance 11 prominent human benchmarks across 5 representative human-centric perception tasks.
We simply refer the lower bound for the ratio of random area to crop before resizing as "scaling ratio". Please find more discussions Section 3

---

2 Related work
Human-centric perception includes broad range of human-related tasks. The goal of person
Re-ID [24] 32 51
50 [1 90
91 84 [6] and text-to-image person RelD [59 20
62
58 is to identify
persons of interest based on their overall appearance or behavior. Pedestrian attribute recognition [35
36 aims to classify the fine-grained attributes and characteristics of persons. 2D
[78] 79 82] and
3D
69
25
pose estimation learn to predict position and orientation of individual body parts.
Recent works make some efforts to pre-train a human-centric model. HCMoCo 33 learns modal- invariant representations by contrasting features of dense and sparse modalities. LiftedCL [ extracts 3D information by adversarial learning. UniHCP [16] and PATH [64] design multi-task co-training approaches. SOLIDER g tries to balance the learning of semantics and appearances.
Compared with these works, this paper focuses on exploring the advantage of utilizing human structure priors in human-centric pre-training. Besides, instead of training with numerous modalities and datasets using a specially designed structure, we pursue a simple pre-training framework by employing two modalities (images and human keypoints) of one dataset (LUPerson (23) and vanilla
ViT structure with the existing popular MIM methods (e.g., BEÏT 2). MAE [30] and CAE (9).
Self-supervieed learning has two main branches: contrastive learning (CL) and masked image modeling (MIM). CL [0] 12 28 31 is a discriminative approach that learns representations
by distinguishing between similar and dissimilar image pairs. Mcanwhile, MIM 2 9 30 56 72 73 is a generative approach that learns to recover the masked content in the corrupted input images. It is shown that CL learns more discriminative representations 4 12 while MIM learns finer grained semantics 2 30. To obtain transferable representations for the diverse tasks, our work combines both of them to extract discriminative and fine-grained human structure information.
77 85 88
Semantic-guided masking is presented in some recent MIM works 66 methods [83] 29 6, 37 utilize the attention map to identify and mask informative image patches, which are then reconstructed to facilitate representation learning. In comparison, this paper explores to directly employ the human structure priors to guide mask sampling for human-centric perception.
68 83 Many
3 HAP
Inspired by the promising performance of MIM in self-supervised learning, we initially employ MIM as the pre-training scheme for human-centric perception. Compared with MIM, our proposed HAP integrates human structure priors into MIM for human-centrie pre-training, and introduces a structure-ing structure-invariant alignment loss. The overview of our HAP framework shown in Figure 2
3.1 Preliminary: great potential of human structure priors
We use MAE 30 with its default settings as an exam- ple for human-centric pre-training. LUPerson (23 is used as the pre-training dataset following 8 23 STI
We designate this setting as our baseline. The results are shown in Figure 3 The baseline achieves poor performance on two representative human-centric per- ception tasks, i.e., person RelD and 2D human pose estimation. This phenomenon motivates us to delve deep into understanding the underlying reasons.
.10%oa4k2
Figure 4: Visualization of training strate- gies. For a given image (a), the base- line (30) uses 20% scaling ratio (b) and 75% masking ratio (c) with random mask sam- pling strategy. yielding meaningless im- age with little human structure information. We adopt 80% scaling ratio and 50% mask- ing ratio with block-wiss mask sampling (d), maintaining the overall body structure.
We start by visualizing the input images under different training strategies in Figure Surprisingly, we observe that it hard to learn information of human body struc- ture from the input images of the previous baseline (see Figure 4 (b-c)). We conjecture that: i) the scaling ratio may be too small to preserve the overall human body structure, as shown in Figure 4 (b), and ii) the mask- ing ratio may be too large preserve sufficient human structure information, which makes the reconstruction task too difficult to address, shown in Figure (c).
3

---

Pixel EMA HOG
regressor & decoder
encoder
DALL-E
CLIP
[CLS]
target
left right upper body
masked X
slare
mask token
share
(CLS)
Pixel
EMA HOG DALL-E CLIP
ingus image
1
regressor & decoder
encodar
Larget
bend, left log,
masked image X
right amn
randomily select human parts for masking
adjust masked patch number to match the pre-defined masking ratio
A smucture-invariant aligument
Our HAP framework. During pre-training, we randomly select human part regions and mask the corresponding image patches. We then adjust the masked patches to match the pre-defined masking ratio. The encoder embeds the visible patches, and the decoder reconstructs the masked patches based latent representations. For given image, HAP generates two views using the mask sampling with human part prior. We apply the proposed structure-invarinn [CLS] token representations of these two views closer together. After pre- pre-training, only the encoder with plain VIT structure is retained for transferring to downstream human-centric perception tasks.
Figure 2:
alignment loss to bring the
Driven by this conjecture, we further study the impacts of scaling ratio and masking ratio in depth. Figure D illlustaaess that both i) high scaling ratio (ranging from 60% to 90%) and ii) mediate masking ratio (ranging from 40% 60%) contribute positively to the performance improvement of human- centric tasks. Especially, an 80% scaling ratio can bring an approximately 3.5% improvement on MSMT17 of the person ReID task. This effectively confirms that information of human body structure crucial during human-centric pre-training. Additionally, we also discover that iii block-wise mask sampling strategy performs sightly better than the random mask sampling strategy, since the former one masks semantic human bodies appropriately due to block regions. We refer to these three factors of the training strategy as human structure priors since they are associated with human structure information. In the following exploration, we employ scaling ratio of 80%, masking ratio of 50% with block-wise masking strategy by default. See example of our optimal strategy Figure (d).
3.2 Human part prior
Inspired by the above analyses, consider incorporate human body parts, which are intuitive human structure priors, into human-centric pre-training. Here, we use 2D human pose keypoints 78
which convenient obtain, to partition human part regions by following previous works [60]. Section 3.1 reveals that the mask sampling strategy plays an important role in human-centric pre-
training. To this end, directly use human structure information provided by human body parts as robust supervision to guide the mask sampling process of our HAP during human-centric pre-training.
Specifically, we divide human body into six parts, including head, upper body, left/right arm, and leftright leg according to the extracted human kcypoints, following [60]. Given a specific input image HAP first embeds into N patches. With pre-defined masking ratio 3, the aim of mask sampling is to mask Nm = max(n E Z < 3 N} patches, where represents the set of integers. Specifically, randomly select P parts, where 0 < P < 6. Total N, image patches located these P part regions are masked out. Then we adjust the number of masked patches to N,, based on the instructions introduced later. In this way, unmasked (visible) human body part regions provide semantically rich human structure information for the reconstruction of the masked human body parts, encouraging the pre-trained model to capture the latent correlations among the body parts.
There are three situations for the masked patches adjustment: i) if N₂ = Nm. everything is well- balanced: ii) if N2 < N further use block-wise sampling strategy to randomly select additional

---

91.2
baselize
baselize block
74-
MAP (%)
73
block
71.9
13.5
72.
PCKb (%6
9008
72.
71.4
12.0
90.6
90
71.0
90.3
90.5
50
70.3
90.2
90.0
20
70
00
100
scaling ratio (%6
scaling ratio (%)
(a) Scaling ratio analysis on (left) MSMTI7 and (right) MPII. A scaling ratio with 60%-600%%9999999 90% works well.
7236
76.0
257
22
255
720
20.4 904
DAP (%)
63.8
66
64.7
PCKh (16
89.5
50
55
baseline
baselino
50
block
55
block
45
56
10
30
50
masking
masking ratio (%6)
(b) Masking ratio analysis on MSMT17 (left) and MPII (right). masking ratio with 40%-66% works well.
Figure Study on training strategy. We investigate three factors of training strategy, i.e. (a) scaling ratio, (b) masking ratio, and (a.b) mask sampling strategy, on MSMT17 (74) (person ReID) and MPII [ (2D pose estimation). The baseline MAE (30) with default settings (20% scaling ratio, 75% masking ratio, random mask sampling strategy). The optimal training strategy for human-centric pre-training with 80% scaling ratio, 50% masking ratio, and block-wise mask sampling strategy.
Nm - N, patches; iii) if Nº Nn, we delete patches according the sequence of human part selection. We maintain total of Nm patches, which contain structure information, to be masked. Discussion. i) We use human parts instead of individual keypoints as guidance for mask sampling. This is based on the experimental result in Section 3.1 i.e., the block-wise mask sampling strat- egy outperforms the random one. More experimental comparisons are available in Appendix. ii) HCMoCo 33 also utilizes keypoints during human-centria pre-training, in which sparse keypoints serve one of the multi-mode mollitmod inputs. Different from HCMoCo, we integrate two modalities. including images and keypoints, together for mask sampling. It should be noted that the keypoints in our method are pseudo labels; therefore they may not be accurate enough for supervised learning as in HCMoCo. However, these generated keypoints can be beneficial for the self-supervised learning.
3.3 Structure-invariant alignment
Learning discriminative human characteristics is essential human-centric perception tasks. For example, person RelD) requires to distinguish different pedestrians based on subtle details and identify the same person despite large variances. To address this issue, we further propose a structure- umucture-invariant alignment loss, to enhance the representation ability of human structural semantic information.
For each person image x, generate two different views through the proposed mask sampling strategy with guidance human part prior. These two views, denoted as and X, display different human part regions for the same person. After model encoding, each view obtains respective latent feature representations Z and Z on the normalized [CLS] tokens. Our structure- atruture-invariant alignment loss Latign is applied and Z for aligning them. We achieve by using the InfoNCE loss (54)
exp Z ZZTT
Lalign
loog
Di=1 exp Z,/T)
where the batch size, T a temperature hyper-parameter set to 0.2 12.. In this way, the encoder encouraged to learn the invariance of human structure, and its discriminative ability is improved.
5

---

LEL
Figure 5: Reconstructions of LUPerson (Ist row), COCO (2nd row), and AIC (3rd row) images using our HAP pre-trained model. The pre-training dataset is LUPerson. For each triplet, we show (left) the original image, (middle) the masked image, and (right) images reconstructed by HAP. We highlight some interesting observations using red dotted ellipses. Although the reconstructed parts are not exactly identical to the original images, they are semantically reasonable as body parts. For example, the first triplet of the 1st row, the reconstructed right leg of the player differs from the original, but still represents sensible leg. It clearly shows that HAP effectively learns the semanties of human body structure and generalizes well across various domains. Zoom in for a detailed view.
3.4 Overall: HAP
To sum up, in order to learn the fine mne-grained semantic structure information, our HAP incorporates the human part prior into the human-centric pre-tran pre-training, and introduces structure- -nnaturr--ivariant alignment loss to capture discriminative human characteristics. The whole loss function to be optimized is:
C = Crecon Calign:
where Lrecon is the MSE loss for reconstruction following (30]. is the weight to balance the two loss functions of Lrecon and Lalign. We set y to 0.05 by default, and leave its discussion in Appendix.
4 Experiments
4.1 Settings
Pre-training. LUPerSon 23 is a large-scale person dataset, consisting of about 4.2M images of over 200K persons across different environments. Following [5]], we use the subset of LUPerson with 2.1M images for pre-training. The resolution of the input image is set to 256 x 128 and the batch size is set to 4096. The human keypoints of LUPerson,, i.e., the human structure prior of HAP, are extracted by ViTPose (78). The encoder model structure of HAP is based on the ViT-Base [21]. HAP adopts Adam [49 as the optimizer in which the weight decay is set to 0.05. We use cosine decay learning rate schedule [48), and the base learning rate is set to 1.5c-4. The warmup epochs are set to and the total epochs are set to 400. The model of our HAP initialized from the official MAE/CAE model pre-trained on ImageNet by following the previous works [T6] 78). After pre-trate pre-training, the encoder is reserved, along with task-specific heads, for addressing the following human-centrie perception tasks, while other modules (e.g., regressor and decoder) discarded.
Human-centria perception tasks. We evaluate our HAP on 12 benchmarks across human--cntric perception tasks, including person RelD) on Market-1500 87 and MSMT17 (74), 2D pose estimation on MPII COCO (46) and AIC 75), text-to-image person RelD on CUHK-PEDES 42). ICFG-
PEDES 20 and RSTPReid
[89 3D pose and shape estimation 3DPW 667, pedestrian attribute

---

Table Main results. We compare HAP with representative task specific methods, numan-centrre
pre-traie pre-training methods, and the baseline MAE 30) indicates using stronger model structures such as
sliding windows
32 and convolutions in [50] HAP* follows the implementation of [50]
(a) Statistics the number of training datasets and training data samples used during the pre-training pro-
(b) Attribute recognition. mA (%) is reported.
cess of the human-centric pre-training methods.
method
PA-100K
RAP
PETA
ALM
80.68
81.87
86.30
method
datasets
samples
DAFL
35
83.54
81.04
87.07
LiftedCL 13
1 1550
RethinkPAR
36
81.61
80.82
85.17
SOLIDER
1
4.2M
Label2Label
46
82.24
HCMoCo
2
822
PATH [64]
85.0
81.2
88.0
UniHCP 16
33
~2.3
UniHCP
[66
86.18
82.34
PATH 64
37
111.0M
SOLIDER
86.37
HAP
~2.1
MAE 30
79.56
75.73
80.82
HAP
86.54
82,91
88.36
(c) Person ReID mAP(%) reported. For compar- isons, we present all the results according to the input image resolution of 256 x 128 (256) and 384 x 128 (384) respectively for each compared method.
(d) 2D pose estimation. PCKh (%) is reported for MPII and AP (%)) is reported for COCO and AIC. represents
using multi dataset training of COCO+AIC+MPI (78)
represents using larger image size of 384 288.
method
MSMT17
Market-
method
MPII
COCO
AIC
resolution
256
384
256
384
HRNet-w448
90.1
75.1
33.5
TransReID
32
64.9
66.6
88.2
88.8
HRFormer
82
75.6
34.4
TransRelD
32
67.4
69.4
88.9
89.5
HRFormer*
77.2
UP-ReID
81
63.3
91.1
ViTPuse 78
75.8
LUP
ViT'ese
78
93.3
77.1
32.0
PASS 90
71.8
74.3
93.0
93.3
LiftedCL
89.3
71.1
MALE 50
73.0
74.2
92.2
92.1
PATH 64
93.3
76.3
35.0
TransRelD-SSL.
51
75.0
93.2
UniHCP 16
76.5
33.6
PATH 64
69.1
89.5
SOLIDER
76.6
UniHCP
16
67.3
90.3
MAE
89.6
75.7
31.3
SOLIDER
77.1
93.9
HAP
91.8
75.9
31.5
MAE
30
62.0
62.9
82.9
82.5
HAP
93.4
77.0
32.2
HAP
76.4
76.8
91.7
91.9
HAP+
92.6
77.2
37.7
HAPl
78.0
93.8
HAP*+
93.6
78.2
38.1
(c) Text-to-image RelD. Rank- ((%) is reported.
(f) 3D pose and shape estimation. MPJPE, PA-
MPJPE, and MPVPE are reported on 3DPW.
method
CUHK- PEDES ICFG-PEDES RSTPReid
SAFA 41
64.13
method
MPJPE
PA-MPIPE MPVPE 1
LBUI. 71
61.95
43.35
MiHu 22
85.1
54.8
CAIBC 70
64.43
47.35
SPIN 38
96.9
59.2
116.4
SSAN
61.37
54,23
Pose2Mesh
14
89.5
56.3
105.3
SRCF 62
64.04
57.18
12L-MeshNet
52
57.7
110.1
LGUR
65.25
59.02
3DCrowNet
15
81.7
$1.5
98.3
MAE 30
60.19
53.68
44.15
MAE
30
95.6
58.0
112.7
HAP
68.05
61.80
49.35
HAP
90.1
56.0
106.3
RAP 39 and PETA 18 For person ReID, we use the implementation of [50] and report mAP. For 2D pose estimation, the evaluation metrics are PCKh for MPII and AP for COCO and AIC. The codebase is based on 78 Rank-1 is reported for text-to-image person ReID with the implementation of [59]. mA reported in the pedestrian attribute recognition task with the codebase of [36]. The evaluation metrics are MPJPE/PA-MPIPP/MPYN
recognition on PA-100K 47
for 3D pose and shape estimation. Since there is no implementation with a plain VIT structure available for this task, [15]. More training details can be found in Appendix.
modify the implementation based
7

---

Table HAP with MAE 30
and CAE
9
The evaluation metrics are the same as those in Table
MSMT17
Market
1501
PA-100K
MPII
COCO
RSTPReid
256
384
256
384
MAE 30
76.4
76.8
91.7
91.9
86.54
91.8
75.9
49.35
CAE 99
76.5
77.0
93.3
93.1
86.33
91.8
75.3
51.70
4.2
Main results
We compare HAP with both previous task-specific methods and human-centric pre-training methods. Our HAP is simple by using only one dataset of ~2. samples, compared with existing pre-training methods (Tabldla), yet achieves superior performance on various human-centrie perception tasks.
In Table T we report the results of HAP with MAE (3 on 12 datasets across 5 representative human-centric tasks. Compared with previous task-specific methods and human-centric pre-training methods, HAP achieves wate-of-the-art results on 11 datasets of tasks. We discuss each task next.
Pedestrian attribute recognition. The goal of pedestrian attribute recognition is assign fine- grained attributes to pedestrians, such as young or old, long or short hair, etc. Table shows that HAP surpasses previous state-of-the-ard
all of the three commonly used datasets, i.e., +0. 17% on PA-100K [47.. +0.57% on RAP [39], and +0.36% on PETA [8] It can be observed that our proposed HAP effectively accomplishes this fine-grained human characteristic understanding tasks.
Person ReID. The aim of person ReID is to retrieve a queried person across different cameras. Table Ic shows that HAP outperforms current state-of-the of-the-rr results of MSMT17 (74] by +5.0% (HAP 78.0% vs. MALE [50] 73.0%) and +1.0% (HAP 78. I 1% vs. SOLIDER (8) 77.1%) under the input resolution of 256 128 and 384 128, respectively, HAP also achieves competitive results on Market-1501 87]. i.e., 93.9% mAP with 384 x 128 input size. For fair comparison, we also report the results of HAP with extra convolution modules following [50], denoted as HAP!, It brings further improvement by +1.3% and +2.0% on MSMT17 and Market-1501 with resolution of 384. Review that previous ReID pre-training methods [8] 23 90 51 usually adopt contrastive learning approach. These results show that our HAP with MIM is more effective on the person ReID task.
Text-to-image person ReID. This task uses textual descriptions to search person images of interest. We use the pre-trained model of HAP as the initialization of the vision encoder in this task. Table shows that HAP performs the best on the three popular text-to-image benchmarks. That is, HAP achieves 68.05% 61.80% and 49.35% Rank- 1 CUHK-PEDES 142,, ICFG-PEDES (20] and RSTPReid [89]. respectively. Moreover, HAP is largely superior than MAE [30], e.g., 68.05% vs. CUHK-PEDES It clearly reveals that human structure priors especially human keypoints and mnnucture-ivvarial alignment indeed provide semantically rich human information, performing well for human identification and generalizing across different domains and modalities.
60.19% (+7.80 86%)
2D pose estimation. This task requires to localize anatomical keypoints of persons by understanding their structure and movement. For fair comparisons, we also use multiple datasets in (78 and larger image size as in 82, g during model training. Table Td shows that HAP outperforms previous of-te-tf results by +0.3%, +1.0%, +3.1% 1% MPII [I, COCO 46). AIC [75). respectively. Meanwhile, we find both multi-dataset training and large resolution are beneficial for our HAP.
3D pose and shape estimation. We then lift HAP perform 3D pose and shape estimation by recovering human mesh. Considering that there is no existing method that performs this task on plain ViT backbone, we thus follow the implementation of [15] and just replace the backbone to a plain ViT. The results are listed in Table IT It is interesting to find that despite this simple implementation, our HAP achieves competitive results for the 3D pose task even HAP does not see any 3D information. These observation may suggest that HAP improves representation learning ability on human structure from the 2D keypoints information, and transfers its ability from 2D 3D.
CAE 9 vs MAE [30]. HAP is model-agnostic that can be integrated into different kinds of MIM methods. We also apply HAP to the CAE framework [9] for human-centie shows that HAP with CAE [9) achieves competitive performance on various human- centric perception tasks, especially on discriminative tasks such as person ReID that HAP with CAE is superior than HAP with MAE 30 by 1.6% on Market-1500 (with the input resolution of 256 x 1 128), and text-to- image person RelD that HAP with CAE performs better than HAP with MAE by 2.35% on RSTPReid. These observations demonstrate the versatility and scalability of our proposed HAP framework.
pre-training. Table 2
8

---

Table 3: Main properties of HAP. "HSP". "HPM", and "SIA" stand for human structure priors, human part prior for mask sampling, and structure invariant alignment, respectively. We compare HAP with the baseline of MAE ImageNet pretrained model on the representative benchmarks.
method
HSPP
HPM
SIA
MSMT17
MPII
CUIKK-EDESS
3DPW
PA-100K
baseline
68.9
90.3
66.49
57.4
83.41
HAP-0
74.1
90.9
67.69
56.2
85.50
HAP-1
75.1
91.1
67.95
56.2
85.83
HAP-2
75.6
91.6
68.00
56.1
86.38
HAP
76,4
91.8
68.05
56.0
86.54
MAE 30
62.0
89.6
60.19
58.0
79.56
Table 4: Ablation on (left) number of samples and (right) training epochs for pre-training HAP.
samples
MSMT17
MPII
epochs
MSMT17
MPI1
~0.5
66.9
90.4
72.2
91.3
1.0M
71.9
91.2
200
73.9
91.6
2.1M
76.4
91.8
400
76.4
91.8
4.3 Main properties
ablate our HAP in Table The analyses main properties are then discussed in the following.
Pre-training human-centrie data. Our baseline, using default settings of [30] and LUPerson (33 as pre-training data, has achieved great improvement than original MAE [30], e.g., +6.9% MAP It shows that pre-training on human-centric
on MSMT17 and +6.30% Rank-1 I on CUHK- data benefits a lot for human-centric perception by effectively bridging the domain gap between the pre-training dataset and downstream datasets. This observation is in line with previous works 23
Human structure priors. denote HAP-O to represent human structure priors, i.e., large scaling ratio and block-wise masking with intermediate masking ratio, introduced in Section 3.1 HAP-0
performs better than the baseline method, especially on MSMT17 with +5.2% improvement. clearly reveals that human structure priors can help the model learn more human body information.
Human part prior. We further use human part prior guide the mask sampling process. Compared with HAP-1 and HAP-0, part-guided mask sampling strategy brings large improvement, e.g., +1.0% on MSMT17 and +0.33% on PA-100K. It clearly shows that the proposed mask sampling strategy with the guidance of human part prior has positive impact on the human- perception tasks.
diructure-invarian alignment. The structure- invariint alignment loss is applied on different masked views to align them together. Compared with HAP-2 vs. HAP-0 and HAP vs. HAP-1, this alignment loss results in +1.5% and +1.3% improvement MPII, respectively. reflects that the model successfully captures human characteristics with this loss. We provide more discussions about the alignment choices as well as the weight Appendix.
MSMT17, and +0.7% and +0.7% improvement
Overall, our HAP greatly outperforms the original MAE [30] and the baseline by large margins across various human-centric perception tasks, showing significant benefits of the designs of our HAP.
4.4 Ablation study
Number samples. We compare using different number of data samples during pre-training in (left. Only using ~0.5M LUPerson data already outperforms original MAE (30] pre-trained on ImageNet [17]. Meanwhile, more pre-training data brings larger improvement for our HAP.
Training epochs. Table (right) shows that pre-training only 100 epochs already yields satisfactory performance. suggests that HAP can effectively reduces training cost. These observations indicate that HAP, as a human-centric pre-training method, effectively reduces annotation, data, and training cost. Therefore, we believe HAP can be useful for real-world human-centrie perception applications.
Visualization. To explore the structure information learned in the plain VIT encoder of HAP, probe attention map by querying the patch of random body keypoint. See visualizations in Figure Interestingly, we observe that HAP captures the relation among body parts. For example,

---

Figure 6: Visualization on attention maps of body structure from the VIT encoder in HAP. For each pair, we show (left) a randomly selected keypoint within a patch marked red and (right) its corresponding attention weight to other patches learned by HAP. Patches in deeper color indicate more relevant to the selected keypoint region. The left two pairs are from LUPerson [33, the middle two are from COCO [46), and the right two are from AIC (75). It reveals that HAP is able to learn human body structure information and successfully identifies the correlations among the body parts.
when querying the right hip of the player in the first image pair (from LUPerson), are strong relations to the left leg, left elbow, and head. And when querying the left knee of the skateboarder in the second image pair (from COCO), it is associated with almost all other keypoints. These observations intuitively illustrate that HAP learns human body structure and generalizes well.
that there
5 Conclusion
This paper makes the first attempt to use Masked Image Modeling (MIM) approach for human-centric pre-training, and presents novel method named HAP. Upon revisiting several training strategies, HAP reveals that human structure priors have great benefits. We then introduces human part prior as the guidance of mask sampling and a -ruatire-invariant alignment loss align two random masked views together. Experiments show that our HAP achieves competitive performance on various human-centric perception tasks. Our HAP framework simple by unifying the learning of two modalities within single dataset. HAP is also versatility and scalability by being applied to different MIM structures, modalities, domains, and tasks. We hope our work can be strong baseline with generative pre-training method for the future researches on this topic. We also hope our experience in this paper can be useful for the general human-centris perception and push this frontier.
Broader impacts. The proposed method is pre-trained and evaluated on several person datasets, thus could reflect biases in these datasets, including those with negative impacts. When using the proposed method and the generated images in this paper, one should pay careful attention to dataset selection and pre-processing to ensure a diverse and representative sample. To avoid unfair discriminatory outcomes, please analyze the model's behavior with respect to different demographics and consider tainess-awaare loss functions or post-hoc bias mitigation techniques. We strongly advocate for responsible use, and encourage the development of useful tools to detect and mitigate the potential misuse of our proposed method and other masked image modeling techniques for unethical purposes.
Limitations. We use human part prior to guide pre-training, which is extracted by existing pose estimation methods. Therefore, the accuracy of these methods could affect the pre-training quality. Besides, due to the limitations in patch size, the masking of body parts may not be precise enough. If there better ways improve this, it should further enhance the results. Nonetheless, the proposed HAP method still shows promising performance for a wide range of human-centrie perception tasks.
Acknowledgments and Disclosure of Funding
We thank Shixiang Tang and Weihua Chen for discussions about the experiments of human-centric pre-training and Yufei Xu and Yanzuo Lu for discussions about downstream tasks. This work was supported in part by Zhejiang Provincial Natural Science Foundation of China (LZ2220000122) National Natural Science Foundation of China (62376243, 62037001, U20A20387), Young Elite Sci- the StarryNight Science Fund of Zhejiang entists Sponsorship Program by CAST (2021 PEZJU--SAS-OO University Shanghai Institute for Advanced Study
Project by Shanghai Al Laboratory (P22KS001 I). Program of Zhejiang Province Science and Technology (2022C01044). the Fundamental Research Funds for the Central Universities (226-0022 2022-0012, 226-2022-00051).
10

---

References
[1] M. Andriluka, L. Pishchulin, Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686-3693, 2014,
[2] H. Bao, Dong, S. Piao, and Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv22006008255 2021.
[3] Z. Cao, Simon, S.-E. Wei, and Y. Sheikh, Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291 19799, 2017.
[4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[5] C. Chen, M. Ye, M. Qi, J. Wu, J. Jiang, and C.-W. Lin. Structure-aware positional transformer for visible-infrared person re-identiifcation. IEEE Transactions on Image Processing, 31:2352-2364, 2022.
[6] H. Chen, W. Zhang, Y. Wang, and X. Yang. Improving masked autocncoders by learning where to mask. arXiv preprint arXiv:2303.088 06583, 2023.
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMIRR 2020.
[8] W. Chen, X. Xu, J. Jia, H. Luo, Y. Wang, F. Wang, R. Jin, and X. Sun. Beyond appearance: semantic controllable self- supervised learning framework human-centric visual tasks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[9] X. Chen, M. Ding, X. Wang. Xin, Mo. Wang. S. Han, P. Luo, Zeng. and Wang. Context autoencoder for self-supervised representation learning, arXiv preprint orXiv:2202.03022 2022.
[10] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint Xxxv:2003.04297 2020.
[11] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[12] X. Chen, S. Xie, and K. He. An empirical study of training supervuee vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[13] Z. Chen, Q Li, X. Wang, and W. Yang. Liftedel: Lifting contrastive learning for human-centric perception. In The Eleventh International Conference on Learning Representations, 2023.
[14] H. Choi, G. Moon, and K. M. Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from 2d human pose. In Computer Esion-ECCT 2020: 16th European Conference, Glasgow, UK, August 23 28-28, 2020, Proceedings, Part VII 16, pages 769-787. Springer, 2020.
[15] H. Choi, G. Moon, J. Park, and K. M. Lee. Learning to estimate robust 3d human mesh from in-the-wild crowded scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1475 1484, 2022.
[16] Y. Ci, Y. Wang. M. Chen, Tang. L. Bai, F. Zhu, R. Zhao, F. Yu, D. Qi, and W. Ouyang. Unihep: unified model for human-centric perceptions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[17] Deng, W. Dong, R. Socher, L..J. Li, Li, and I., Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[18] Y. Deng, P. Luo, C. C. the 22nd ACM international conference on Multimedia, pages 789-792, 2014.
C.C.Loy Loy, and X. Tang. Pedestrian attribute recogmition at far distance. In Proceedings of
[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018
[20] Z. Ding, C. Ding, Z. Shao, and D. Tao. Semantically self-aligned network for text-to-image part-aware person re-identification. arXiv preprint arXiv:210711266, 2021.

---

[21] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
[22] Q. Fang, Q. Shuai, J. Dong, H. Bao, and X. Zhou. Reconstructing 3d human pose by watching humans in the mirror. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2021.
[23] D. Fu, D. Chen, J. Bao, H. Yang, L. Yuan, L. Zhang, H. Li, and D. Chen. Unsupervised pre-training for person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and puttern recogmition, pages 14750-14759, 2021.
[24] D. Fu, D. Chen, H. Yang. Bao, Yuan, L. Zhang, H. Li, F. Wen, and D. Chen. Large-scale pre-training for person re-identification with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2476-2486 2486, 2022.
[25] S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik. Humans in 4d: Reconstructing and tracking humans with transformers. arXiv preprint arXiv:2305.cm 230.::335020091, 2023.
[26] K. Gong, X. Liang, D. Zhang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark human parsing. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 932-940, 2017.
[27] A. Graves and A. Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012. 2012.
[28] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires,
Z. Guo, M. Gheshlaghi Azar, al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[29] Gui, Z. Liu, and H. Luo. Good helper is around you: Attention-driven masked image modeling. arXiv preprint Xrx::2221 153662, 2022.
[30] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[31] K. He, Fan, Wu, S. Xic, and Girshick. Momentum contrast for unsupervised visual represcntation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[32] S. He, H. Luo, P. Wang. F. Wang, H. Li, and W. Jiang. Transreid: Transformer- object re- identification. In Proceedings of the IEEF/CVF international conference on computer vision, pages 15013-15022, 2021.
[33] F. Hong, L. Pan, Z. Cai. and Z. Liu. Versatile multi-modal pre-training for human-centric perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16156- 16166, 2022.
[34] J. Jia, X. Chen, and K. Huang. Spatial and semantic consistency regularizations for pedestrian attribute recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 962 -911, 2021.
[35] J. Jia, N. Gao, F. He, X. Chen, and K. Huang. Learning disentangled attribute representations for robust pedestrian attribute recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1069-1077, 2022.
[36] J. Jia, H. Huang. X. Chen, and K. Huang. Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting. arXiv preprint trXiv:2107.03576 2021.
[37] I. Kakogeorgiou. S. Gidaris, B. Psomas. Y. Avrithis, A. Bursuc, K. Karantzalos. and N. Komodakis. What to hide from your students: Attention-guided masked image modeling. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 23-27, 2022, Proceedings, Part XXX, pages 300-318. Springer, 2022.
[38] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis, Learning to reconstruct 3d human pose and shape via model fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2252-2261, 2019.
12

---

[39] D. Li. Z. Zhang, X. Chen, and K. Huang. A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE transactions image processing, 28(4):1575-1390, 2018.
[40] G. Li, H. Zheng. D. Liu, C. Wang, B. Su, and C. Zheng. Semmae: conantic-guider masking for learning masked autoencoders. arXiv preprint (XXi:2226..100) 2022.
[41] S. Li, M. Cao, and M. Zhang. Learning semantic-aligned feature representation for text-based person search. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 22228, IEEE, 2022.
[42] S. Li. T. Xiao, H. Li, B. Zhou, D. Yue. and X. Wang. Person search with natural language description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1970-1979, 2017.
[43] W. Li, Z. Cao. Feng, J. Zhou, and J. Lu. LabelZlabel: A language modeling framework for attribet learning. Computer ECCin 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pages 562-579. Springer, 2022.
[44] Y. Li, H. Mao, Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 280-296. Springer, 2022.
[45] Z. Li, Z. Chen, F. Yang, W. Li, Y. Zhu, C. Zhao, R. Deng, L. Wu, R. Zhao, M. Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34: 14:13165-13176 2021.
[46] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. I.. Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Mision-ECCV'
[47] X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, and X. Wang. Hydraplus-net: Attentive deep features for pedestrian analysis. In Proceedings of the IEEE international conference computer vision, pages 350-359, 2017.
[48] I. Loshehilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[49] 1. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR. 2019.
[50] Y. Lu, M. Zhang, Y. Lin, A. J. Ma, X. Xie, and J. Lai. Improving pre-trained masked autoencoder via locality enhancement for person re-identification. In Pattern Recognition and Computer Vision: 5th Chinese Conference, PRCV 2022, Shenzhen, China, November 4 7, 2022, Proceedings, Part II, pages 509 521. Springer, 2022.
[51] H. Luo, P. Wang, Y. Xu, F. Ding, Y. Zhou, F. Wang, H. Li, and R. Jin. Self-supervised pre-training for wransformer-bseed person re-identificatio. arXiv preprint arX1v22111./2084, 2021.
[52] G. Moon and K. M. Lee. 12l-meshnet: Image-to-lixe lixel prediction network for accurate 3d human pose and mesh estimation from a single rgh image. In Computer Vision-FCCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 752-768. Springer, 2020.
[53] G. Moon and K. Lec. Neuralannot: Neural annotator for in-the-wild expressive 3d human pose and mesh training sets, arXiv prepriat arXiv:201 1.1222 5, 2020.
[54] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 107.:180.003744 2018
[55] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. Osman, D. Tzionas, and M. Black. Expressive body capture: 3d hands, face, and body from a single image. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10984 2019.
[56] Z. Peng, L. Dong, Bao, Q. Ye, and Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.0633 2022.
[57] R. Quan, X. Dong, Wu, L. Zhu, and Y. Yang. Auto-reid: Searching a part-aware convnet for person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3750-3759, 2019.
[58] Z. Shao, X. Zhang, C. Ding, J. Wang, and J. Wang. Unified pre-training with pseudo texts for text-to-imme -mage person re- iddeiiiiccaiinnn...mmmm In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11174- 11184, 2023.
13

---

[59] Z. Shao, X. Zhang, M. Fang, Z. Lin. J. Wang, and C. Ding. Learning granularity for text-to-image person re-identification. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5566-5574, 2022.
crauulrity-unifiedd unified representations
[60] C. Su, J. Li. S. Zhang, J. Xing. W. Gao, and Q. Tian. Pose-driven deep convolutional model for person re- identification. In Proceedings of the IEFE international conference on computer vision, pages 3960-3969, 2017.
[61] K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693-5703, 2019.
[62] W. Suo, M. Sun, K. NIu, Y. Gao, P. Wang. Zhang, and Q. Wu. A simple and robust correlation filtering method for text-based person search. In Computer Vision-ECCV 2022: 17th Luropean Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings. Part XXXV, pages 226-742. Springer, 2022.
[63] C. Tang, L. Sheng, Z. Zhang, and X. Hu. Improving pedestrian attribute recognition with supery-s multi-scale atrribate-speciffc localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4997-5006, 2019.
[64] S. Tang, C. Chen, Q Xie, M. Chen, Y. Wang, Y. Ci, I. Bai, F. Zhu, H. Yang, I.. Yi, et al. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEFF/CVF Conference Computer Vision and Puttern Recognition, 2023.
[65] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. 12602, 2022.
[66] A. Vaswani. N. Shazeer. N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[67] T. Von Marcard, R. Henschel, M. J. Black, B. Rosenbahn, and G. Pons-Moi Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[68] H. Wang. K. Song. J. Fan, Y. Wang. J. Xie, and Zhang. Hard patches mining for masked image modeling. arXiv preprint arxiv:2304.059119 2023.
[69] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[70] Z. Wang. A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li. Caibe: Capturing all-round information beyond color for text-based person retrieval, In Proceedings of the 30th ACM International Conference on Multimedia, pages 5314-5322, 2022.
[71] Z. Wang. A. Zhu, J. Xue, Wan, Liu, T. Wang, and Y. Li. Look before you leap: Improving text-based person retrieval by learning a consistent cross- common manifold. Proceedings of the 30th ACM International Conference on Multimedia, pages 1984-1992, 2022.
[72] C. Wei, H. Fan, Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self- supervised visual pre-training. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 14668 14678, 2022.
[73] L. Wei, L. Xie, W. Zhou, H. Li, and Q. Tian. Mvp: Multimodality-guiee visual pre-training. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXX, pages 337-353. Springer, 2022.
[74] L. Wei, S. Zhang, W. Gao, and Q Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 79-88, 2018.
[75] J. Wu, H. Zheng, B. Zhao, Y. Li, B. Yan, R. Liang, W. Wang, S. Zhou, G. Lin, Y. Fu, et al. Ai challenger: A large-scale dataset for going deeper in image understanding. arXiv preprint arXiv: 1ete10117...7 2017.
[76] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In Proceedings of
the European conference on computer vision (ECCV), pages 466-481, 2018
[77] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653 00333966,,200 2022.
14

---

[78] Y. Xu. J. Zhang, Q Zhang. and D. Tao. Vitpose: Simple vision transformer baselines for human pose estimation, arXiv preprint arXiv:2204. 12484, 2022.
[79] S. Yang, 7. Quan, M. Nie, and W. Yang. Transpose: Keypoint localization via transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 1802-11812, 2021.
[80] Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transactions on pattern analysis and machine intelligence, 35 (51222287802280 2012.
[81] 7. Yang, X. Jin, K. Zheng, and F. Zhao. Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 103077 2022.
[82] Y. Yuan, R. Fu, L.. Huang. W. Lin, C. Zhang. X. Chen, and Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint arXiv:21 2021.
[83] S. Zhang, F. Zhu, Zhao, and J. Yan. Contextual image masking modeling via synergized contrasting without view augmentation faster and better visual pretraining, The Eleventh International Conference on Learning Representations, 2023.
[84] X. Zhang, Cao, C. Shen, and M. You. Self-training with progressive augmentation for unsupervised cross-domain person re-identification. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 2222-8231, 2019.
[85] X. Zhang, J. Chen, J. Yuan, Q. Chen, J. Wang, X. Wang. S. Han, X. Chen, J. K. Yao, J. Han, E. Ding, and
J. Wang. CAE v2: Context autoencoder with CLIP latent alignment. Transactions on Machine Learning Research, 2023.
[86] X. Zhang, D. Li, Z. Wang, J. Wang, E. Ding, J. Q. Shi, Z. Zhang, and J. Wang. Implicit sample extension for unsupervised person re-identification. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7369-7378 2022.
[87] L. Zheng, L. Shen, L. Tian, S. Wang. J. Wang. and Q. Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pages 1116-1124. 2015.
[88] J. Zhou, C. Wei, H. Wang. W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre rre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
[89] A. Zhu, Z. Wang. Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua. Dssl: deep surroundings-person sepa- ration learning for text-based person retrieval, In Proceedings of the 29th ACM International Conference on Multimedia, pages 209-217,20 2021.
[90] K. Zhu, H. Guo, Yan, Y. Zhu, Wang, and M. Tang. Pass: Part-aware sepe-supe pre-training for person re-identification. In Computer Vision ECCV 2022: 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, pages 198-214, Springer, 2022.
[91] K. Zhu, H. Guo, S. Zhang, Y. Wang. G. Huang, H. Qiao, J. Liu, J. Wang, and M. Tang. Aaformer: Auto-aligned transformer for person re-identification. arXiv preprint arXiv:2104.0991,, 2021.
15

---

A Details of pre-training
the basic structure of HAP. It consists of an encoder and lightweight decoder. The encoder follows the standard ViT-base [21] architecture, comprising 12 common Transformer blocks [66] with 12 attention heads and hidden size of 768. The decoder is relatively small, which consists of 8 Transformer blocks with 16 attention heads and hidden size of 512. Sine-cosine position embeddings are added to representations of both encoder and decoder We also use CAE-base 9 as the basic structure. Please refer to 9 for more details of the structure.
Model structure. We adopt MAE-base [30
Training data. LUPerson 23 is large-scale person dataset with ~.22M images (see Figure Sa). Following [5]], we use subset of LUPerson with ~2.1 images for pre-training. The resolution of the input images is to 256 128 following previous works [23] 90 50 The data augmentation follows [50]. which involves random resizing and cropping, and horizontal flipping. We utilize ViTPose 78). popular 2D pose estimation method, extract the keypoints of LUPerson as the human structure prior in our HAP. Given that our HAP does not rely on ground-truth labels, it can be pre-trained on combination of several acage-scale person datasets after extracting their keypoints. We speculate that employing multi multi-dataset co- training can further improve the performance of HAP, as observed in 78.. We also recognize that our HAP would derive great benefits from more accurate keypoints (e.g., human-annotated keypoints) if available. We will discuss this point in details later.
Table 5: Hyper-parameters of pre-training. HAP follows most of the settings of MAE and CAE,
dataset
method
batch size epochs warmup epochs base learning rate optimizer weight decay
LUPerson MAE
4096
400
40
1.5e-4
AdamW AdamW 0.05
0.05
LUPerson
CAE
30
2048
400
10
1.5e-4
Implementations details. We follow most of the training settings of MAE/CAE except using training epochs of 400 (T. Different from MAE/CAE, HAP has a large scaling ratio of intermediate masking ratio of 50% with block-wise masking strategy, as introduced in the main paper.
80%, and an
B Details of human-centric perception tasks
To We list the training details of human-centric perception tasks in Table of the main paper as follows.
B.1
Person RelD
Table 6: Hyper-parameters of person ReID.
dataset
batch size
epochs
learning rate
optimizer
weight decay
layer decay
drop path
MSMT17
64
100
8c-3
AdamW Adam
0.05 0.05
0.4
0.1
Market- 64
100
8c-3
0.4
0.1
We consider two RelD datasets: MSMT17 74 (Figure Kbp and Market-1500 87 (Figure 8c with
the codebase of [50]. The input image resolutions are set to 256 x 128 and 384 x 128. Data augmentations are the same as [50] including resizing, random flipping, padding, and random cropping. Following [32 64, 8 [6], the model is fine-tuned using a cross-ents cross-entropy loss and a triplet
loss with equal weights of 0.5. We also consider using a convolution- module (50) to further improve results. The evaluation metric of mAP (%) is reported. See hyper-parameter in Table 6
B.2
2D human pose estimation.
HAP evaluated on MPII (1 (Figure 8dh, MS COCO 46 (Figure Seh, and AI Challenger (AIC) 75 (Figure 8f. We use the codebase of 78) Resolutions of 256 x 192 and 384 x 288 are employed.
IIn this paper, we conduct all experiments on base-sized model to verify the effectiveness of our method. In fact, all scales of models compatible with HAP. 'haps://github.com com/ Yan//// LuMALL a/MALE
https://gituub..om/iT.. www.t//aaattATteeee/e//e//eeaeee TiaPeoe
16

---

Table 7: Hyper- hopee-paaaaeterss of 2D puse estimation. t/*: multi-ddtaset training/384 x 288 resolution.
datasct
batch size epochs learning rate optimizer weight decay layer decay drop path
MPII/MPII*
512
210
1.0e-4
Adam
0.30
COCO/COCO
512
210
2.0c-4
AdamW
0.1
0.80
0.29
AIC/AIC+
512
210
5.0c-4
Adam
0,30
MPIT/CCCOO/AICT
1024
210
1.1c-3
AdamW
0.1
0.77
0.30
MPII*//CCOO//ACTT
512
210
5.0e-4
AdamW
0.1
0.75
0.30
Data augmentations are random horizontal flipping, half body transformation, and random scaling and rotation. We follow the common top-down setting, i.e., estimating keypoints of the instances detected [76]. We mean square error (MSE) [78) the loss function to minimize the difference between predicted and ground-trun truth heatmaps. Following [78], we also perform multi-dataset training setting by co-training MPII, MSCOCO, and AIC. Evaluation metric of PCKH (%) is reported for
MPII, and AP (%) is reported for MS COCO and AIC. See the details of hyper-parameters in Table
B.3 Texa-to-image person ReID
Table 8: Hyper-parameters of text-to-image person ReID.
dataset
batch size
epochs
learning
optimizer
JUHK-PEDES
64
60
le-3
Adam
ICFG-PEDES
64
60
le-3
Adam
RSTPReid
64
60
le-3
Adam
We emplov three commonly used datasets, i.e., CUHK- 42 (Figure Kg ICFG-PEDES 20
(Figure 8h), RSTPReid [89] (Figure 8i). The codebase 8 of [59 is used. The resolution is 384 x 128. We adopt resizing and random horizontal flipping data augmentations. BERT [19] is used to extract embeddings of texts, which are then fed to Bi-LSTM [27]. Feature dimensions of images and texts both 384. The BERT is frozen while Bi-LSTM and ViT are fine-tuned. We report Rank-1 (%) metric for the results on all datasets. The details of the hyper-parameters can be found in Table
B.4 3D pose and shape estimation
Table 9: Hyper-parameters of 3D pose and shape estimation.
dataset
batch size
epochs
learning
optimizer
3DPW
192
lc-4
SGD
8j using the codebase of [15 The resolution is 256 x 256. Data augmentations follow [5] We minimize pose loss, i.e., I distance between the predicted 3D joint coordinates and the pseudo ground-truth, and mesh loss, iee, loss for predicted SMPL parameters [53] 55]. We report MPJPE, PA-MPJPE, MPVPE. The former for 3D pose, while the latter is for 3D shape. Hyper-parameters are in Table
We perform 3D pose and shape estimation 3DPW 667 (Figure
B.5 Pedestrian attribute recognition
We evaluate HAP on three popular datasets, i.e., P0-K 47 (Figure 8k RAP 39
(Figure 81 and
PETA [8]] (Figure 8m). We implement using the codebase of B6 The resolution 256 192. Data augmentations consist of resizing and random horizontal flipping. The binary cross-entropy loss employed, and the evaluation metric of mA (%) reported. Find the hyper syper-parrmmters in Table T0
htaps:/ggttih..oomm/oggnnccnn//i Btt.er/wwwweLLLLLLLELSS.S.. RELEASE http://ghhibcccommvaaeeeeoobelet of PAR
17

---

Table 10: Hyper-parameters pedestrian attribute recognition.
dataset
batch size
epochs
learning rate
schedule
optimizer
weight decay
PA-100K
64
55
1.7e-4
cosine cosine cosine
AdamW
5c-4
RAP
64
8
2.4e-4
AdamW
5e-4
PETA
64
50
1.9c-4
AdamW
5c-4
C Study of human part prior
head
upper body
left arm
right arm
left leg
right leg
Figure 7: Part masking with keypoints. We divide human body structure into parts following [60]. For each input image, we randomly select 0 to 6 parts (following uniform distribution) and mask patches located on them (according to the sequence of human part selection). For each selected part, we mask the patches within the bounding boxes of the corresponding keypoint pairs, i... head: ((noe,, left eye), (nose, right eyc), (left eye, right eye). (left eyc, left ear), (right eyc, right car) upper body: ((left shoulder, right hip). (right shoulder, left hip)); left arm: {Aeft shoulder, left elbow), (left elbow, left wrist)); right arm: ((right shoulder, right elbow), (right elbow, right wrist)); left leg: {left hip, left knee), (left knee, left a right leg: ((right hip, right knee), (right knee, right ankle)).
Details of part masking with keypoints. We incorporate human parts, a human structure prior, into pre-training. Figure iilustrates the details of how to mask each of the six body parts with keypoints.
Table II: Guidance in mask sampling. Baseline uses no guidance, i.e., random block-wise masking.
guidance
MSMT17
MPII
attention map
71.7
90.4
keypoint
75.9
91.6
part
76.4
91.8
baseline
75.6
91.6
Masking guidance in mask sampling strategy. We study different guidance, reported in Table
6 37 use attention map to identify and mask the informative image patches. However, this design performs poorly in human-centric pre-training, achieving even lower performance than baseline (71.7 vs. 75.6 on MSMTI7). We conjecture that underlying reason the challenge for the model to obtain precise and meaningful attention maps, especially in the initial stage of the pre-training process. That may be why recent works 40.37] 6) start to employ strong pre- trained models to assist mask sampling. In contrast, we directly employ intuitive and explicit human part prior, i.e., human keypoints, to guide mask sampling, which shows great potential. Besides, observe that only masking patches located on keypoints, instead of human parts, improves little. indicates that masking a whole human body works better, similar to the finding that blick-wise masking performs better than random masking in human-centric pre-training (see the main paper).
Some MIM works [3 29
Part selection. Here, we ablate different part selection strategies in Table 12 We observe that selecting fixed number of parts is not beneficial. Compared to a whole range, selecting within small range has little improvement. shows that more part variations brings more randomness,
which is good for pre- pre-training Similar to the findings in recent studies [83 29
6 37 that deliberate
18

---

Table 12: Part selection for masking. Baseline uses no guidance. i.e., random block-wiss masking.
part selection
MSMTI7
MPII
75.4
91.5
74.4
91.0
unif (0.3) unif (0.6)
76.1
91.7
76.4
91.8
baseline
75.6
91.6
design of mask sampling strategy would have positive impact, our part selection strategy beneficial for human-centric pre-training, resulting in great performance
range of downstream tasks.
Using keypoints as target. Since keypoints can also be used as supervision, we perform this baseline by making the decoder predict keypoints instead of pixels. The obtained results are 50.3% on MSMT17 and 89. 99.8% MPII with 100-epoch pre-training, while our HAP 72.2% and 91.3%. The possible reason is that regressing keypoints may lose informative details provided by pixels, which has negative impact on the tasks. In contrast, HAP incorporates prior knowledge of structure into pre-training, which is beneficial to maintain both semantic details and information of body structure.
Using another keypoint detector. To study the robustness of HAP the extracted keypoints, make a simple exploration by employing another off-the-shelf keypoint detector named OpenPose []] achieves 72.0% on MSMT17 and 91.2% comparable to our HAP with ViTPose [78) as the pose detector (72.2% and 91.3% on MSMT17 and MPII respectively), demonstrating that our HAP is generally robust the keypoint detector method.
MPII with 100-epoch pre-training. The results are
D Study of structure-invariant alignment
Table 13: Analysis of loss weight 7. 7 0 represents our baseline that does not use alignment loss.
MSMT17
MPII
0.5
75.6
91.2
76.3
91.8
0.05
76.4
91.8
0.01
76.0
91.6
(basclinc)
75.1
91.1
Loss weight Table
demonstrates that the weight of Eq.2 is beneficial across wide range.
Table 14: Type of two views in alignment. The "feature 1" and "feature 2" aligned via Lalign-
feature
feature
MSMT17
MPII
masked view
global view
74.5
91.0
masked view
visible view
76.1
91.7
masked view
masked view
76.4
91.8
baseline
75.1
91.1
Type of two views in alignment. By default, we use two random masked views for alignment. We further explore other two types of alignment, see Table 14 Aligning the masked view with the global view (i.e., without masking) shows poor performance. We attribute to high degree of overlap between the two views, which makes the task too easy. Aligning the masked view and the visible view with no overlap yields improvement, but still inferior than the default. It shows that learning subtle humans characteristics by aligning random views of corrupted body structure is important.
Removing negative samples in the alignment loss. We the loss SimSiam || instead. The results are 76.16 % (SimSiam) vs. 76.4% (ours) on MSMT17, 91.6% (SimSiam) vs. 91.8% (ours) on MPII. It demonstrates that our HAP still works well if removing the negatives in the alignment loss.
19

---

(a) LUPerSon 2
(for pre-training).
(b) MSMT17 74
(for person ReID).
Market-1550 88
(for person ReID).
(d) MPII 1 (for 2D pose estimation).
(e) MS COCO [46] (for 2D pose estimation).
(I) AI Challenger (AIC) [75] (for 2D pose estimation).
CUHK CUHK-PEDES PEDES
42 (for -ix-ee--mmage ReID).
(h) ICEG-PEDES 20 (for text-to-image RelD).
(i) RSTPRRed 89
(for text-to-image ReID).
(j) 3DPW 67 (for 3D pose and shape estimation).
(k) PA-100K 47 (for attribute recognition).
(1) RAP 33 (for attribute recognition).
(m) PETA [8] (for attribute recognition).
Figure 8: Example images of the used datasets. Our proposed HAP is pre-trained (a) LUPerson, and evaluated on broad range of tasks and datasets, including (b-c) person ReID, (d-f) 2D pose estimation, (g-i) text-to-image person ReID, (j) 3D pose and shape estimation, (k-m) pedestrian attribute recognition. These datasets encompass diverse environments with varying image quality, where persons are varied by scales, occlusions, orientations, truncation, clothing, appearances, and postures, demonstrating the the great generalization ability of HAP in human-centric perception.
20


Reviews:
[{"review_id": "NMDohvDopf", "reviewer": "Reviewer_QQMw", "summary": "This paper propose a pertaining strategy for human-centric vision tasks. They extend the mask image modelling approach by incorporating a prior on the human body parts to guide the mask sampling strategy. In short, they mask parts of the image which contains body part. Authors also propose an alignement loss to make sure that the same image with two different masks have same feature representations. They show fine-tuning performance on several standard benchmarks.", "questions": "1) l.44 “Or a given input image, we generate two views. Not sure that we can really say that there are two views. Using the word ‘view’ makes reference to 3D and here the masked images are from the same image not from two different camera viewpoints. So I would suggest to find a different explaination.\n2) Most of my questions are in the weaknesses. Weaknesses 1) and 2) are the most important points to me.", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-written and easy to read. The figures are clear and they help to get the main idea of the paper. Since authors do not claim to bring a very novel idea/technical contribution they explain that they start from an existing work (MAE) and try to extend this method for human-centric tasks. They perform experiments to identify issues for this method and they propose patches to fix them. The intuition behind the proposed fixs are based on human prior and are explained with references to prior works (i.e. body parts). Even if we could argue that the novelty is somehow limited, the proposed paper is well-constructed and may have a positive impact in the vision community specially for researchers working on human0centric vision.\n2) The introduction introduced well the research problem, and summarise the main idea and claim of the paper. I appreciate that authors do not claim their method as SSL (even if they do in the Appendix l.13) because they use pseudo-GT (keypoints) for sampling masks in the input images.\n3) Section 3.1 is very much appreciated. I guess that the project started with these experiments so it make sens t.\n4) Authors presented good ablation studies (Table 2-3), as a reader we understand the impact of each components during the pretraing (some important ones are still missing see weaknesses).\n5) Supplementary materials contain all the details for each downstream tasks which make the results reproducible. Furthermore authors also include the code for pretraining using their approach. Authors mentioned that code will be release soon.\n6) Good performance on several downstream benchmarks where they compare against recent works. They reach SoTA on multiple datasets.", "weaknesses": "1) One major weakness of this submission is the lack of a simple baseline which is; pretraining on LUPerson by training to regress the 2d keypoints extracted by VITPose. I understand that they are noisy because they are pseudo-GT but I guess that because there are more than 2M images in LUPerson it should already give a better understand of why we need to pretrain using supervision signal from the pixel instead of using supervision signal for the 2d keypoints. For example for the downstream tasks of 3D human mesh recovery most of the training data used by the research community are pseudo-GT extracted on in the wild images and it works better than using indoor image with perfect ground-truth.\n2) Authors mentioned that 2d keypoints are obtained by VITPose, it would be great to study the impact of the keypoint quality by using another off-the-shell detector such as OpenPose or pretraining on a big dataset with ground-truth 2D pose such as MSCOCO. I think that such ablation is very important for making sure that the key component of your proposed method is: a) to leverage a large-scale dataset for pretrainning — or b) to use a very recent off-the-shell 2d pose detector. At the moment we do not have any answer to this question in the proposed manuscript. I would suggest to pretrain your method on either LUPerson or MSCOCO with the same number of pretraining examples and using either GT 2d pose of 2D pose extracted from VITPose.\n3) I did not find the information in the paper of Supp. Mat. About the initialisation of the weights for the pretraining stage. Do you train from scratch or from MAE-Imagenet?\n4) It is a bit surprising that the fine-tuning on the downstream task of 3D pose does not work as well as [14] since you are just changing the backbone. It is a bit counter-intuitive with the results that you get on the other tasks. Moreover the backbone in [14] is a CNN pertained on 2d pose estimation so the conclusion regarding this task are a bit weird in comparison to the other tasks.\n5) The Structure Alignement Loss is appealing since it bring contrastive learning in the paper but given results in Table 2 it seems that it does not bring a significant gain. The improvements are quite marginal and similar to HAP-2 given a certain variance during the fine-tuning stage (only on MSMT17 results are better than HAP-2).\n6) Other human-centric pretraining method such as PATH are proposing zero-shot or results on downstream task with a frozen encoder. Do you have some results following this fine-tuning strategy (i.e. training only the head). It would be great to see how important it is to fine-tune the encoder for each task. Or maybe using adaptor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lack of a simple baseline: pre-training on LUPerson by training to regress the 2D keypoints extracted by ViTPose.**\n\nA1: Thanks for your constructive suggestion. We experiment this simple baseline and achieve 50.3\\% on MSMT17 and 89.8\\% on MPII with 100-epoch pre-training. This baseline is significantly inferior than our original HAP with RGB pixel as supervision signal (72.2\\% on MSMT17 and 91.3\\% on MPII). The underlying reason is that regressing 2D keypoints may lose informative details provided by RGB pixels, which has negative impact on human-centric tasks, especially on ReID tasks (refer to MSMT17 results). In contrast, our HAP is able to incorporate prior knowledge of human body structure into the pre-training, which is beneficial to maintain both details and body-structure information. We will add the above experiment and analysis in the revised version.\n\n**Q2: It would be great to study impact of the keypoint quality by using another off-the-shell detector such as OpenPose or pre-training on a big dataset with ground-truth 2D pose such as MSCOCO.**\n\nA2: Thanks for your advice.\n\ni) Using 2D keypoints extracted by another off-the-shell detector OpenPose in HAP achieves 72.0\\% on MSMT17 and 91.2\\% on MPII with 100-epoch pre-training. The results are similar to our original HAP with ViTPose as pose detector (72.2\\%/91.3\\% on MSMT17/MPII), reflecting that our method is robust to the off-the-shell detector method.\n\n\nii) MSCOCO has about **260k** images for the 2D pose task. Using MSCOCO with its 2D ground-truth pose in HAP achieves 60.3\\%/89.5\\% on MSMT17/MPII with 100-epoch pre-training. Moreover, using the same number of pre-training samples of LUPerson (about 260k) achieves 63.0\\%/89.7\\%. These results are inferior than our original HAP with **2.1M** LUPerson (72.2\\%/91.3\\% on MSMT17/MPII), showing that large-scale dataset is important for pre-training. \n\n**Q3: I did not find the information in the paper of Supp. Mat, about the initialization of the weights for the pre-training stage. Do you train from scratch or from MAE-ImageNet?**\n\nA3: Our HAP is trained with the weight initialized from MAE-ImageNet (Line 18-19 in Appendix).\n\n**Q4: The conclusion regarding 3D pose task area bit weird in comparison to the other tasks.**\n\nA4: Thanks for pointing out it. The underlying reasons are:\n\ni) Backbone. ViT and CNN work differently.\nViTs process image patches sequentially and use attention mechanism to capture global information, resulting in the destroy of spatial information yet it is important in 3D pose.\nIn contrast, CNN can leverage local receptive fields through convolutions to capture the spatial dependencies, which is beneficial for 3D pose.\nGiven that most of existing methods [14,a,b,c] use CNN to extract 2D pose and then use transformer for 3D estimation, there lacks plain ViT-based 3D pose method.\nThus we conjecture that plain ViT does not work well on 3D pose until now.\n\nii) Training. CNNs have been well studied in 3D pose task with various training experiences and tricks, while these experiences are lacked of study in ViT.\nOur utilization of [14] with ViT backbone is only a simple and preliminary attempt to validate the possibility of our HAP on 3D pose task, which is not optimal to achieve good performance.\nWe will continue to study ViT on 3D pose task in the future.\n\n[a] Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation, TMM2022.\n\n[b] Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation, CVPR2022.\n\n[c] P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation, ECCV2022.\n\n**Q5: The Structure Alignment Loss is appealing since it brings contrastive learning in the paper, but given results in Table 2, it seems that it does not bring a significant gain. The improvements are quite marginal and similar to HAP-2 given a certain variance during the fine-tuning stage (only on MSMT17 results are better than HAP-2).**\n\nA5: There might be a slight misunderstanding here. In fact, HAP-2 is the one which uses the structure alignment loss. Compared with HAP-2 vs. HAP-0 and HAP vs. HAP-1, our structure alignment loss results in +1.5\\%/+1.3\\% improvement on MSMT17, and +0.7\\%/+0.7\\% improvement on MPII, respectively.\n\nWe also appreciate your positive comments on the appealing structure alignment loss.\n\n**Q6: Other human-centric pre-training method such as PATH are proposing zero-shot or results on downstream task with a frozen encoder. Do you have some results following this fine-tuning strategy (i.e. training only the head). It would be great to see how important it is to fine-tune the encoder for each task. Or maybe using adaptor.**\n\nR6: Thanks for your insightful advice.\n\ni) Unfortunately, our HAP fails on the evaluations of training only the head, achieving largey inferior performance than PATH.\nThe reason is that HAP is a self-supervised method that pre-trains on the datasets **different** from that in the downstream tasks, while PATH is a supervised learning method using most of the downstream datasets.\n\nii) Fine-tuning the encoder for each task is important.\nUnder the fine-tuning setting, our HAP performs better than PATH on most of the benchmarks.\nMoreover, PATH also points out that fine-tuning is necessary to obtain high performance on in-dataset, out-of-dataset and unseen-task evaluations.\n\n**Q7: Find a different explanation of \"views''.**\n\nR7: Thank you for pointing out this. We would like to accept your suggestion and use \"masked images'' to replace \"views'' in the revised version."}, {"review_id": "RLgqIZ4Z4P", "reviewer": "Reviewer_BM1x", "summary": "The work studies masked image modeling (MIM) in human-centric perception. It first revisits the vanilla MIM and finds that human structure prior (2D pose) helps the downstream human-related tasks. This encourages the authors to incorporate this prior into the classical MAE. Based on this human-centric masking strategy, a structure-invariant alignment loss is developed as a regularization for pre-training. To evaluate the effectiveness of the proposed method, the authors conduct extensive experiments on 11 human-centric benchmarks. The competitive performance is observed when compared to the recent human-centric pre-training counterparts.", "questions": "As the technical novelty of this paper is human-related masking, the authors are encouraged to ablate the part selection. For example, the P parts are randomly selected, as denoted in Section 3.2. I am wondering how the performance will be if only making one or two parts, such as the head or upper body. Then that will be six ablation studies (each one masks a part only) to investigate the impact of each part.", "limitations": "In the limitation section, the authors state that the accuracy of pose estimation could affect the pre-training quality. However, the quantitative or qualitative results are not presented.  ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The overall writing and organization are satisfactory. The paper well describes the necessity of human structure prior to human-centric pre-training. The statistics are convincing to convey the motivation to the audience.  \n\n2) The pipeline incorporates MAE with pose-related masking is simple and effective. It can be easily applied to any MIM method. \n\n3) The experiments are sufficient to verify the effectiveness of the proposed method, with informative visualization and ablation studies.\n\n4) Codes are attached, and implementation details are well described for re-implementation.", "weaknesses": "The performance on multiple benchmarks is overclaimed. For example, the results of Attribute recognition are marginal when comparing HAP with other human-centric pre-training methods. With bells and whistles, HAP (multi-dataset training and larger image size) outperforms other methods significantly. However, the vanilla HAP is inferior to the LiftedCL. PATH, and UniHCP, though they use different pre-training datasets. So, I reckon the superior performance this paper claims should be re-examined, or restated at least.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The performance on multiple benchmarks is overclaimed. For example, the results of Attribute recognition are marginal when comparing HAP with other human-centric pre-training methods. With bells and whistles, HAP (multi-dataset training and larger image size) outperforms other methods significantly. However, the vanilla HAP is inferior to the LiftedCL, PATH, and UniHCP, though they use different pre-training datasets. So, I reckon the superior performance of this paper claims should be re-examined, or restated at least.**\n\nR1: Thanks for your valuable advice. We will re-examine and restate the claim of superior performance in the revised version following your advice. \nFor clarification, our HAP is a self-supervised method, while PATH and UniHCP are supervised learning method, thus the comparison is not totally fair.\nWhen comparing with other self-supervised methods (like SOLIDER), our HAP achieves superior performance.\n\n**Q2: I am wondering how the performance will be if only making one or two parts, such as the head or upper body. Then that will be six ablation studies (each one masks a part only) to investigate the impact of each part.**\n\nThanks for this intriguing point. Following your suggestion, we implement six ablation studies, in which each one only masks one part, i.e., head, left arm, right arm, left leg, right leg and upper body, to investigate the impact of each part.\nOther settings keep the same as in overall HAP.\nThe results in the below table show that our overall HAP achieves slightly better performance than only masking one part.\nAmong six parts, the upper body part can provide relatively more information than others.\n\n| dataset | unif{0, 6} | head | left arm | right arm | left leg | right leg | upper body |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| MSMT17 | 72.2 | 71.8 | 72.0 | 71.9 | 71.9 | 72.0 | 72.1 |\n| MPII | 91.3 | 90.9 | 91.2 | 91.1 | 91.1 | 91.1 | 91.2 |\n\n**Q3: In the limitation section, the authors state that the accuracy of pose estimation could affect the pre-training quality. However, the quantitative or qualitative results are not presented.**\n\nR3: Thank you for pointing out this. \nWe add an experiment that uses 2D keypoints extracted from OpenPose, another pose estimation method, for the pre-training of HAP.\nIt achieves 72.0\\%/91.2\\% on MSMT17/MPII with 100-epoch pre-training, which are similar to our original HAP with ViTPose as pose detector (72.2\\%/91.3\\% on MSMT17/MPII).\nIt reflects that the accuracy of pose estimation has slight effect on the pre-training quality.\nWe will further study more pose estimation methods to validate our statement.\n\n\n"}, {"review_id": "AvlxmNd4Ww", "reviewer": "Reviewer_ueQY", "summary": "The authors introduce masked image modeling as a pre-training method specifically designed for human-centric perception tasks. To this end, the authors incorporate human structure priors (high scaling ratio, mediate masking ratio, block-wise masking), human part prior (2D keypoints as guidance for mask sampling), and structure-invariant alignment loss (contrastive loss on the [CLS] tokens across views). Extensive experiments demonstrate that the proposed method achieves competitive performance on various human-centric perception tasks.", "questions": "I have an additional question. The authors use contrastive loss with negative samples. What about removing negatives? Prior works (e.g., BYOL, SimSiam, DINO) have proven that negative samples are not necessary for better representation learning. Overall, I am leaning towards borderline accept considering that the authors provide a new formulation of previous techniques for a new human-centric perception scenario.", "limitations": "The authors have discussed the limitations and the broader impacts in Sec. 5, which looks good to me.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tThe proposed method is well motivated. Directly applying MIM in human-centric perception tasks will indeed induce some problems. The authors exploited some priors from the person dataset to make pre-training tailored for specific downstream tasks.\n-\tThe paper is generally well-written and easy to follow.\n-\tThe experiments are extensive and the results are promising.\n", "weaknesses": "-\tThe technical novelty is limited. The proposed method is essentially a combination of many previous techniques used in masked image modeling. For example, the block-wise masking has been proposed in BEiT [2]. Semantic-guided masking has been proposed in prior works [5, 25, 33, 36, 41, 60, 74]. The only difference is that these works use attention maps while the authors use keypoints. Adding alignment loss is actually combining masked image modeling with contrastive learning (like what has been done in iBOT [77]).\n-\tThe authors use 2D keypoints to guide pre-training, which are extracted by existing pose estimation methods. I am afraid the obtained keypoints are not fully unsupervised.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The technical novelty is limited. The proposed method is essentially a combination of many previous techniques used in masked image modeling. For example, the block-wise masking has been proposed in BEiT [2]. Semantic-guided masking has been proposed in prior works [5, 25, 33, 36, 41, 60, 74]. The only difference is that these works use attention maps while the authors use keypoints. Adding alignment loss is actually combining masked image modeling with contrastive learning (like what has been done in iBOT [77]).**\n\nA1: Thanks, we agree that many techniques have been used in masked image modeling (MIM), while\nour HAP further **revisits and reformulates** these techniques in a novel way to introduce **human structure priors** (our key contribution) into MIM pre-training for human-centric perception tasks. In details, \n\ni) Block-wise masking. \nUsing block-wise masking (proposed in BEiT [2]) is just a **finding** that it performs sightly better than random-wise masking in human-centric perception tasks (refer to preliminary study in Sec. 3.1), thus it is not the main point in our HAP.\n\nii) Human part prior guided masking is different from semantic-guided masking. \nThe proposed human part prior guided masking is the key point in HAP that is **specialized** for human-centric tasks by introducing keypoints.\nIt can be regarded as an **external and human-friendly** guidance for masking, instead of **self** guidance from attention maps in semantic-guided masking [5, 25, 33, 36, 41, 60, 74].\nWe also experiment HAP by directly using attention maps to guide masking.\nThis setting achieves 71.7\\%/90.4\\% on MSMT17/MPII (refer to Table 7 in Appendix), largely inferior than our HAP with human part guidance (76.4\\%/91.8\\%) and with keypoints (75.9\\%/91.6\\%),\nwhich evidently shows the superiority of the proposed masking strategy guided by human prior.\n\niii) Adding alignment loss combines MIM with contrastive learning.\nWe agree. Despite similar formulation as contrastive loss, the proposed structure-aware alignment concept further applies contrastive loss on masked views generated from human part prior to enhance the feature discriminative ability.\niBOT [77] is different from ours in: a) the masked views are randomly generated without prior guidance. b) the loss is formulated by cross-entropy loss, not contrastive loss as in ours.\n\n**Q2: The authors use 2D keypoints to guide pre-training, which are extracted by existing pose estimation methods. I am afraid the obtained keypoints are not fully unsupervised.**\n\nR2: Thanks for pointing out this concern.\nThe 2D keypoints only provides the prior guidance for mask sampling in HAP, rather than the supervision signal for pre-training.\nHAP only use image pixels as the pre-training targets, and it is a self-supervised learning method.\nWe will carefully revise the manuscript to make it clear.\n\n**Q3: The authors use contrastive loss with negative samples. What about removing negatives? Prior works (e.g., BYOL, SimSiam, DINO) have proven that negative samples are not necessary for better representation learning.**\n\nThanks for your constructive suggestion. \nFollowing your advice, we perform our HAP using the loss of SimSiam to replace our original alignment loss. \nThe results are 76.1\\% (loss of SimSiam) vs. 76.4\\% (our alignment loss) on MSMT17 for Person ReID, 91.6\\% (loss of SimSiam) vs. 91.8\\% (our alignment loss) on MPII for 2D pose estimation.\nIt shows that our HAP still works well if removing the negatives in contrastive loss, while our alignment loss with negatives performs slightly better. \nWe will include this interesting finding in the revised version. "}, {"review_id": "1jrNxPEtE1", "reviewer": "Reviewer_rHqg", "summary": "This paper proposes a Human structure-Aware Pre-training (HAP) method that incorporates human structure priors into the masked image modeling (MIM) training strategy [26] for tasks related to human-centric perception. The authors have demonstrated the advantages of the proposed method on 5 human-centric perception tasks across 12 benchmark datasets. Overall, the method is simple and intuitive, and extensive experiments have provided solid evidence of its superiority in human-centric perception tasks.", "questions": "1. The accuracy of pose estimation methods seems to significantly impact the performance of the HAP proposed by the author. However, it appears that the author hasn't conducted sufficient analysis, such as how OpenPose or AlphaPose specifically affect the performance of HAP.\n\n2. In 3D human pose and shape estimation task, the performance of 3D human mesh estimation often decreases due to occlusions. The HAP method proposed by the author seems promising in solving this issue. However, it's unfortunate that the author did not conduct relevant experiments to further highlight the advantages of the proposed HAP. It would be beneficial for the author to incorporate such experiments to enhance the strengths of their HAP.\n\n\n\n\n", "limitations": "The author mentions that their proposed method may have potential negative societal impacts when applied to image generation tasks. Personally, I consider this to be a problem that we collectively need to address in the process of advancing artificial intelligence. Therefore, it does not diminish my positive evaluation of the research work. However, I still hope that the author can provide possible solutions to mitigate these potential negative impacts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author, through analyzing the deficiencies of the current MIM training strategy [26] in human-centric perception tasks, further proposes the introduction of human structure priors to expand the MIM training approach. This method is intuitive and appropriate.\n\n2. Although the author uses the existing InfoNCE loss to implement structure-invariant alignment loss, the application of this structure-invariant alignment concept to different masked views in order to align them, thereby enhancing the feature representation ability of human structure information, is both straightforward and promising.\n\n3. The author not only demonstrated the superiority of the proposed method across twelve benchmark datasets, but also validated the benefits of each proposed training strategy or loss function through ablation experiments.\n\n4. The organization of the article and the presentation of the methods are both clear and well-structured.\n", "weaknesses": "While this method effectively integrates prior knowledge and existing loss functions to enhance the performance of human-centric perception tasks, its technical originality and novelty still leave something to be desired.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The accuracy of pose estimation methods seems to significantly impact the performance of the HAP proposed by the author. However, it appears that the author hasn't conducted sufficient analysis, such as how OpenPose or AlphaPose specifically affect the performance of HAP.**\n\nR1: Thanks for pointing out this. We further experiment HAP with 2D keypoints extracted by OpenPose.\nThe results are 72.0\\%/91.2\\% on MSMT17/MPII with 100-epoch per-training, which are similar to the results with the keypoints extracted by ViTPose (our original HAP), i.e., 72.2\\%/91.3\\% on MSMT17/MPII.\nThey both achieve superior performance than baseline (69.4\\% on MSMT17 and 90.4\\% MPII).\nIt reflects that the accuracy of pose estimation has slight effect on the pre-training quality.\nWe will further study more pose estimation methods to validate our statement.\n\n**Q2: In 3D human pose and shape estimation task, the performance of 3D human mesh estimation often decreases due to occlusions. The HAP method proposed by the author seems promising in solving this issue. However, it's unfortunate that the author did not conduct relevant experiments to further highlight the advantages of the proposed HAP. It would be beneficial for the author to incorporate such experiments to enhance the strengths of their HAP.**\n\nR2: Thanks for your constructive suggestion.\nWe will add occlusion-based experiments in 3D human pose and shape estimation task by carefully studying experimental settings in the revised version to further enhance the strengths of our HAP.\n\n**Q3: The author mentions that their proposed method may have potential negative societal impacts when applied to image generation tasks. Personally, I consider this to be a problem that we collectively need to address in the process of advancing artificial intelligence. Therefore, it does not diminish my positive evaluation of the research work. However, I still hope that the author can provide possible solutions to mitigate these potential negative impacts.**\n\nR3: Thanks a lot for your positive evaluation of our research work. \nAnd very thanks for pointing out the suggestion that we can \"provide possible solutions to mitigate these potential negative impacts''.\nWe will carefully update the \"Broader impacts'' section as follows based on your suggestion in the revised version.\n\n\"Broader impacts and possible solutions: The proposed method is pre-trained and evaluated on person datasets, thus could reflect biases in these datasets, including those with negative impacts. When using the proposed method and the generated images, one should pay careful attention to dataset selection and pre-processing to ensure a diverse and representative sample. To avoid unfair or discriminatory outcomes, please analyze the model's behavior with respect to different demographics and consider fairness-aware loss functions or post-hoc bias mitigation techniques. We advocate for responsible use, and encourage the development of tools to detect and mitigate the potential misuse of our HAP and other masked image modeling techniques for unethical purposes.\""}]

Author rebuttal:
We thank all the reviewers for their insightful and  constructive comments. We are encouraged that reviewers generally recognize the strengths of our paper in:

- Method: positive impact [QQMw], simple and effective [BM1x], make sense [QQMw], well motivated [ueQY], intuitive and appropriate [rHqg], straightforward and promising [rHqg].
- Experiment: sufficient and extensive [BM1x, ueQY], good and informative ablation studies [QQMw, BM1x], good performance [QQMw, ueQY].
- Presentation: clear [QQMw, rHqg], well-written [QQMw, BM1x, ueQY], well-constructed [QQMw, rHqg], well-described details [BM1x].

The point-wise responses have been provided below. We hope our responses can clarify all reviewers' confusion and alleviate all concerns. 

Note that due to the limitation of rebuttal time, we only conducted pre-training for 100 epochs for the additional experiments suggested by the reviewers. In the future, we will expand to 400 epochs to further demonstrate the reliability of the conclusion.

Meta-review: This work introduces human structure prior to human-centric pre-training, achieving performance gain on multiple downstream human-centric benchmarks.

Although this paper got a borderline score (averaged 5.25), all reviewers rated positively. During the rebuttal & discussion process, several concerns like baseline settings, more ablation studies, and the effect of occlusions raised by the reviewers are partly addressed by the author.

Although the novelty is somewhat limited (as mentioned by Reviewer QQMw, ueQY, and rHqg), I agree with the reviewers that this work may have a positive impact on the human-centric vision community as it reveals the importance of introducing human prior.

The AC is inclined to accept this paper.
Decision: Accept (poster)