SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junkun Yuan $^{1,2*}$ , Xinyu Zhang $^{2*‡}$ , Hao Zhou $^{2}$ , Jian Wang $^{2}$ , Zhongwei Qiu $^{3}$ , Zhiyin Shao $^{4}$ , Shaofeng Zhang $^{5}$ , Sifan Long $^{6}$ , Kun Kuang $^{1‡}$ , Kun Yao $^{2}$ , Junyu Han $^{2}$ , Errui Ding $^{2}$ , Lanfen Lin $^{1}$ , Fei Wu $^{1}$ , Jingdong Wang $^{2‡}$
$^{1}$ Zhejiang University  $^{2}$ Baidu VIS  $^{3}$ University of Science and Technology Beijing  $^{4}$ South China University of Technology  $^{5}$ Shanghai Jiao Tong University  $^{6}$ Jilin University  $^{\{zhangxinyu14, wangjingdong\} @baidu.com}$  {yuanjk, kunkuang}@zju.edu.cn Code: https://github.com/junkunyuan/HAP
Project Page: https://zhangxinyu-xyz.github.io/hap.github.io/
Model pre-training is essential in human-centric perception. In this paper, we first introduce masked image modeling (MIM) as a pre-training approach for this task. Upon revisiting the MIM training strategy, we reveal that human structure priors offer significant potential. Motivated by this insight, we further incorporate an intuitive human structure prior - human parts - into pre-training. Specifically, we employ this prior to guide the mask sampling process. Image patches, corresponding to human part regions, have high priority to be masked out. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across a range of human-centric perception tasks. To further capture human characteristics, we propose a structure-invariant alignment loss that enforces different masked views, guided by the human part prior, to be closely aligned for the same image. We term the entire method as HAP. HAP simply uses a plain ViT as the encoder yet establishes new state-of-the-art performance on 11 human-centric benchmarks, and on-par result on one dataset. For example, HAP achieves  $78.1\%$  mAP on MSMT17 for person re-identification,  $86.54\%$  mA on PA-100K for pedestrian attribute recognition,  $78.2\%$  AP on MS COCO for 2D pose estimation, and 56.0 PA-MPJPE on 3DPW for 3D pose and shape estimation.
Human-centric perception has emerged as a significant area of focus due to its vital role in real-world applications. It encompasses a broad range of human-related tasks, including person re-identification (ReID) [23, 32, 62, 90], 2D/3D human pose estimation [15, 78, 82], pedestrian attribute recognition [34, 36, 63], etc. Considering their training efficiency and limited performance, recent research efforts [8, 13, 16, 33, 64] have dedicated to developing general human-centric pre-trained models by using large-scale person datasets, serving as foundations for diverse human-centric tasks.
Among these studies, some works [8, 13, 33] employ contrastive learning for self-supervised pretraining, which reduces annotation costs of massive data in supervised pre-training [16, 64]. In this work, we take the first step to introduce masked image modeling (MIM), which is another form of self-supervised learning, into human-centric perception, since MIM [2, 9, 30, 77] has shown considerable potential as a pre-training scheme by yielding promising performance on vision tasks [44, 65, 78].
*Equal contribution. This work was done when Junkun Yuan, Zhongwei Qiu, Zhiyin Shao, and Shaofeng Zhang were interns at Baidu VIS. ‡Corresponding author.
(a) Human-centric perception with HAP.
(b) High performance of HAP.
Figure 1: (a) Overview of our proposed HAP framework. HAP first utilizes human structure priors, e.g., human body parts, for human-centric pre-training. HAP then applies the pre-trained encoder, along with a task-specific head, for addressing each of the human-centric perception tasks. (b) Our HAP demonstrates state-of-the-art performance across a broad range of human-centric benchmarks.
There is no such thing as a free lunch. Upon initiating with a simple trial, i.e., directly applying the popular MIM method [30] with the default settings for human-centric pre-training, it is unsurprising to observe poor performance. This motivates us to delve deep into understanding why current MIM methods are less effective in this task. After revisiting the training strategies, we identify three crucial factors that are closely related to human structure priors, including high scaling ratio $^2$ , block-wise mask sampling strategy, and intermediate masking ratio, have positive impacts on human-centric pre-training. Inspired by this, we further tap the potential of human structure-aware prior knowledge.
In this paper, we incorporate human parts [5, 26, 57, 60, 80], a strong and intuitive human structure prior, into human-centric pre-training. This prior is employed to guide the mask sampling strategy. Specifically, we randomly select several human parts and mask the corresponding image patches located within these selected regions. These part-aware masked patches are then reconstructed based on the clues of the remaining visible patches, which can provide semantically rich body structure information. This approach encourages the model to learn contextual correlations among body parts, thereby enhancing the learning of the overall human body structure. Moreover, we propose a structure-invariant alignment loss to better capture human characteristics. For a given input image, we generate two random views with different human parts being masked out through the proposed mask sampling strategy. We then align the latent representations of the two random views within the same feature space. It preserves the unique structural information of the human body, which plays an important role in improving the discriminative ability of the pre-trained model for downstream tasks.
We term the overall method as HAP, short for Human structure-Aware Pre-training with MIM for human-centric perception. HAP is simple in terms of modalities, data, and model structure. We unify the learning of two modalities (images and human keypoints) within a single dataset (LUPerson [23]). The model structure is based on the existing popular MIM methods, e.g., MAE [30] and CAE [9], with a plain ViT [21] as the encoder, making it easy to transfer to a broad range of downstream tasks (see Figure 1a). Specifically, HAP achieves state-of-the-art results on 11 human-centric benchmarks (see Figure 1b), e.g.,  $78.1\%$  mAP on MSMT17 for person ReID,  $68.05\%$  Rank-1 on CUHK-PEDES for text-to-image person ReID,  $86.54\%$  mA on PA-100K for pedestrian attribute recognition, and  $78.2\%$  AP on MS COCO for 2D human pose estimation. In summary, our main contributions are:
- We are the first to introduce MIM as a human-centric pre-training method, and reveal that integrating the human structure priors are beneficial for MIM in human-centric perception.
- We present HAP, a novel method that incorporates human part prior into the guidance of the mask sampling process to better learn human structure information, and design a structure-invariant alignment loss to further capture the discriminative characteristics of human body structure.
- The proposed HAP is simple yet effective that achieves superior performance on 11 prominent human benchmarks across 5 representative human-centric perception tasks.
2We simply refer the lower bound for the ratio of random area to crop before resizing as "scaling ratio". Please find more discussions in Section 3
Human-centric perception includes a broad range of human-related tasks. The goal of person Re-ID [24, 32, 51, 50, 81, 90, 91, 84, 86] and text-to-image person ReID [59, 20, 62, 58] is to identify persons of interest based on their overall appearance or behavior. Pedestrian attribute recognition [35, 36] aims to classify the fine-grained attributes and characteristics of persons. 2D [78, 79, 82] and 3D [69, 14, 25] pose estimation learn to predict position and orientation of individual body parts.
Recent works make some efforts to pre-train a human-centric model. HCMoCo [33] learns modal-invariant representations by contrasting features of dense and sparse modalities. LiftedCL [13] extracts 3D information by adversarial learning. UniHCP [16] and PATH [64] design multi-task co-training approaches. SOLIDER [8] tries to balance the learning of semantics and appearances.
Compared with these works, this paper focuses on exploring the advantage of utilizing human structure priors in human-centric pre-training. Besides, instead of training with numerous modalities and datasets using a specially designed structure, we pursue a simple pre-training framework by employing two modalities (images and human keypoints) of one dataset (LUPerson [23]) and vanilla ViT structure with the existing popular MIM methods (e.g., BEiT [2], MAE [30], and CAE [9]).
Self-supervised learning has two main branches: contrastive learning (CL) and masked image modeling (MIM). CL [4, 7, 10, 11, 12, 28, 31] is a discriminative approach that learns representations by distinguishing between similar and dissimilar image pairs. Meanwhile, MIM [2, 9, 30, 56, 72, 73, 77, 85, 88] is a generative approach that learns to recover the masked content in the corrupted input images. It is shown that CL learns more discriminative representations [4, 12], while MIM learns finer grained semantics [2, 30]. To obtain transferable representations for the diverse tasks, our work combines both of them to extract discriminative and fine-grained human structure information.
Semantic-guided masking is presented in some recent MIM works [6, 29, 37, 40, 45, 68, 83]. Many methods [83, 29, 6, 37] utilize the attention map to identify and mask informative image patches, which are then reconstructed to facilitate representation learning. In comparison, this paper explores to directly employ the human structure priors to guide mask sampling for human-centric perception.
3 HAP
Inspired by the promising performance of MIM in self-supervised learning, we initially employ MIM as the pre-training scheme for human-centric perception. Compared with MIM, our proposed HAP integrates human structure priors into MIM for human-centric pre-training, and introduces a structure-invariant alignment loss. The overview of our HAP framework is shown in Figure 2.
3.1 Preliminary: great potential of human structure priors
We use MAE [30] with its default settings as an example for human-centric pre-training. LUPerson [23] is used as the pre-training dataset following [8, 23, 51]. We designate this setting as our baseline. The results are shown in Figure 3. The baseline achieves poor performance on two representative human-centric perception tasks, i.e., person ReID and 2D human pose estimation. This phenomenon motivates us to delve deep into understanding the underlying reasons.
We start by visualizing the input images under different training strategies in Figure 4. Surprisingly, we observe that it is hard to learn information of human body structure from the input images of the previous baseline (see Figure 4 (b-c)). We conjecture that: i) the scaling ratio may be too small to preserve the overall human body structure, as shown in Figure 4 (b), and ii) the masking ratio may be too large to preserve sufficient human structure information, which makes the reconstruction task too difficult to address, as shown in Figure 4 (c).
Figure 4: Visualization of training strategies. For a given image (a), the baseline [30] uses  $20\%$  scaling ratio (b) and  $75\%$  masking ratio (c) with random mask sampling strategy, yielding a meaningless image with little human structure information. We adopt  $80\%$  scaling ratio and  $50\%$  masking ratio with block-wise mask sampling (d), maintaining the overall body structure.
Figure 2: Our HAP framework. During pre-training, we randomly select human part regions and mask the corresponding image patches. We then adjust the masked patches to match the pre-defined masking ratio. The encoder embeds the visible patches, and the decoder reconstructs the masked patches based on latent representations. For a given image, HAP generates two views using the mask sampling with human part prior. We apply the proposed structure-invariant alignment loss to bring the [CLS] token representations of these two views closer together. After pre-training, only the encoder with a plain ViT structure is retained for transferring to downstream human-centric perception tasks.
Driven by this conjecture, we further study the impacts of scaling ratio and masking ratio in depth. Figure 3 illustrates that both i) high scaling ratio (ranging from  $60\%$  to  $90\%$ ) and ii) mediate masking ratio (ranging from  $40\%$  to  $60\%$ ) contribute positively to the performance improvement of human-centric tasks. Especially, an  $80\%$  scaling ratio can bring an approximately  $3.5\%$  improvement on MSMT17 of the person ReID task. This effectively confirms that information of human body structure is crucial during human-centric pre-training. Additionally, we also discover that iii) block-wise mask sampling strategy performs sightly better than the random mask sampling strategy, since the former one masks semantic human bodies appropriately due to block regions. We refer to these three factors of the training strategy as human structure priors since they are associated with human structure information. In the following exploration, we employ scaling ratio of  $80\%$ , masking ratio of  $50\%$  with block-wise masking strategy by default. See an example of our optimal strategy in Figure 4 (d).
3.2 Human part prior
Inspired by the above analyses, we consider to incorporate human body parts, which are intuitive human structure priors, into human-centric pre-training. Here, we use 2D human pose keypoints [78], which are convenient to obtain, to partition human part regions by following previous works [60]. Section 3.1 reveals that the mask sampling strategy plays an important role in human-centric pretraining. To this end, we directly use human structure information provided by human body parts as robust supervision to guide the mask sampling process of our HAP during human-centric pre-training.
Specifically, we divide human body into six parts, including head, upper body, left/right arm, and left/right leg according to the extracted human keypoints, following [60]. Given a specific input image  $\mathbf{x}$ , HAP first embeds  $\mathbf{x}$  into  $N$  patches. With a pre-defined masking ratio  $\beta$ , the aim of mask sampling is to mask  $N_{m} = \max \{n \in \mathcal{Z} | n \leq \beta \times N\}$  patches, where  $\mathcal{Z}$  represents the set of integers. Specifically, we randomly select  $P$  parts, where  $0 \leq P \leq 6$ . Total  $N_{p}$  image patches located on these  $P$  part regions are masked out. Then we adjust the number of masked patches to  $N_{m}$  based on the instructions introduced later. In this way, unmasked (visible) human body part regions provide semantically rich human structure information for the reconstruction of the masked human body parts, encouraging the pre-trained model to capture the latent correlations among the body parts.
There are three situations for the masked patches adjustment: i) if  $N_{p} = N_{m}$ , everything is well-balanced; ii) if  $N_{p} < N_{m}$ , we further use block-wise sampling strategy to randomly select additional
(a) Scaling ratio analysis on (left) MSMT17 and (right) MPII. A scaling ratio with  $60\% -90\%$  works well.
Figure 3: Study on training strategy. We investigate three factors of training strategy, i.e., (a) scaling ratio, (b) masking ratio, and (a,b) mask sampling strategy, on MSMT17 [74] (person ReID) and MPII [1] (2D pose estimation). The baseline is MAE [30] with default settings (20% scaling ratio, 75% masking ratio, random mask sampling strategy). The optimal training strategy for human-centric pre-training is with 80% scaling ratio, 50% masking ratio, and block-wise mask sampling strategy.
(b) Masking ratio analysis on MSMT17 (left) and MPII (right). A masking ratio with  $40\% -60\%$  works well.
$N_{m} - N_{p}$  patches; iii) if  $N_{p} > N_{m}$ , we delete patches according to the sequence of human part selection. We maintain a total of  $N_{m}$  patches, which contain structure information, to be masked.
Discussion. i) We use human parts instead of individual keypoints as guidance for mask sampling. This is based on the experimental result in Section 3.1, i.e., the block-wise mask sampling strategy outperforms the random one. More experimental comparisons are available in Appendix. ii) HCMoCo [33] also utilizes keypoints during human-centric pre-training, in which sparse keypoints serve as one of the multi-modality inputs. Different from HCMoCo, we integrate two modalities, including images and keypoints, together for mask sampling. It should be noted that the keypoints in our method are pseudo labels; therefore they may not be accurate enough for supervised learning as in HCMoCo. However, these generated keypoints can be beneficial for the self-supervised learning.
3.3 Structure-invariant alignment
Learning discriminative human characteristics is essential in human-centric perception tasks. For example, person ReID requires to distinguish different pedestrians based on subtle details and identify the same person despite large variances. To address this issue, we further propose a structure-invariant alignment loss, to enhance the representation ability of human structural semantic information.
For each person image  $\mathbf{x}$ , we generate two different views through the proposed mask sampling strategy with guidance of human part prior. These two views, denoted as  $\mathbf{X}$  and  $\widetilde{\mathbf{X}}$ , display different human part regions for the same person. After model encoding, each view obtains respective latent feature representations  $\mathbf{Z}$  and  $\widetilde{\mathbf{Z}}$  on the normalized [CLS] tokens. Our structure-invariant alignment loss  $\mathcal{L}_{\mathrm{align}}$  is applied to  $\mathbf{Z}$  and  $\widetilde{\mathbf{Z}}$  for aligning them. We achieve it by using the InfoNCE loss [54]:
$$
\mathcal {L} _ {\text {a l i g n}} = - \log \frac {\exp (\mathbf {Z} \cdot \widetilde {\mathbf {Z}} / \tau)}{\sum_ {i = 1} ^ {B} \exp (\mathbf {Z} \cdot \mathbf {Z} _ {i} / \tau)}, \tag {1}
$$
where  $B$  is the batch size.  $\tau$  is a temperature hyper-parameter set to 0.2 [12]. In this way, the encoder is encouraged to learn the invariance of human structure, and its discriminative ability is improved.
Figure 5: Reconstructions of LUPerson (1st row), COCO (2nd row), and AIC (3rd row) images using our HAP pre-trained model. The pre-training dataset is LUPerson. For each triplet, we show (left) the original image, (middle) the masked image, and (right) images reconstructed by HAP. We highlight some interesting observations using red dotted ellipses. Although the reconstructed parts are not exactly identical to the original images, they are semantically reasonable as body parts. For example, in the first triplet of the 1st row, the reconstructed right leg of the player differs from the original, but still represents a sensible leg. It clearly shows that HAP effectively learns the semantics of human body structure and generalizes well across various domains. Zoom in for a detailed view.
3.4 Overall: HAP
To sum up, in order to learn the fine-grained semantic structure information, our HAP incorporates the human part prior into the human-centric pre-training, and introduces a structure-invariant alignment loss to capture discriminative human characteristics. The whole loss function to be optimized is:
$$
\mathcal {L} = \mathcal {L} _ {\text {r e c o n}} + \gamma \mathcal {L} _ {\text {a l i g n}}, \tag {2}
$$
where  $\mathcal{L}_{\mathrm{recon}}$  is the MSE loss for reconstruction following [30].  $\gamma$  is the weight to balance the two loss functions of  $\mathcal{L}_{\mathrm{recon}}$  and  $\mathcal{L}_{\mathrm{align}}$ . We set  $\gamma$  to 0.05 by default, and leave its discussion in Appendix.
Pre-training. LUPerson [23] is a large-scale person dataset, consisting of about 4.2M images of over 200K persons across different environments. Following [51], we use the subset of LUPerson with 2.1M images for pre-training. The resolution of the input image is set to  $256 \times 128$  and the batch size is set to 4096. The human keypoints of LUPerson, i.e., the human structure prior of HAP, are extracted by ViTPose [78]. The encoder model structure of HAP is based on the ViT-Base [21]. HAP adopts AdamW [49] as the optimizer in which the weight decay is set to 0.05. We use cosine decay learning rate schedule [48], and the base learning rate is set to 1.5e-4. The warmup epochs are set to 40 and the total epochs are set to 400. The model of our HAP is initialized from the official MAE/CAE model pre-trained on ImageNet by following the previous works [16, 78]. After pre-training, the encoder is reserved, along with task-specific heads, for addressing the following human-centric perception tasks, while other modules (e.g., regressor and decoder) are discarded.
Human-centric perception tasks. We evaluate our HAP on 12 benchmarks across 5 human-centric perception tasks, including person ReID on Market-1501 [87] and MSMT17 [74], 2D pose estimation on MPII [1], COCO [46] and AIC [75], text-to-image person ReID on CUHK-PEDES [42], ICFG-PEDES [20] and RSTPReid [89], 3D pose and shape estimation on 3DPW [67], pedestrian attribute
Table 1: Main results. We compare HAP with representative task-specific methods, human-centric pre-training methods, and the baseline MAE [30].  $\ddagger$  indicates using stronger model structures such as sliding windows in [32, 8] and convolutions in [50]. HAP follows the implementation of [50].
(a) Statistics on the number of training datasets and training data samples used during the pre-training process of the human-centric pre-training methods.
method	datasets	samples
LiftedCL [13]	1	~150K
SOLIDER [8]	1	~4.2M
HCMoCo [33]	2	~82K
UniHCP [16]	33	~2.3M
PATH [64]	37	~11.0M
HAP	1	~2.1M
(b) Attribute recognition. mA  $(\%)$  is reported.
method	PA-100K	RAP	PETA
ALM [63]	80.68	81.87	86.30
DAFL [35]	83.54	81.04	87.07
RethinkPAR [36]	81.61	80.82	85.17
Label2Label [43]	82.24	-	-
PATH [64]	85.0	81.2	88.0
UniHCP [16]	86.18	82.34	-
\( SOLDER^‡ \)[8]	86.37	-	-
MAE [30]	79.56	75.73	80.82
HAP	86.54	82.91	88.36
(c) Person RelD.  $\mathrm{mAP}(\%)$  is reported. For fair comparisons, we present all the results according to the input image resolution of  $256\times 128$  (256) and  $384\times 128$  (384) respectively for each compared method.
method	MSMT17	Market-1501
resolution	256	384	256	384
TransReID [32]	64.9	66.6	88.2	88.8
TransReID‡[32]	67.4	69.4	88.9	89.5
UP-ReID [81]	63.3	-	91.1	-
LUP [23]	65.7	-	91.0	-
PASS [90]	71.8	74.3	93.0	93.3
MALE‡[50]	73.0	74.2	92.2	92.1
TransReID-SSL‡[51]	-	75.0	-	93.2
PATH [64]	69.1	-	89.5	-
UniHCP [16]	67.3	-	90.3	-
SOLIDER‡[8]	-	77.1	-	93.9
MAE [30]	62.0	62.9	82.9	82.5
HAP	76.4	76.8	91.7	91.9
HAP‡	78.0	78.1	93.8	93.9
(d) 2D pose estimation. PCKh  $(\%)$  is reported for MPII and AP  $(\%)$  is reported for COCO and AIC.  $\dagger$  represents using multi-dataset training of COCO+AIC+MPII [78].  $^+$  represents using a larger image size of  $384\times 288$
method	MPII	COCO	AIC
HRNet-w48 [61]	90.1	75.1	33.5
HRFormer† [82]	-	75.6	34.4
HRFormer‡+ [82]	-	77.2	-
ViTPose [78]	-	75.8	-
ViTPose† [78]	93.3	77.1	32.0
LiftedCL [13]	89.3	71.1	-
PATH [64]	93.3	76.3	35.0
UniHCP [16]	-	76.5	33.6
SOLIDER‡+ [8]	-	76.6	-
MAE [30]	89.6	75.7	31.3
HAP	91.8	75.9	31.5
HAP†	93.4	77.0	32.2
HAP+	92.6	77.2	37.7
HAP†+	93.6	78.2	38.1
(e) Text-to-image ReID. Rank-1(%) is reported.
method	CUHK-PEDES	ICFG-PEDES	RSTPReid
SAFA [41]	64.13	-	-
LBUL [71]	61.95	-	43.35
CAIBC [70]	64.43	-	47.35
SSAN [20]	61.37	54.23	-
SRCF [62]	64.04	57.18	-
LGUR [59]	65.25	59.02	-
MAE [30]	60.19	53.68	44.15
HAP	68.05	61.80	49.35
(f) 3D pose and shape estimation. MPJPE, PA-MPJPE, and MPVPE are reported on 3DPW.
method	MPJPE ↓	PA-MPJPE ↓	MPVPE ↓
MiHu [22]	85.1	54.8	-
SPIN [38]	96.9	59.2	116.4
Pose2Mesh [14]	89.5	56.3	105.3
I2L-MeshNet [52]	93.2	57.7	110.1
3DCrowNet [15]	81.7	51.5	98.3
MAE [30]	95.6	58.0	112.7
HAP	90.1	56.0	106.3