PLIP: Language-Image Pre-training for Person Representation Learning
Authors: Jialong Zuo
Jiahao Hong
Feng Zhang
Changqian Yu
Hanyu Zhou
Changxin Gao
Nong Sang
Jingdong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Person Representation Learning
Language-Image Pre-training
Contrastive Learning
Primary area: machine_vision

Abstract: Language-image pre-training is an effective technique for learning powerful representations in general domains. However, when directly turning to person representation learning, these general pre-training methods suffer from unsatisfactory performance. The reason is that they neglect critical person-related characteristics, i.e., fine-grained attributes and identities. To address this issue, we propose a novel language-image pre-training framework for person representation learning, termed PLIP. Specifically, we elaborately design three pretext tasks: 1) Text-guided Image Colorization, aims to establish the correspondence between the person-related image regions and the fine-grained color-part textual phrases. 2) Image-guided Attributes Prediction, aims to mine fine-grained attribute information of the person body in the image; and 3) Identity-based Vision-Language Contrast, aims to correlate the cross-modal representations at the identity level rather than the instance level. Moreover, to implement our pre-train framework, we construct a large-scale person dataset with image-text pairs named SYNTH-PEDES by automatically generating textual annotations. We pre-train PLIP on SYNTH-PEDES and evaluate our models by spanning downstream person-centric tasks. PLIP not only significantly improves existing methods on all these tasks, but also shows great ability in the zero-shot and domain generalization settings. The code, dataset and weight will be made publicly available.

PLIP: Language-Image Pre-training for Person
Representation Learning
Jialong Zuo Hanyu Zhou
Jiahao Hong
Feng Zhang
Changqian Yu Jingdong Wang
Changxin Gao 1*
Nong Sang
National Key Laboratory of Multispectral Information Intelligent Processing Technology,
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology. Skywork AI, 3 Department of Computer Vision, Baidu Inc. ( @baidu.
(jlongzuo, chas) edu. cn, (wangjingdong)
Abstract
Language-image pre-training is effective technique for learning powerful rep- resentations in general domains. However, when directly turning to person repre- sentation learning, these general pre-training methods suffer from unsatisfactory performance. The reason is that they neglect critical person-related characteristics, i.e., gine-graindd attributes and identities. To address this issue, we propose novel lunguage-image pre- pre-training framework for person representation learning, termed PLIP. Specifically, we elaborately design three pretext tasks: 1) Text-guided Image Colorization, aims to establish the correspondence between the person person-related related image regions and the fine-grained color-part textual phrases. 2) anage-guided Attributes Prediction, aims to mine fine-grained attribute information of the person body the image; and 3) Identity-bant -based Vision-L-anuura Contrast, aims to corre- late the cross-modal representations at the identity level rather than the instance level. Moreover, to implement our pre-train framework, we construct large-scale person dataset with image-text pairs named SYNTH-PEDES by automatically gen- erating textual annotations. We pre-train train PLIP on SYNTH- models by spanning downstream person- censonc tasks. PLIP not only significantly improves existing methods on all these tasks, but also shows great ability the zero-shot and domain generalization settings.
and evaluate our
Introduction
Person-centric tasks, such as image/text- based person re- re-ientifficaton, person attribute recognition, person search and human parsing, are becoming increasingly influential in widespread applications, such as security monitor, smart city, virtual reality and scene understanding. Benefiting from the advances in designing task-specific methods [78, 9, 41, 54, 50]. these tasks have achieved significant progress for the past few years.
However, it has become evident through recent advancements in research that the mere development of sophisticated models based on specific tasks has already encountered a performance bottleneck. At the same time, some works [35, 15, 10, 14] general representation learning have shown great potential to further improve model performance. Due to the above reasons, some researchers [12] have attempted to learn generic person representations by utilizing rich person images. However, their pre- tre-training method based on sparse visual information falls short in terms of both training performance and efficiency.
Meanwhile, some works [72, 49, 85] have demonstrated that introducing the language modality helps to learn better representations in general domains, for the language naturally enjoys higher information density. However, when it comes to person representation learning, these general language-image pre-training methods like CLIP [72] are often unsatisfactory in performance, We believe the reason is that they neglect some critical person-related characteristics, i.e., fine- -en-igr attributes and identities.
Corresponding Author. Project Link: https://gitee com//pplusdaagonPPL.I.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

On the one hand, these general language-imann image pre-training methods typically implement the global alignment between cross-modality rep- resentations and lack explicit consideration for fine-grained information. However, in person domain, the fine-grained information, such as the person attributes, plays a key role in dis- tinguishing a specific person. Neglecting fine- grained attributes will easily lead to difficulty in learning discriminative person representations. On the other hand, they are based on instance level and only incorporate the concept of image- text pairs. They simply treat each image-text pair in the same person identity as different pairs, and assume that images and texts not in the same pair do not have a corresponding relationship. However, in person domain, there exists no- table concordance between the images and texts within the same person identity. If only conducting the optimization at the instance level, it will lead to instability to learn more meaningful representations.
Large-scale Dataset Construction
Downstream Tasks
riige faidd Fe
Alil sherts
SPAC
Hight
Language-Iman Image Pre-training
Rod ded
Visual Encoder
Teatual Encoder
Viiees- Vhicrs-lnggage contrast
Faiaa
Image giiteed Buits protiction
Figure 1: Illumination of our framework. Based on the constructed dataset, we pre-train a language- image model by three pretext tasks and transfer the model some downstream person-centre tasks.
To address these limitations mentioned above, we deeply consider the characteristies of persons and attempt to introduce the language modality into person representation learning. We propose well-motivated language-image pre-training framework for learning generic and discriminative person representations, termed PLIP, to help the downstream person-centric tasks. Also, to implement the pre-training, we construct a large-scale person dataset with rich image-text pairs. The whole framework is illustrated in Fig. 1.
Specifically, to explicitly learn fine-grained and meaningful cross-modal associations, we design three pretext tasks in PLIP:
(l)Text-guided Image Colorization, given a complete textual description, is designed to restore the color information of grayscale transformed person image. This task establishes the correspondence between the person-related image regions and the fine-grained color-par polor-part textual phrases, which robustly helps the model to learn the semantic concept of person body parts.
(2) emage-guided Attributes Prediction, by exploiting the paired colorful images, is designed to predict the masked attribute phrases in textual descriptions. This task primarily focuses on predicting attributes, rather than predicting any random masked words as in the general domains. Through this multi-modal masked language modeling, it helps the model understand the key areas and fine-grained attribute information of the person body in the image, which is crucial to identifying a person.
(3) Identity-bssed Vision Eision-language Contrast is designed to associate representations between vision and language at the identity level rather than the instance level in general domains. This means it is optimized by narrowing the distance between any images and texts in the same person identity and widening the distance between those not in the same person identity. By taking identity into consideration, this task achieves more robust and meaningful association between different modalities.
As is well known, the quality and quantity of training data is essential for learning rich representations. However, there exists huge domain gap between the large-scale image-text pairs used in general domains and the specific person data. Also, the scale of the existing person datasets [52, 24] with manual textual descriptions is limited due to expensive hand-labeled annotations. Therefore, we construct a new large-scale person dataset with image-text pairs named SYNTH- based on the LUPerson-NL- and LPW datasets [30, 83]. The text annotations are automatically synthetized by our proposed person image captioner named Stylish Pedestrian ittributes-union Captioning (SPAC). The dataset contains 312,321 identities, 4,791,77 1,711 images and 12,138, 1557 textual descriptions. At the same time, extensive experiments have been conducted verify the competitive quality of our synthetic dataset compared to manually annotated datasets [52, 24, 122], which guarantees the superior performance of the learned person representations.
2

---

We utilize PLIP to pre-train our models on the SYNTH-PEDES dataset, and then evaluate the model performance on spanning downstream person-centric tasks. Extensive experiments show that our pre- trained models have learned generic person representations, pushing many state-of-the-art methods to a higher level on a wide range of person-centric tasks without bells and whistles. For example, for unsupervised person Re-ID, by applying our pre-trained ResNet50 model on PPLR [20], we improve the mAP metric on Market [114] and MSMT17 [94] by 5.1% and 14.7%, respectively. The key contributions of this paper can be summarized as follows:
(1) We propose a novel aagguage--magee pre-training framework with three pretext tasks, termed PLIP, which deeply takes verson-related characteristics into consideration. It can facilitate fine-grained mrosl- association and learning generic person representations explicitly.
(2) To implement the pre-training, we construct a large-scale person dataset with generated text annotations, called SYNTH-PEDES. The dataset provides rich image-text pre-training data for this community.
(3) We pre-train PLIP on SYNTH-PEDES and the learned representations perform remarkable ability various downstream person-centric tasks. demonstrated as generic to bring significant improvements to various baseline methods.
2 PLIP: Representation Learning Framework
This section presents our proposed language-image based person representation learning framework PLIP via three pretext tasks, i.e., text-guided image colorization (TIC), image-guided attributes prediction (IAP), and identity-based vision-language contrast (IVLC). As illustrated in Fig. 2, the whole architecture is dual branch encoder structure, and generic person representations can be learned through joint training of these three pretext tasks.
2.1 Text-guided Image Colorization
The TIC task is designed to restore the original color information of grayscale transformed images by exploiting the complete textual descriptions. Such cross- modal colorization promotes the construction of image-text association. The reason is that the attribute phrases in the descriptions generally contain fine-grained person-centric information especially color information and this colorization process naturally enables the model to understand the key components in textual descriptions and achieve a relationship construction between the textual phrases and visual regions. The overall task can be converted into pixel-wiss regression problem.
As illustrated in Fig. 2, for a pair of gray image and complete textual description (igrags tcomplete J in the encoding stage, the input gray image igray is firstly fed into hierarchical network, which as the visual encoder. Secondly, the hierarchical features are up-sampled the same scale and concatenated to produce the feature F gray- Then, we feed the complete textual description commlette to the textual encoder [23] and adopts the average of the hidden-state outputs as the textual global embedding T gloobat-
In the decoding stage, the visual feature F gray and textual global embedding T Tooobal should be fused for colorization. Specifically, we adopt the mudti-modal SE-blocks [95] as the cross-modal feature fusing module, so that the textual global embedding can play role in the visual feature channels. In the block, the visual feature F Fessuul is compressed into a feature vector V through max pooling operation. Then we concatenate the visual feature vector and the textual global embedding T giobal. and feed the concatenated vector into several fo layers and a softmax layer to generate an attention vector A Finally, we utilize A on the visual feature F Frisual to generate multi-modal feature Zm. The decode is also made up of several de-convolution layers, which are employed to restore the feature dimensions. Finally, we generate a mutilmodal feature map with same dimensions as the input gray image igray and it can be utilized to predict the target color image Lestor.
We denote fise as the parameters of the trainable regression model mentioned above. maps the textual global embedding T gloobal and the gray image extracted feature F Frray to the output recovered color image icolor target. TIC is supervised by:
N
1
Lise
globat, + Diccotor Orio (Totobat, F graa)),
N
where can be any differentiable distance function such as Euclidean distance we adopt and N represents the total number of samples within batch.
3

---

Framework:
SIC:
SIC
VAP
VLC
MSE Boock
MISC-Block
Pooling
Visual Encoder
Textual Encoder
Z
VAP:
Makket Aedice Halre
Snoeilk ceneee
sharo carsingg bdea bartaack"
Masiel
Escoderr
Readom
Caption
[poket]
woutin (NSS [MASK (MASK)
MAS
with Teduual Feature Extraction
HAANC poiGa]"
Visual Festure Edtraction
Figure 2: Overview of our proposed framework incorporating a text-guided image colorization task, an maage-guided attributes prediction task and an identity- -based vision- langugge contrast task.
As displayed in Fig. 5 of the appendix, altering the color word in textual description significantly affects the colorization of image. However, our model may not fully understand the semanties of more detailed image regions. As shown in the last row, our model fails to distinguish between the blue clothing region and the red shoulder strap region (marked with yellow boxes), instead blending the two into unified coloration. This is due to the fact that the level of detail in manually annotated datasets still not sufficient, resulting in the model being unable to theoretically learn representations with higher levels of detail and greater discrimination capabilities. However, it is undeniable that our model has a preliminary understanding of the meaning of attributes and colors, and can associate them with related image regions, rather than simple memorization. This ability to distinguish between different parts of the person body guarantees the superior performance on the subsequent person-centric tasks.
2.2 Image-guided Attributes Prediction
The IAP task requires utilizing original color images to predict the masked attribute phrases in textual descriptions. Unlike previous methods [42, 95] that randomly mask any words or only color words in description, our method focuses on masking attribute phrases. For each sentence, the attribute phrases are partially masked to create a masked textual description. In this multi-modal masked language modeling (MMLM) way, the correlation between images and texts can be bridged more in depth and more discriminative representations can be learned, The reason that the prediction process enables the model to further understand the person- centric regions in images and extract the key semantic information. Meanwhile, by exploiting the visual representations, the MMLM enhances the perception of context and strengthens the interaction between vision and language modality.
In the encoding stage, as illustrated in Fig. 2, for pair of a color image Tostor and masked textual description tyaaked with masked words Wm {W {Wmi..... wms } (M is the number of masked words), we feed them to respective encoders to extract the visual global embedding V global and textual hidden-state outputs h, {hr,.... b., (L is the length of the tokens in the masked textual description). The visual global embedding V globat is obtained from a pooling operation on the last stage feature output of the visual encoder.
In the decoding stage, to perform the IAP task, we specially design a simple but effective mutil-modal fusion module that mainly consists of self- blocks and prediction head. Firstly. we adopt the element-wiss summation of h, and V global as the preliminary fused embeddings. Then, the embeddings will be served as query(Q), key(K.) and value())) simultaneously. Finally, we obtain the multi-modal fused embeddings for each masked position by:
fmm 1M = Blocks V).
(2)
where M is the total number of masked tokens in the masked textual description, and Blocks are the self-attention blocks.
For each embedding representing the masked word, we use prediction head realize the correspond- ing probability prediction. The IAP can be optimized by minimizing the negative log- likelloo:ood::

---

NV
M
Liop
log P( P(Wmy, (mm));
MN
where M is the total number of masked words in masked textual description, N denotes the number of samples within batch and P denotes the probability distribution mapping.
2.3 Identity-based Vision-language Contrast
In order to further strengthen the correlation between vision and language modalities, we must optimize the model to learn unified cross-modal feature space. A preliminary approach based on contrastive learning [72] is to shorten the distance the representations in image-text pairs and simultancously amplify the distance of representations not in the same pair. This approach considers image-text pairs as instances and ignores the identity, where different image-text pairs of the same identity are treated negative samples. However, in person-centric field, the identity plays crucial role distinguishing different people. Therefore, unlike the usual practices in general domain, we must take the identity into consideration.
For a group of a color image, a complete textual description and a corresponding identity {icolor, Ecompletes Id]. The color image is firstly fed into the visual encoder. Then the last-stage feature pooled to get the visual global embedding global- The description is directly fed into the textual encoder and the average pooling of hidden-states will be served as the textual global embedding T globblt-
Then, given batch of N image-text pairs, for each visual global embedding Vi globat we construct set of visual-textur embedding pairs {(V) globat T )looaa) Yt. J==1+ where yı means that the pair is matched and from the same identity, and Vi.j = 0 indicates the unmatched pair. Let sim x, z) = x z/ ||||||||| denotes the matching probability of and z. Then, the probability of matching pairs can be calculated with the following function:
exp
(sim
(V
T globaa
PV,)
(4)
E kml
exp
sim
Vi glabal
The global
Then, the IVLC loss from vision to language in a batch can be computed by:
N
N
Pi.j 9b. +
Le21
Psy
log
(5)
N
where Bi / Vi. is the true matching probability and small number to avoid numerical problems.
Similarly, the IVLC loss from language vision Liav can be be computed by exchange Vatobal and T globbal in above equations. Finally, the IVLC task can be optimized by:
LIVIC = Loz1 + Lizes
(6)
Indeed, the IVLC loss can be any cross-modal contrastive loss that takes identity into consideration such as the Mooss-Modal Projection Matching (CMPM) loss [112] we adopt, which promotes the representation association between multiple modalities by incorporating the cross-projection into KI. divergence.
Then, to supervise the model to learn discriminative and highly generic person representations, the overall multi-task loss C is computed as:
L Cinte + Ctic + AgCiaps
where \1, A2 E I are hyper-parameters control the importance of each pretext task.
5

---

1.4 worinn with long black har weoring long black oeat, Maakh loasings and block hests.
2.4 man in his mid oventies with short blakk
2.4 man secering a block shirt, pair like
hair i searfing lemon colour t shirt pver
jeans and par black shoes.
blace storts. is carry rng bad sockkeck
2.A midid eageed Asan man with
2.A woman twertbee straight long weating long black cost She is ring a pair reggings
strape vtutble un t-stet sover rg canwas bes with
recading haatkin w
sale.
black col aned shirt gite pearn.
weading green
wearing
pair
somethingg
green
jears.
She black
shorts, wecaring carrying backkpck..
Figure 3: Visualization of some examples in our SUNTH-PEDER dataset.
3 SYNTH-PEDES: A Large-scale Image-text Person Dataset
We build the SYNTH-PEDES dataset to pre-train our PLIP models at a large-scale. In this section, we show the general process of constructing our SYNTH-PEDES dataset, which can be described as three steps. The complete construction details can be found in Sec. A.5 of the appendix.
Firstly, we collect and process two large-scale person datasets to form the image dataset. The first is LUPerson-NL [30]. It is a new variant of LUPerson [29] on top of raw videos from LUPerson and assign the noisy labels to each person image with automatically generated tracklet. It consists of 10M images with about 430K identities collected from 21 scenes. The second is LPW [83]. It consists of 2,731 different persons and 592,438 images collected from three different crowded scenes.
Secondly, we propose an image captioner named SPAC to generate satisfactory textual descriptions for the images. Given an input person image, there is no specific work targeting at generating captions that detailedly describe the person's 's appearance. To this end, we propose a simple but effective method for person image captioning. It can generate attribute-annotationss and stylish eextual-deccrptionns which simulate the diverse perspectives that different annotators may have on the same person image. The specific technics for SPAC can be found in Sec. 1.5.1 of the appendix.
Thirdly, we adopt some post-processing approaches to eliminate the noises and improve the dataset quality. We propose Seed Filter Strategy to filter the noises in LUPerson-NL, which includes three processes of Filter-out, Reassignment and Merger. Meanwhile, we propose Data Distribution Strategy to ensure the quality of generated attributes, the consistency of gender annotation, and identity distribution balance. The specific details for the strategies can be found in Sec. A.7 of the appendix.
Thanks to the outstanding generating ability of our proposed SPAC, the SYNTH-PEDES dataset is full of high-quality textual deseriptions in variety of styles, which can be utilized to train the representation learning model. Compared with existing person datasets, SYNTH-PEDES has the following advantages:
Diversified. Our dataset contains a wide range of variations in the textual descriptions. Unlike the previous person datasets with only or two image-text pairs, most images of our dataset are annotated with three textual descriptions.
High-quality. As some typical qualitative examples can be seen in Fig. 3, the generated annotations achieve an accurate and detailed description of the person appearance. The further experiments conducted on the dataset quality evaluation can be found in Sec. A.9.2 of the appendix. Researchers can use this dataset with confidence to conduct relevant studies.
Large-scale. In Tab. 11 of the appendix, we have compared the properties of SYNTH-PEDES with other popular person datasets. As we can see, SYNTH-PEDES is the largest real person dataset with high-quality image-text pairs by far, which contains 312,321 identities, 4,791,711 images, and 12,138,157 157 textual descriptions.
4 Experiments
During the training of PLIP, we adopt four types of backbone as the visual encoder, i.e., ResNet50, ResNet101, ResNet152 52 and Swin Transformer Base. The pre-trained BERT [23] is utilized as the textual encoder with the last 5 layers unfrozen. We train our model on 4 Geforce 3090 GPUs for 70 epochs. For each person cessoicc downstream task, we reproduce a range of state-of-the the art methods as the baselines. If not specially stated, we perform the experiments by just replacing the backbone in each baseline to the our pre-trained models. We perform in-depth experiments on eleven datasets for five downstream person-centrie tasks. Meanwhile, we perform thorough ablation studies and analyses in Sec. A.9 of the appendix. More details and experiments can be found in the appendix.
Implementation.

---

Table 1: Performance comparisons of our PLIP with some other SoTA pre-train pre-trained models on downstream tasks. The table is divided into two parts: the upper part covers some general-domate pre-trained models and the lower part focuses on some person-domain pre-trained models. The baseline downstream methods for the five tasks are CMPM/C 12, ABDNet [9]. Rethinking [41], SeqNet [54] and SCHP [50], respectively. Due to the inability of certain non- non-hierarchical models to
be applied to some downstream methods, their performance results are indicated as
Mechod
Backhone
CUBK
Tou Rc-ID
ICPG
Marke
Image Re ID
Dake
FETA
Alsibaze Roseg.
BATGOX
SYSU
PePiu Search
PBW
LIP
Haman Punsing
PASCAL
Bodlin:
BS90
4711715.8
18 D
75 75,99915
RT S1SW56
RUZUNT 300
54 54.39157
59.5
71.45
Badin::
BSL.1
56 75
51.76/77.22
59 19.395.7
70.5833
85. G.1775..4 44
81.0157 62
95. 15.199.11
48-487.7
60.57
72.12
Badlne:
RS4522
53.1379.188 34
80
90.18900
65.8186.66 64
82. 17:57777 96
95.396.3
48.837.9
61.71
72.87
MoCwet (17)
RN90
52.1281.5 84
46.31734.50
87 28500
77.387.7
K7.87055
1315 56
94 94.104.5
46,286.1
57.63
69.16
McCwol (17)
55200265
46 60
---
K1 51.51855 96
796605710 13
-F-
MoCand (17)
VIBB
55 83
90.2677632
-
64.9286.5 83
80 80.9057.30 24
CLIP
RN30
37908512
St367764
5o 595.7
843086655
810075785 RT
92.5054
4K08867
30
70.33
CLIP
RS
52,8079.31
90.696.0
81,489.9
81.62/57.05
93.3003.8
48.5
60.74
72.30
CLIP
VITB
58.0106 21
53.7379.88
86.37197.80 05
82.41/88.4
RL (RV)
MARIRA 39
856 46
REIP
MEE
43.21.88
87.0288
83.23/5
KN30
17518436
03337663
64,17756110
5941
71.89
RNIDI
573495
50087699
4
6041
(19)
49.0000
71
83.900.9
RN8O
37868407
So RARR
8T.1800
03 4005
40
3988
LUP-NL (30)
RNIDI
13
83.8675.85
4866
62.10000
65-77866
95
49
60.92
72.51
LUP-NL
RNIS2
29.278800
542189012
91 5.69965
51.50006
86.1829300 09
61.00
73.12
312
6.30720 16
"SSTTAFOT
56 8566
3:30
SOE MAR [1]]
Swi-8
56 414444
197795
59 1988S
99-2280-1 7
63 63-42899
95
9AR9S5 7
69:21
69.81
SOL COLDER (13)
Skin-8
580663364
60.37/9042
50 6
81.2/00.3
65.7687 46
85.9199966
94 040095
59 7996
60.50
70.07
PLIRooiss
RS30
64.8861.2 22
91 199.88
K5.1296994
62.204866
96.0962
5559889
80031
72.14"
PLIPoours)
RNIDI
12 32.36933
689220 06
92 1990 9
K2. 20.3011. 8
K3 2620 42
82.5258 16
529999.4 4
01.32
72.61
PLIP(cur)
RNI52
23.6894222
63 63.52288.40
92.697.1
63.1892
86.28888 17
82.9158.83
96.59770
55. 55.799..
62.44
73.51
PLIP(ous))
SWIn
75.3699487
66.1778.994
93.2997.3
$4,492.2
87.1228.84
93.03999113
96.49667
56.69999
63.52
73.93
4.1 Comparison With Other Pre-trained Models
We have compared the performance of our PLIP pre-trained models with other SoTA pre-trained models [17, 29, 30, 12, 72, 49, 106] on five downstream tasks. We reproduce a range of popular and open-sourced methods [112, 41, 54, 50] as the baselines, which are initialized by ImageNet supervised pre-trained ResNet backbones. Then, we compare the performance of different pre- pre-trained models by simply replacing the backbone in each method. The compared models can be divided into general-domain pre- pre-trained models and person-domain pre-trained models, with the latter typically exhibiting better performance in person-centric tasks. The performance metrics for these tasks @@ 1/R @ 10, mAP/R @1 1, mAP/F1, mAP/R @1 and mloU, respectively. As shown in Tab. 1, our PLIP models consistently demonstrate a significant performance advantage over other pre-trained models in fair comparison with roughly equal parameters.
supervisea
4.2 Evaluation on Text-based Person Re-ID
Transfer capability. To evaluate the transfer ca- pability of our pre-trained models, we conduct three different experiments with ResNet50 as the visual encoder. Firstly, we directly evaluate the model's ' zero-shot performance without any extra fine-tuning. Secondly, we perform linear probing by adding a trainable linear embedding layer to each modal frozen encoder. Finally, we unfreeze all encoders and perform fine-tuning. We use the simple CMPM loss [112] as the train- ing target. From Tab. 2 we can see, our pre-
Table 2: The results of our transfer experiments. We show the best score in bold. z-s: zero-shot setting; I-p: lineargprobing setting; f-r: fine-tune setting. 8 stands for the results reproduced with public checkpoints released by the authors.
CUHK-PEDES
KCRG-PEDES
Method
R@1
RØ5
R 10
R
RW5
RØ 10
VITAA [93]
55.97
75.84
83.52
50.98
68.79
75.78
SSAN [24]
61.37
80.15
86.73
54.23
72.63
79.53
LapsCorc [95]
63.40
87.80
TIPCB1 [18]
63.63
82.82
89.01
54.96
74.72
81.89
LGUR [78]
81.94
87.93
57.42
74.97
81.45
trained model not only competitive with some fully supervised methods [93, 24] even without fine-tune training, but also greatly exceeds them by a large margin with a simple fine-tune. These results demonstrate that our pre-trained models have excellent transfer capability for this task.
PLIP+3-
52.86
74.69
82.46
49,86
68.78
76.27
PLIP+LPP
63.63
82.85
89.36
58.51
77.83
84.24
PLIP464
70.11
86.60
91.89
64.58
81.30
86.82
Table 3: Comparison on domain generalization.
"C" and "I" denote CUHK-PEDES and ICFG- PEDES, respectively.
C+1
I-C C
Domain generalization. To verify our models' domain generalization capability, we carry out experiments with cross-domain settings. We use the CMPM loss as the training target. As illus- trated in Tab. 3, our model achieves improve- ments by large margins when compared with all
Method
R&1
RG5
R6
R@1
R
RW
Dual Path [118]
29.80
38.19
7.63
17.14
23.52
SCAN [43]
21.27
39.26
48.83
13.63
28.61
37.05
SSAN [24]
29.24
49.00
58.53
21.07
38.94
48.54
LGUR 1781
52.58
60.85
25.44
$4.39
Rasa [I]
50.59
67.46
74.09
50.70
72.40
79.58
PLIP
56.64
75.65
82.38
57,34
77.60
84.49
7

---

other methods. Specifically, our model outperforms LGUR by 22.4% and 31.9% in terms of Rank- 1 metric on the C-+I and I-+C settings, respectively. These results demonstrate that our pre-trained models have great ca capability in domain generalization for this task.
Improvement over existing methods. We reproduce three representative baseline meth- ods [112, 24, 78] and explore the performance difference by changing the encoders with differ- ent pre-trained models. From Tab, 4, we can see
Table 4: Comparison on three methods by using different pre-trained models. All results are shown
in 1aank-1/Rank-100
Pre-train
CMPMIC [112]
SSAN [24]
LGUR [78]
Baseline.
54.81/83.22
61.37/86.73
64.21/87.93
ICFG-PEDES CUHK-PEDES
that equipped with our pre-trained model, all the
MoCov3
52.17/81.94
61.97/86.63
65.33/88.4 47
baseline methods achieve higher and best accu-
CLIP
57.92/85.12
62.09/86.89
64.70/88.76
racy on each dataset. It is worth noting that, due
LUP
57.51/84.86
63.91/88.36
65.42/89.36
LUP-NL
57.85/84.97
63.71/87.46
64.68/88.66
to the fact that our PIIP models have already
PLIP
71.13/2.81
65,90/89.44 .4
67.62/89.90
learned an excellent joint visual-textual feature space through lcrge-scale pre-training, achiev-
Buseline.
47.61/75.48
54.23/79.53
57.42/81445
MoCov3
46.51/74.86
55.27/79.64
59.90/82.08
ing outstanding performance is easily attained by fine-tuning with the simple CMPM/C [112]
CLIP
51.30/77.64
53,58/78.96
58. 58.35/82.02 02
LUP
50.52/76.63
60.33/83.00 56.51/80.41
LUP-NL
50.64/76.55
55.59/79.78
60.25/82.84
loss rather than complicated designing such as
PLIP
64.88/87.32
60.70/83.17
62.38/84.28
SSAN [24] and LGUR [78].
Table 5: Comparison with the state-of-the-art meth- ods on text-based person Re-ID. We show the best score in bold.
Comparison with state-of- ofett-rrt methods. In Tab. 5, we compare our results with some SoTA methods on each dataset. The compared meth- ods can be classified based on whether they rely on the multi-modal pre-traines models. Gener- ally, multi-modal pre-trained models can bring about noticeable performance improvements for this task. It is worth noting that RaSa utilizes larger image resolution of 384x384, while APTM adopts a two-stage inference method similar to the re-rank mechanism. These ap- proaches lead to RaSa and APTM achieving op- timal performance, however, introducing addi- tional training and inference costs. Instead, our PLIP achieves competitive performance without bells and whistles. Specifically, with ResNet50 as backbone, PLIP outperforms LGUR by 6.9% and 7.5% rank-1 on each dataset respectively.
Method
Image
CUHK-PEDES R
R@1
ICEG-PEDES
10
GINA-RNN [
VCRG
1ST1
19.05
$3.64
CMPMIC [112]
RN50
LSTM
49.37
79.27
43.51
65.64
74.26
VIDAA 1931
RN30
LSIM
53.97
75.84
05.52
50.98
08.79
75.78
NAPS [3]]
RN50
BFRT
59.94
70.86
86.70
SSAN [24]
RN30
LSTM
61.37
$0.15
$6.73
54,23
72,63
79.53
LapoCtore [95]
RN30
8ERT
63.40
$7.80
TIPCB: [18]
RN30
BERT
$1.94 63.63
82.82
89.01
54.96
74.72
81.89
LGUR
RN30
BЕKT
87.93
57.42
74.97
81.45
IVT12]
WT
BERT
60:22
CHine [108]
VIF-8
BERT
55.93
91.15
82.42
IRRA [47]
VTTBB
XGomer
73.38
80.93
93.71
63.46
80.25
85.80
APTM [106]
Swin-B
BERT
76.17
80:47
93.57
68.22
80.87
K7.50
RaSa [I]
WT
DERT
76.51
90.29
91,25
65.28
80.40
85.12
PLIP
RN30
BERI
1.11
87.37
92.81
04.35
31.70
8733
PLI
RN101
BEKT
59.15
93.46
64.92
87.06
RN1522
04.22
65.52
82.64
Swin-B
67.17
Table 6: Comparison on two baseline methods by using different pre-trained models. The best results are shown in bold.
4.3 Evaluation on Image-based Re-ID
Pre train
MAP
RSFT
MarsedISO
RBF5
MAY
RIFT
NISMITZ
1005
RT0
Bascline
81.3
12.55
17.1
95.1
31.4
61.11
73.4
17.8
Unsupervised methods achieve significant im- provements. With simply replacing the back- bone, our pre-trained models benefit unsuper- vised image-based person Re-ID methods signif- icantly. We evaluate the improvement brought by different pre-trained ResNet50 models to the SoTA unsupervised methods PPLR [20] and ISE [110]. As shown in Tab. 6, PLIP outper- forms all other pre-trained models by a large margin. Specifically, applied to ISE, PLII achieves new SoTA performance, outperforming the previsous SoTA by 2.9% and 11.4% mAP on 550 and MSMT17, respectively.
McCov2 [1:5]
79.6
91.6
96.5
97.9
26.7
57.5
69.6
74.66
CLIP (72)
75.0
89.0
95.5
97.1
7.6
20.3
29.9
34.9
LUP [29]
57.5
75.2
$5.9
88.9
22.5
48.9
62.0
56.9
LUP NL [10]
84.2
93.4
99:7
98.6
25.0
50.3
65.1
68.7
PLIP
86.6
94.5
96.0
98.7
46.1
73.4
83.7
$7.0
Bascline
BL.7
BLU
97.3
93.
33.00
64.77
753
15.4
McCov2 [1:5]
84.9
93.5
97.5
98.7
34.1
64.5
75.0
79.0
ISE
CLIP (72)
79.5
92.0
96.8
98.0
14.9
35.2
46.3
51.3
LUP (29)
81.5
94.2
91.8
98.3
27.8
56.7
68.8
73.4
LUP-NE (10)
87.4
95.0
98.2
98.9
33.9
62.5
75.2
77.5
PLJPP
W7.6
05.3
96.2
98.9
46.4
74.9
84.3
61.8
Table 7: Comparison with SoTA methods on fully supervised image-based person Re-ID. The best results are shown in bold.
Method
Seilings
Marketl500 550
DuscMTTMC
Backbone
Pretrain
mAP
MGN [89]
RN50
IMG
87.5
95.1
79.4
89:0
ABDNer [9]
RX50
IMG
58.3
95.6
78.6
89.0
[69]
RN50
IMG
78.6
87.9
Comparison with state-of-the-art meth- ods. We compare our results with exist- ing SoTA image-based person Re-ID methods on Market1 501 and DukeMTMC. Any results gained from post-processing techniques like re- rank [121] are excluded for a fair comparison. As indicated in Tab. 7, by applying our PLIP
ISP [129]
RNS0
IMG
88.6
95.3
80.0
89.6
TD [38]
VIT-B
IMG
58.2
95.0
80.6
89.6
UPRaD T [i
RS:50
"LOF'
S1.
'97.1
[29]
RN50
LUP
96.4
82.1
91.0
LUP-NL (30)
RNS0
LUP-NL
91.9
96.6
84.3
92.0
PASS [124]
VIT-B
LUP
93.0
96.88
FLIP
RNS0
SYNTH
BT.7
91.1
PLIP
RN101
SYNTH
92.0
96.9
82.3
91.8
PLIP
RN152
SYNTH
92.6
97.1
83.1
92.1
PLIP
Swin-B
SYNTH
93.2
97.3
84.4
92.2
8

---

on ABD-Net [9], we achieve competitive per- formance on all datasets. Moreover, we achieve new SoTA performance with Swin-Base as the backbone. This demonstrates that the learned representations benefits this task significantly.
Table 8: Comparison on two baseline methods by using different pre-trained models. We show the best score in bold.
Pre unin
mAP
CURK-SYSL
R:055
R
R:01
PXW
R
R10
Rieeline
94.8
08.1
98.7
37.6
94.4
SayNei [54
McCoe2 [U5]
94.0
94.4
98.2
98.5
48.3
87.3
94.5
95.8
4.4 Evaluation on Person Search
C13P [?2]
92.8
91.4
97.8
08.2
48.0
86.7
94.5
95.0
1 [2]
94.3
94.9
98.2
98.7
47.7
87.3
94.5
95.7
Improvement over existing methods. Our pre- trained models bring significant performance gain to existing person search methods. To verify this, we choose two representative meth-
LUPNL
95.4
58.8
87.5
Biseline
95.3
96.2
98.3
58.8
47.5
87.8
541.5
95.3
MoCon2 [15]
95.1
95.6
98.0
98, 4
18.4
67.8
$41.5
95.9
GILONN
CLIP [?2]
93.0
93.6
97.7
96.1
48.2
87.1
94.5
96.0
1 [2]
48.
NL [30]
ods SeqNet [54] and GLCNet [70] as the base- lines, and evaluate the improvements brought by different pre-trained models. As shown in Tab. 8, under the ResNet50 setting, our model brings maximum performance improvements to all methods. Specifically, when applying our pre-trained ResNet50 to GLCNet, it achieves 96.3% and 53.7% mAP on the CUHK-SYSU and PRW datasets, respectively.
PLTP
96.3
97.0
99.0
99.3
53.7
89.0
95.4
96.4
Table 9: Comparison with state-of of-the-art art methods
on the end-to-end person search. We show the best score in bold.
Method
Backhone
CUHK-SYSU
PRW
mAP
R@1
mAP
R@Y
OIM [98]
RN50
21.3
49.9
NPSM [59]
RN50
77.9
81.2
24.2
53.1
CTX [104]
RN50
84.1
B6.5
33.4
73.6
QEEPS [67]
RN50
88.9
89.1
37.1
76.7
BINet [25]
RN50
90.0
90.7
45.3
81.7
NAB [5]
RN50
92.1
92.9
44.0
81.1
Comparison with state-of-the-art methods. We compare our results with existing SoTA per- son search methods in Tab. 9. By applying our pre-trained models on GL.Net [70), we achieve new SoTA performance on each dataset. Specif- ically, under ResNet50 setting, our method sur- passes the previous SoTA GLCNet by 5.9% mAP on PRW dataset. Also, with Swin-Base as the backbone, mance compared with all other methods. These results demonstrate that our pre-training frame- work shows great potential in learning discrim- inative person representation for this task.
SegNet [54]
RN50
94,8
95.7
47.6
87.6
GLCNett [20]
RNS0
95.8
96.2
47.8
87.8
SOU IDER [12)
Swin-B
86.8
PLIP
RN50
96.3
97.0
53.7
89.0
PLIP
RN101
96.5
97.2
55.1
89.4
PLIP
RN152
96.7
97.3
56.2
89.9
PLIP
Swin.B
96.6
97.5
57.8
90.3
Table 10: Ablation study on the effectivenss of each pretext task, all using default settings.
achieve the best perfor-
TVI4 TIC
Compoments
IAP
Rel
CURK-PEDES
Ras
R810
Ria
Markel
Res
2022
33.3
64.0
62.3
79.9
85.7
314
58.2
68.9
62.9
80.7
86.2
30.5
62.7
39.3
61.4
70.4
32.5
56.3
66.6
63.1
89.8
86.3
Text- Person Re-ID
Image based Prrson Ro
75
95
4.5 Ablation studies and analyses
am
a Bross B Rosoc
We perform ablation studies with ResNet-50 as the visual encoder and pre-training on the sub- dataset of SYNTH-PEDES which has 10,000 identities, 139,564 images and 353,617 textual descriptions. To assess the effectiveness of pre- text tasks on the generalizability our models, we directly evaluate the zero-shot performance of pre-trained models on downstream datasets. As we can see in Tab. 10, each task contributes to the model's zero-shot capability and combining all of the tasks leads to the best performance. Meanwhile, to validate the effectiveness of textual diversity in SYNTH-PEDES, we have studied four different degrees of textual diversity from weak to strong. As shown in Fig. 4, the fourth case with the highest degree of textual diversity has the best performance. More thorough ablation studies and analyses about effectiveness of each component, dataset quality evaluation, pre-training settings and on can be found in Sec. A.9 of the appendix.
55
75
35-
65
35
PEDES
CURK-FEDES
Drdber TMC
Morwet500
Figure 4: The diversity of textual descriptions mat-
ters. PC and GC mean prompt caption and gener- ated caption, respectively.
5 Conclusion
In this paper, we propose a novel language-ing -mage self-supervised person representation learning framework named PLIP, which consists of three well-motivated pretext tasks. Also, we build large-scale real-scenario image-text person dataset SYNTH-PEDES by auto-captioning procedures. We achieve good generic person representation learning by utilizing PLIP and SYNTH-PEDES. Equipped with our pre-trained models, we push many existing methods to much higher level without bells and whistles. We hope that our simple and effective framework can inspire researchers to devote further attention to this area.
9

---

Acknowledgements. This work was supported by the National Natural Science Foundation of China No.62176097, and the Hubei Provincial Natural Science Foundation of China No.2022CFA055.
References
[1] Yang Bai, Min Cao. Daming Gao, Ziqiang Cao, Chen Chen, Zhenfeng Fan, Liqiang Nie, and Min Zhang. Rasa: Relation and sensitivity aware representation learning for text-based person search. arXiv preprint arXiv:2305. 13653, 2023.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.. 082544 2021.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, pages 9650-9660, 2021.
[5] Di Chen, Shanshan Zhang. Jian Yang, and Bernt Schiele. Norm-aware embedding for efficient person search. In CVPR, 2020.
[6] Di Chen, Shanshan Zhang, Jian Yang, and Bernt Schiele. Norm-aware embedding for efficient person search. In CVPR, pages 12615-12624, 2020.
[7] Liang-Chieh Chen, Yi Yang. Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale:
Scale-aware semantic image segmentation. In CVPR, pages 3640-3649, 2016.
[8] Tianlang Chen, Chenliang Xu, and Jiebo Luo. Improving text-based person search by spatial matching and adaptive threshold. In WACV, pages 1879-1887. IEEE, 2018
[9] Tianlong Chen, Shaojin Ding, Jingyi Xie, Ye Yuan, Wuyang Chen, Yang Yang, Zhou Ren, and Zhangyang Wang. Abd-net: Attentive but diverse person re-identification. In ICCV, pages 8351-8361, 2019.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[11] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet network for person re-identification. In CVPR, pages 403-4122 2017.
[12] Weihua Chen, Xianzhe Xu, Jian Jia, Hao Luo, Yaohua Wang, Fan Wang, Rong Jin, and Xiuyu Sun. Beyond appearance: semantic controllable self-supervised learning framework for human-centric visual tasks. In CVPR, 2023.
[13] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pages 1979-1986, 2014.
[14] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03022 2022.
[15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 042997 2020.
[16] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR,
pages 15750-15758, 2021.
[17] Xinlei Chen Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint 1rXiv:2104.00057 2021.
[18] Yuhao Chen, Guoqing Zhang, Yujiang Lu, Zhenxing Wang, and Yuhui Zheng. Tipcb: A simple but effective part-based convolutional baseline for text-based person search. Neurocompating,
494:171-181, 2022.
[19] Xinhua Cheng, Mengxi Jia, Qian Wang, and Jian Zhang. A simple visual-textual baseline for pedestrian attribute recognition. TCSV7, (1))6699947 2022.
[20] Yoonki Cho, Woo Jae Kim, Seunghoon Hong, and Sung-Fui Yoon. Part-based pseudo label refinement for unsupervised person re-identification. In CVPR, pages 7298-7308, 2022.
10

---

[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248 255. Iece, 2009.
[22] Yubin Deng, Ping Luo, Chen Change Loy, and Xiaoou Tang. Pedestrian attribute recognition at far distance. In ACMMM, pages 789-792, 2014.
[23] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers).
[24] Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao. Semantically self-aligned net- work for text-to-image part-aware person re-identification. arXiv preprint arXiv:2 (tt:::207..2000 12666, 2021.
[25] W. Dong, Z. Zhang, C. Song, and T. Tan. Bi-directional interaction network for person search. CVPR, 2020.
[26] Wenkai Dong, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan. Instance guided proposal network for person search. In CVPR, pages 2585-2594, 2020.
[27] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint (xXKv:/707.0550 2017.
[28] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 32(9)::627-1655 2009.
[29] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen. Unsupervised pre-training for person re-identification. In CVPR, pages 14750-14759, 2021.
[30] Dengpan Fu, Dongdong Chen, Hao Yang, Jianmin Bao, Lu Yuan, Lei Zhang, Houqiang Li, Fang Wen, and Dong Chen. Large-scalt pre- pre-training for person re-identification with noisy labels. In CVPR, pages 2476-2488 2022.
[31] Chenyang Gao, Guanyu Cai, Xinyang Jiang, Feng Zheng, Jun Zhang, Yifei Gong, Pai Peng. Xiaowei Guo, and Xing Sun. Contextual non-local alignment over full-scale representation for text-based person search. arXiv preprint arXiv:2101.03033 2021.
[32] Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, and Liang Lin. Graphon- omy: Universal human parsing via graph transfer learning. In CVPR, pages 7450-7459, 2019.
[33] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In CVPR, pages 6757-6765, 2017.
[34] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. NIPS, 33:21271- 21284, 2020.
[35] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[38] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In ICCV, pages 15013-15022, October 2021.
[39] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- ieiiniiiiiaaion.. arXiv preprint arXiv:1703.07733 2017.
[40] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and -anguala representation learning with noisy text supervision. In ICLR, pages 4904-4916, 2021.
11

---

[41] Jian Jia, Houjing Huang, Xiaotang Chen, and Kaiqi Huang. Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting. arXiv preprint 2021.
[42] Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to- image person retrieval. In CVPR, 2023.
[43] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pages 201-216, 2018.
[44] Dangwei Li, Xiaotang Chen, and Kaiqi Huang. Multi-attribute learning for pedestrian attribute recognition in surveillance scenarios. In ACPR, pages 111-115. IEEE, 2015.
[45] Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang. A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. 7IP, 28(4): 1575-1590, 2018.
[46] He Li, Mang Ye, and Bo Du. Weperson: Learning a generalized re-identification model from all-weather virtual data. In ACMMM, 2021.
[47] Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan. and Jiashi Feng. Multiple-human parsing in the wild. arXiv preprint arXiv:1705.07206, 2017.
[48] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023.
[49] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICML, pages 12888-12900, 2022.
[50] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. TPAMI, 44(6):3260 3271, 2022.
[51] Qiaozhe Li, Xin Zhao, Ran He, and Kaiqi Huang. Visual-semantic graph reasoning for pedestrian attribute recognition. In AAAI, volume 33, pages 8634-8641, 2019.
[52] Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural language description. In CVPR, pages 1970-1979, 2017.
[53] Wanhua Li, Zhexuan Cao, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Label2label: A language modeling framework for multi-attribute learning. In ECCV, pages 562-579. Springer, 2022.
[54] Zhengjia Li and Duoqian Miao. Sequential end-to-end network for efficient person search. In AAAI, volume 35, pages 2011-2019, 2021.
[55] Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin. Look into person: Joint body parsing & pose estimation network and a new benchmark. TPAMI, 41(():871-885, 2018.
[56] Shengeai Liao and Ling Shao. Interpretable and Generalizable Person Re-Identification with Query-Adaptive Convolution and Temporal Lifting. In ECCV, 2020.
[57] Shengeai Liao and Ling Shao. TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification. 2021.
[58] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick. Kaiming Hc, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125, 2017.
[59] Hao Liu, Jiashi Feng, Zequn Jie, Karlekar Jayashree, Bo Zhao, Meibin Qi, Jianguo Jiang, and Shuicheng Yan. Neural person search machines. In ICCV, pages 493-501, 2017.
[60] Kunliang Liu, Ouk Choi, Jianming Wang, and Wonjun Hwang. Cdgnet: Class distribution June 2022.
guided network for human parsing. In CVPR, pages
[61] Pengze Liu, Xihui Liu, Junjie Yan, and Jing Shao. Localization guided learning for pedestrian 2018.
attribute recognition. arXiv preprint
[62] Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Junjie Yan, and Xiaogang Wang. Hydraplus-net: Attentive deep features for pedestrian analysis. In ICCV, pages 1-9, 2017.
[63] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2017.
[64] Yawei Luo, Zhedong Zheng, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Macro-miero adversarial network for human parsing. In ECCV, pages 418-434, 2018.
12

---

[65] Zhou Luowei, Palangi Hamid, Zhang Lei, Hu Houdong, Jason J. Corso, and Gao Jian- feng. Unified vision- language pre-training for image captioning and vqa. arXiv preprint arXiv:1909. 1059, 2019.
[66] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.097.3 2021.
[67] Bharti Munjal, Sikandar Amin, Federico Tombari, and Fabio Galasso. Query-guided end-to- end person search. In CVPR, pages 81 811-820, 2019.
[68] Kai Niu, Yan Huang, Wanli Ouyang, and Liang Wang. Improving description- basedd person re-identification by multi- -uulti-grauularity image-text alignments. TIP, 29:5542-5556, 2020.
[69] Hyunjong Park and Bumsub Ham. Relation network for person re-identification. In AAAI, volume 34, pages 11839-11847 2020.
[70] Jie Qin, Peng Zheng, Yichao Yan, Quan Rong, Xiaogang Cheng, and Bingbing Ni. Movienet- ps: A large-scale person search dataset in the wild. In ICASSP, 2023.
[71] Zhiwu Qing. Shiwei Zhang, Ziyuan Huang, Yi Xu, Xiang Wang, Changxin Gao, Rong Jin, and Nong Sang. Self-supervised learning from untrimmed videos via hierarchical consistency. TPAMI, 2023.
[72] Alec Radford. Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[73] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[74] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, pages 7263-7271, 2017.
[75] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NIPS, 28, 2015.
[76] Tao Ruan, Ting Liu, Zilong Huang, Yunchao Wei, Shikui Wei, and Yao Zhao, Devil in the details: Towards accurate single and multiple human parsing. In AAAI, volume 33, pages 4814-4821, 2019.
[77] Nikolaos Sarafianos, Xiang Xu, and Ioannis Kakadiaris. Adversarial representation learning for text-to-image matching. In ICCV, pages 5814-5824, 2019.
[78] Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, and Changxing Ding. Learning urfifed representations for text-to-image person re-identification. In ACMMM, pages 5566,5574, 2022.
[79] Yantao Shen, Hongsheng Li, Shuai Yi, Dapeng Chen, and Xiaogang Wang. Person re- identification with deep similarity-guided graph neural network. In ECCV, pages 486-504, 2018.
[80] Jiangming Shi, Xiangho Yin, Yeyun Chen, Yachao Zhang, Zhizhong Zhang, Yuan Xie, and Yanyun Qu. Multi-memory matching for unsupervised visible-infrared person re-identification. arXiv preprint arXiv:240) 066225 2024.
[81] Jiangming Shi, Yachao Zhang, Xiangbo Yin, Yuan Xie. Zhizhong Zhang, Jianping Fan, Zhongchao Shi, and Yanyun Qu Dual pseudo-lahels interactive self-training for semi- supervised visible-infrared person re-identification. In ICCV, pages 11218-1 11228, 2023.
[82] Xiujun Shu. Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, and Xiao Wang. See finer, see more: Implicit modality alignment for text-based person retrieval. arXiv preprint arXiv:2208. 08608, 2022.
[83] Guanglu Song. Biao Leng, Yu Liu, Congrui Hetang, and Shaofan Cai. Region-based quality estimation network for large-scale person re-identification In AAAI, volume 32, 2018.
[84] Yumin Suh, Jingdong Wang, Siyu Tang, Tao Mei, and Kyoung Mu Lee. Part-aligned bilinear representations for person re-identification. In ECCV, pages 402-419, 2018.
[85] Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu. Lightning- dot: Pre-training visual-semantic embeddings for real-time image-text retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 982-997, 2021.
13

---

[86] Yifan Sun, Liang Zheng. Yi Yang. Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and strong convolutional baseline). In ECCV, pages 480-496, 2018.
[87] Cheng Wang, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen. Tets: A task- consistent two-stage framework for person search. In CVPR, pages 11952-11961, 2020.
[88] Guangrun Wang, Guangcong Wang, Xujie Zhang, Jianhuang Lai, Zhengtao Yu, and Liang
Lin. Weakly supervised person re-id: Differentiable graphical learning and new benchmark. TNNLS, 32 2156, 2020.
[89] Guanshuo Wang. Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. Learning discriminative features with multiple granularities for person re-identification. In ACMMM, pages 274-282, 2018.
[90] Jingya Wang, Xiatian Zhu, Shaogang Gong, and Wei Li. Attribute recognition by joint recurrent learning of context and correlation. In ICCV, pages 531-540, 2017.
[91] Wenguan Wang, Zhijie Zhang, Siyuan Qi, Jianbing Shen, Yanwei Pang, and Ling Shao. Learning compositional neural information fusion for human parsing. In CVPR, pages 5703- 5713, 2019.
[92] Yanan Wang, Shengcai Liao, and Ling Shao. Surpassing Real-World Source Training Data: Random 3D Characters for Generalizable Person Re-Identification. In ACMMM, 2020.
[93] Zhe Wang. Zhiyuan Fang, Jun Wang, and Yezhou Yang. Vitaa: Visual-textual attributes alignment in person search by natural language. In ECCV, pages 402-420, 2020.
[94] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identiffcation. In CVPR, pages 79-88, 2018.
[95] Yushuang Wu, Zizheng Yan, Xiaoguang Han, Guanbin Li, Changqing Zou, and Shuguang Cui. Lapscore: language-guidet person search via color reasoning. In ICCV, pages 1624-1633, 2021.
[96] Fangting Xia, Peng Wang, Xianjie Chen, and Alan L Yuille. Joint multi- perssoneronnnn pose estimation and semantic part segmentation. In CVPR, pages 6779-6778, 2017.
[97] Suncheng Xiang, Zirui Zhang, Mengyuan Guan, Hao Chen, Binjie Yan, Ting Liu, and Yuzhuo Fu. Vtbr: Semantic-based pretraining for person re-identification. arXiv, pages arXiv-2110 10, 2021.
[98] Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detection and identification feature learning for person search. In CVPR, 2017.
[99] Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detection and identification feature learning for person search. In CVPR, July 2017.
[100] Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.066 066777 2022.
[101] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, pages 9653-9663, 2022.
[102] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICMI,, pages 2048-2055 PMLR, 2015.
[103] Shuanglin Yan, Neng Dong, Liyan Zhang, and Jinhui Tang. Clip-driven fine-grained text-image person re-identification. 2023.
[104] Yichao Yan, Qiang Zhang, Bingbing Ni, Wendong Zhang, Minghao Xu, and Xiaokang Yang. Learning context graph for person search. In CVPR, pages 2153-2162, 2019.
[105] Xuezhi Liang Yanan Wang and Shengcai Liao. Cloning Outfits from World Images to
3D Characters for Generalizable Person Re-Identification. CVPR, 2022.
[106] Shuyu Yang. Yinan Zhou, Yaxiong Wang, Yujiao Wu, Li Zhu, and Zhedong Zheng. To- wards unified text-based person retrieval: A large-scale multi-attribuu and language search benchmark. In ACMMM, 2023.
14

---

[107] Zizheng Yang, Xin Jin, Kecheng Zheng, and Feng Zhao. Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification. In CVPR, pages 14278-14287, 2022.
[108] Ye Yuan, Wuyang Chen, Yang Yang, and Zhangyang Wang. In defense of the triplet loss again: Learning robust person re-identification with fast approximated triplet loss and label distillation. In CVPR, pages 354-355, 2020.
[109] Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, and Qi Tian. Unrealperson: An adaptive pipeline towards costless person re-identification. In CVPR, 2021.
[110] Xinyu Zhang. Dongdong Li, Zhigang Wang. Jian Wang, Errui Ding, Javen Qinfeng Shi, Zhaoxiang Zhang, and Jingdong Wang. Implicit sample extension for unsupervised person re-identification. In CVPR, pages 7369-7378, 2022.
[111] Yifu Zhang, Chunyu Wang. Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification multiple object tracking. IJCV, 129:3069-3087, 2021.
[112] Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In ECCV, pages 686-701, 2018.
[113] Xin Zhao, Liufang Sang, Guiguang Ding, Jungong Han, Na Di, and Chenggang Yan. Recurrett attention model for pedestrian attribute recognition. In AAAI, volume 33, pages 9275-9282, 2019.
[114] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. ICCV, pages 1116-1124, 2015.
[115) Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian. Person re-identification in the wild. In CVPR, pages 1367-1376, 2017.
[116] Ruochen Zheng, Changxin Gao, and Nong Sang. Viewpoint transform matching model for person re-identification. Neurocomputing, 433:19-27, 2021.
[117] Ruochen Zheng, Lerenhan Li, Chuchu Han, Changxin Gao, and Nong Sang. Camera style and identity disentangling network for person re-identification. In BMVC, page 66, 2019.
[118] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang. Mingliang Xu, and Yi-Dong Shen. Dual-path convolutional image-text embeddings with instance loss. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(2):1-23, 2020.
[119] Zhedong Zheng, Liang Zheng, and Yi Yang. A discriminatively learned cnn embedding for person reidentification. ACM transactions on multimedia computing, communications, and applications (TOMM), 14(1):1-20 2017.
[120] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In ICCV, pages 3774-3782, 2017.
[121] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. In CVPR, pages 1318-1327, 2017.
[122] Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang
Hua. Dssl: deep surroundings-pertt -prsson separation learning for text-based person retrieval. In
ACMMM, pages 209-217, 20122.
[123] Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang. Identity- guided human semantic parsing for person re-identification. In ECCV. Springer, 2020.
[124] Kuan Zhu, Haiyun Guo, Tianyi Yan, Yousong Zhu, Jinqiao Wang, and Ming Tang. Pass: Part-aware self-supervised pre-training for person re-identification. arXiv preprint 2022.
arXiv:2203.0399
[125] Jialong Zuo, Ying Nie, Hanyu Zhou, Huaxin Zhang, Haoyu Wang, Tianyu Guo, Nong Sang, and Changxin Gao. Cross-vides identity correlating for person re-identification pre- me-itrannnn. arXiv preprint arXiv:2409. 18569, 2024.
[126] Jialong Zuo, Hanyu Zhou, Ying Nie, Feng Zhang, Tianyu Guo, Nong Sang, Yunhe Wang, and Changxin Gao. Ufinebench: Towards text-based person retrieval with ultra-fine granularity. In CVPR. pages 2010-220019, 2024.
15

---

A Appendix
A.1 Related Work
General Representation Learning. To avoid the labor-intensive manual annotation process, many general representation learning approaches have been explored utilize unlabeled image data. Several representative works [36, 101, 10, 34, 16, 2, 35, 4, 14] have achieved performance comparable to, or even surpassing, supervised methods. For example, CAE [14] presents a novel masked image modeling approach and shows the benefit to representation learning through an encoder- regressor-decoder architecture. Besides to pre-training on unlabeled image data, there is currently popular trend of attempting to establish the relationship between vision and language to learn more discriminative representations. For example, CLIP [72] and ALIGN [40] perform cross cross-modal contrastive learning on hundreds thousands of millions of image-text pairs crawled from the web and can directly perform veca-vocayulary image classification. BLIP [49] shows that the noisy web texts are suboptimal for vision-language learning, and proposes Captioning and Filtering method to improve the quality of the text corpus.
Person Representation Learning. In person related field, it is common practice to leverage the backbones pre-trained on ImageNet [21]. which ignores domain gap between general images and person-related images, and leads to limited performance. To address such a problem, LUP [29] constructs a large-scale unlabeled person dataset and makes the first attempt of performing general unsupervised pre-training method MoCov2 [15] for learning person representations. The good experimental results on the person Re-ID task verified the effectiveness of it.
However, simply adopting the general pre-training method may result in the ignorance of person fine- ime-grained characteristics. Therefore, considering the particularity of Re-ID tasks, the proposed UP- RelD [107] introduces an intra-identity regularization and is the first attempt toward Re-ID specific pre-training framework by explicitly pinpointing the difference between the general pre-training and Re-ID pre-training. Before long, LUP-NL [30] develop a large-scale pre-training framework utilizing noisy labels, and demonstrate that learning directly from raw videos is a promising alternative for pre-training, which utilizes spatial and temporal correlations as weak supervision. This simple pre-training task provides a scalable way to learn good Re-ID representations from scratch without bells and whistles. Also, PASS [124] is proposed to several learnable tokens to extract pert- features offering fine-graine fine-grained information. It helps the ViTs set good performance on several person Re-ID datasets. More recently, CION [125] is proposed to learn identity-in innariitn representations across large-scale different videos for person Re-ID pre- trainingg..
However, these approaches are only target at learning person representations especially for promoting the person Re-ID performance. They have poor generalization ability and perform poorly on other person-centric tasks. To learn general human representation, SOLIDER [12] is proposed recently by introducing prior knowledge and more semantic information.
In general, these pure-vision based works are rather aimed at only promoting the person Re-ID performance, or limited to the person-centric visual tasks. Their performance in learning generic person representation is still unsatisfactory and lacks the ability of multi-modal understanding.
Person-centric Tasks. In computer vision community, there are many tasks directly or indirectly related to person, i.e., image/text based person Re-ID, person attribute recognition, person pose estimation, person search, multi-object tracking and human parsing. We name such tasks as person- centric tasks and sort out five representative tasks among them for study.
1) Text-based person Re-ID aims to search for person images of specific identity by textual descriptions. Existing works can be divided into attention based and attention-fres methods. The former [8, 52, 68, 24, 78] attempts to establish region-text correspondences but ignores the efficiency. To better align the multi-modal features, the latter usually focuses on designing various objective functions [1 12, 27, 77] and model structures [93, 1 118].
2) Image-based person Re-ID aims to search for person images by given person images. Most works are based on supervised learning, The hard triplet loss [1 1, 39, 108] and classification loss [79, 119] are introduced to learn a global feature. Also, some works [84, 86) focus on learning a past-based feature instead. For example, Sun el al. [86] proposed to represent features as horizontal stripes and learn with separate classification losses. Some works utilize camera style [1 [117] and viewpoints [1 16] to train a more robust model. Some works [81, 80] focus on semi-supervised or unsupervised setting.
16

---

3) Person attribute recognition aims to identify the person's 's attributes. Many methods [44, 62, 61] treat this task as multi-label classification problem, while some others [90, 51, 113] adopt recurrent neural networks for exploring the attribute context. Also, some works [19, 53] introduce an additional language modality to get better performance.
4) Person search aims to jointly localize and identify a query person from natural and uncropped images. Existing works can be generally categrized into two-step and one-step ones. In two-step works [26, 87], persons are detected and then fed into a Re-ID model for identification, while one-step approaches [54, 70, 6] makes the joint framework more effective and efficient.
5) Human parsing is a fine- aiee-graned semantic segmentation task in human analysis. Some early works [33, 96] combine human parsing with pose estimation. Moreover, some works [47, 32] study the human parsing task in a multi-person scenario, which needs to distinguish the human instances. Meanwhile, the work [50] proposes self-correction mechanism and leads better performance.
A.2 Broader Impact
This paper proposes a language-ine -mage pre- pre-training framework for person representation learning, and contributes a large-scale synthetic person- person-centric dataset with rich image-text pairs. Our pre-trained models and dataset can help diverse person-related applications such as person re-identification, person attribute recognition and human parsing, thus boosting the development of smart retail, smart transportation, smart security systems and so on in the future metropolises.
Nevertheless, the application of person Re-ID and attribute recognition, such as for recognizing and tracking pedestrians in surveillance systems, might raise privacy concerns, It typically depends on the utilization of surveillance data for training without the explicit consent of the individuals being recorded. Therefore, governments and officials need to carefully establish strict regulations and laws to control the usage of these technologies. Otherwise, these technologies can potentially equip malicious actors with the ability to surveil pedestrians through multiple cameras without their consent. Furthermore, we should be cautious of the misidentification of the Re-ID systems to avoid possible disturbance. Also, note that the demographic makeup of the datasets used is not representative of the broader population.
At the same time, we have utilized substantial amount of person-containing video data from the internet for pre-training purposes. Consequently, it is inevitable that the resulting models may inherently contain information about these persons. Researchers should adhere to relevant laws and regulations. and strive to avoid using our models for any improper invasion of privacy. We will have a gated release of our models and training data to avoid any misuse. We will require that users adhere to usage guidelines and restrictions to access our models and training data. Mcanwhile, all our open- open-sourced assets can only be used for research purpose and are forbidden for any commercial use.
A.3 Limitations
PLIP presents a preliminary attempt to introduce language modality into generic person representation learning. Despite its effectiveness existing public datasets, PLIP may still be difficult to learn good fine-grained person representations for it does not explicitly achieve local information correlation across different modalities. Also, its pretext tasks require multiple forward propagation, directly increasing the memory overhead and affecting computational efficiency. Meanwhile, as we have followed the conventional practice of previous works by utilizing large amount of person-containing internet video data to implement pre- pre- training, there is a potential for privacy and security issues to some extent. Therefore, in our subsequent work, we will focus on addressing fine- fine-grained issues and improving the efficiency of our framework. We will take every possible measure to prevent the misuse of our models and dataset well.
A.4 Altering Color Word Affects Image Colorization: Visualization
As displayed in Fig 5, in our pretext task of text-guided image colorization, altering the color word in textual description significantly affects the colorization of image. However, our model may not fully understand the semantics of more detailed image regions. As shown in the last row, our model fails to distinguish between the blue clothing region and the red shoulder strap region (marked with yellow boxes), instead blending the two into unified coloration. This is due to the fact that the level
17

---

of detail in manually annotated datasets is still not sufficient, resulting in the model being unable to theoretically learn representations with higher levels of detail and greater discrimination capabilities. However, it is undeniable that our model has a preliminary understanding of the meaning of attributes and colors, and can associate them with related image regions, rather than simple memorization. This ability to distinguish between different parts of the person body guarantees the superior performance on the subsequent person-centrie tasks.
original
grayscale
[blue]
[purple]
[orange]
[pink]
[preen]
[red]
[yellow)
The with short black hair is wearing a jacket and black straight pants. wearing black shoes with white edging.
"A woman in her twenties
with straight shoulder length brown hair
wearing short dress.""
"A man with black short hair and is wearing a dark
blue hooded jacket with
shorts paired with black shoes with white edging carrying a backpack."
"A woman with brown hair wearing ] and
black leggings. She also wearing black dam shoes. She is carrying yellow backpack."
with brown hair is wearing a pink shirt and black leggings, She is also wearing black shoes. She is carrying a backpack."
"A man with short black hair wearing I ] shirt light blue jeans. wearing black sneakers with white soles."
"A man with short black hair is wearing a blue shirt and [ ] pants. He wearing black sneakers with white soles."
bald man wearing I shirt and pair of light blue jeans. He is wearing a pair of black shoes."
Figure 5; Visualization of gray-scale person image colorization results by changing the color words in textual descriptions.
18

---

A.5 How to Construct SYNTH-PEDES and its Properties
We build the SYNTH-PEDES dataset to pre-train the PLIP models a large-scale. In this section, we show the details how to construct our SYNTH-PEDES dataset and its characteristic properties.
A.5.1 Dataset Construction
The dataset construction process can be described as three steps, Firstly, we collect several person datasets [30, 83] to form the large-scale image dataset. Secondly, as a person image captioner, a simple but effective method is proposed to achieve automatic diversified text annotation with high- quality. Finally, we adopt some post-processing approaches to eliminate the noises and improve the dataset quality.
Image Collection.
We collect and process two large-scale person datasets to form the image dataset. The first is LUPerson-NL. [30]. It is new variant of LUPerson [29] top of raw videos from LUPerson and assign the noisy labels to each person image with automatically generated tracklet. It consists 10M images with about 430K identities collected from 21 scenes. The second LPW [83]. It consists of 2,731 different persons and 592,438 images collected from three different crowded scenes.
Image Captioner (SPAC).
Miriaine geoeting young
Given an input person image, there is no spe- cific work targeting at generating captions that detailedly describe the person's 's appearance. To this end, we propose a simple but effective method for person image captioning. It can gen- erate attribute-an annoutatonnnssionns and stylish textual- descriptions, which simulate the diverse perspec- tives that different annotators may have the same person image.
Generator
short hair
Profix Encodor
white bluer blue dentm jeans running shoes bluw backlaade
Erseches
peramsto
Actributes
Seit
Styde Enctddr
Style Encider
Generstor
Castions gererating
As illustrated in Fig. 6, Stylish Pedestrian Attributes-union Captioning (SPAC) mainly comprises two modules, i.e., prefix encoder and a shared generator. Specifically, we use ResNet101-FPy [37, 58] as the encoder and GPT2 [73] as the generator to capture rich per- son image details and generate high-quality texts. In the existing image-text person datasets [52, 24], many unique workers were involved in the labeling tasks. The same images usually have inconsistent-style language descriptions. Con- structing image-text pairs for an image with multiple descriptions will lead to unstable training and affect the model performance. Thus, in order to replicate the stylized variations that arise from multiple workers' labels and mitigate the issuc of redundant real labels, we have incorporated style encoders into pipeline. We let the concatenated prefixes pass through different style encoders to get the prefixes of their own style and then send them to the generator for the subsequent generation.
yniing man beens
A young boy with short black hair is wearing groy jacket ... He corniing black white bockaock around shoulders.
hair carriing blue bockpock shoulder.
Figure 6: Architecture of Stylish Pedestrian Attributes-union Captioning. It mainly consists of prefix encoder and generator. We use it to generate descriptions from a person image.
The entire training process can be seen as an autoregressive problem. Given dataset of paired images, attributes and captions (x", A yt) i- where A an attribute
image x containing six attribute descriptions, the learning goal is to generate meaningful attribute descriptions and captions from an unseen person image. The attributes and captions can be referred as a sequence
of padded tokens A' accordingly.
a
i.k
3.k
k-1 . -I with maximal lengths e,, &2
Following recent works [66, 65], our key solution is to jointly train prefix encoder and generator. The former is to capture the semantic embeddings as the prefixes from the image, and the latter, as an autoregressive language model, is to use the prefixes to predict the next token one by one. As shown in Fig. 6, we first feed the input image x into the prefix encoder PF and different branches prefix embeddings:
BR 1==1 to get the attribute-and
19

---

ei......e e (BR((P((x))
(8)
And then the concatenated embeddings are fed into different style encoders (SEA |- to get the stylized caption embeddings:
c....., cm { (concattt( e,,))
each embedding has the same dimension as a token embedding. We then concatenate the obtained embeddings the atrribute and caption token embeddings, where ef is selected from the stylized caption embeddings in turn:
Z concal( aj 1.k ..... a ;
(10)
Z concat(le'. .y yy......yy
Finally, we feed the embeddings [ into the shared generator G to predict the attribute and caption tokens in an autoregressive fashion, using the cross-entropy loss:
N
C. log G (y)) ee v'...'....,,).)))
(11)
c.
log G
[aj eg at .....,aa a}m ) )
(12)
j=1
Define À R+ balance factor, then the overall loss Lspac is computed as:
Sspac -L. AC
(13)
Pust-Processing.
Noise Filter Strategy. To filter the noises in LUPerson-NL. we propose Seed Filter Strategy. Specifically, it mainly includes three processes. 1) Filter-out. For all samples with the same iden- tity, we cycle to calculate the similarity between the current sample and the center of other sam- ples and exclude the samples whose similarity does not meet the threshold until the similarity of all samples with the same identity meets 2) Reassignment. According to the similarity and threshold, we reassign the excluded samples excluded dataset to the correct identity within a certain identity continual range. 3) Merger. We merge the samples that should belong to the same identity but are divided into different iden- tities. Through this process, the samples with incorrect identity label annotations can be well filtered out included in their expected identity set. More details can be found Sec. A.7
Dotaset
Attributes Generating
man
Generator
Prompt Engineering
[person] with
(down).
(show) carry (athers).
man with brack hair white shirt with carrstng
shoes
Figure 7: Visual example of Attributes Prompt Engineering. Given an image, we generate the attribute annotations based on SPAC and embed them into the masked sentence randomly chosen from the sentence library to form a complete cap- tion.
Data Distribution Strategy. There are some samples with poor imaging conditions in the image part. Also, the number images per identity in LUPerson-NN. is very unbalanced. To this end, we have adopted some strategies to ensure the quality of generated attributes, the consistency of gender annotation, and identity distribution balance. For example, adopt gender voting mechanism to automatically synchronize the gender annotation of an identity in dispute. For the text part, we aim to generate three sentences for an image. Two of them directly generated by SPAC while the another is generated by Attributes Prompt Engineering exploiting the attribute annotations generated by SPAC, as shown Fig. 7. There are averagely 2.53 sentence per image. More details can be found in Sec. A.7
20

---

Table 11: Statistics comparison existing popular datasets. SYNTH-PEDES is by far the largest
person dataset with textual descriptions without any human annotation effort.
Datasees
year
Vimages
ialenttties
Pdescriptions
view
label
lahel method
crop sizre
Market1301 [114]
Identit
hand+DPM [28]
128x64
DukeMTMC [120]
2017
36,411
1,852
fix camera
idcatity
hand
vary
CUHK PEDES 1521
2017
40,206
13,003
80,412
fix <lemtity-descriptioo
hand
vary
LPW [83]
2018
2,731
camera
idemtity
hand-Detectorss
vary
MSMT17 [94]
2018
126,441
4,101
fix camera dymamic
identity
FasterRCNN [75]
vary
SYSU3OK [88]
2020
29, 616666918
30,508
identity
YOLOv2
vary
RSTPReid [122]
2021
20,505
4,101
41,010
fix camera fix camera
identity +dsserrponii
hand
vary
CCPGPEDESS [24]
2021
54,522
4,102
54,522
identity edeserppiooo
hand
vary
LUPerson [29]
4,180.243
dymamic dymamic dynamic dynamic
YOLOV5
vary
LUPerson NL [30]
2022
10.683.716
433,997
identity
FairMOT [111]
vary
MALS [106]
2023
1,510,350
1,510,330
description
ImaginAlry
vary
U1ine6926 [126]
26,206
vary
SYNTH-PEDES
2024
4,791,771
312,321
12,138,157
dymamic
SPAC
vary
A.5.2 Dataset Properties
Thanks to the outstanding generating ability of our proposed SPAC, the SYNTH-PEDES dataset is full of high-quality textual descriptions in variety of styles, which can be utilized to train the representation learning model. Compared with existing person datasets Tab. 11, SYNTH-PEDES has the following advantages:
Diversified. Our dataset contains wide range of variations in the textual descriptions. Unlike the previous person datasets with only one or two image-text pairs, most images of our dataset are annotated with three textual descriptions.
High-quality. As some typical qualitative examples can be seen in Fig. 3, the generated annotations achieve accurate and detailed description of the person appearance. The further experiments conducted on the quality research can be found in the following sections. Researchers can use this dataset with confidence conduct relevant studies.
Large-scale. Tab. 11, we compare the properties of SYNTH-PEDES with other popular person datasets. As text pairs by far, which contains 312,321 identities, 4,791,711 images, and 12,138,157 textual descriptions.
see, SYNTH-PEDES is the largest real person dataset with high-quality image-
A.6 Details of Network Structures
In this section, we will provide a detailed explanation of the structure and some parameter settings of our representation learning network, mainly including some modifications made the ResNet networks [37] and the decoder designs for the two pre-text tasks. The dimension of global embeddings set to 768.
A.6.1
Modifications to ResNets
We made modifications the ResNet networks like CLIP [72]. Firstly, there now "stem" convolutions as opposed to 1, with an average pool instead of a max pool. For each "stem" convolutions, it is consisted of a convolution layer, batch normalization layer and ReLU activation funtion. Secondly, we perform anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride 1. Thirdly, the final pooling layer is pool instead of an average pool and we add a trainable linear layer to compress feature dimensions to 768.
A.6.2 Specifications of TIC Decoder
For an image with 3 H x W. we firstly obtain the multi-scale visual feature map with 1024 x x like FPN. Then, we send the feature map and textual global embedding to 2 Multimodal SE-Block and deconvolution layers to acquire the final feature map with x H x W, which is utilized to restore the color information. Next, specify the detailed designs of the Multimodal SE-Block.
In each MSE-Block, for a visual feature map with C x W, we firstly perform an average pooling operation on it to obtain visual feature embedding with C. Then we concat the visual feature embedding and the input texutal global embedding with 768 to obtain fused feature embedding with 1 (C + 768). Then, we feed it to FC layer, which contains a linear (ayer((C. + 768), C),
21

---

ReLU activation, linear layer(C, C) and a sigmoid funtion, to obtain the fused attention embedding with C. Finally, we apply the channel of the original feature map to obtain the multi-moda fused feature with C' x H x W.
After each MSE-Block, there is deconvolution layer operated on the output feature map. For the first block, the deconv is with input 1h24ncl=1024, output channel=256, kernel size=3, stride=2, padding=1 and output padding=1. For the second block, the deconv is with input channe]=256 output channel=3, and others the same.
After the approaches above, we finally obtain output feature map with the same shape like the input original color image, and it can be used to predict the ground truth.
A.6.3 Specifications of IAP Decoder
For each masked token, we obtain its hidden state output as the masked embedding. We then simply concat the masked embedding and the visual global embedding to obtain the fused embedding. Then the fused embedding will be regarded as the QKV and set to normal transformer block to improve the multimodal fusion. Finally, the output fused embedding of transformer block will be sent to prediction head to predict the masked word. The prediction head is a simple linear layer with (768, nvvcab), where the nwocab means the size of the dictionary.
A.7 Details of Dataset Construction
In this section, we specify the details of our gathering SYNTH-PEDES dataset. We mainly provide detailed explanation of the designs of our texutal description generated model SPAC. Also, demonstrate the noise filter strategy and data distribution strategy adopted in the process of dataset constrution.
A.7.1 Designs of SPAC Model
The prefix encoder is a ResNet101-FPN [58] pre-trained on ImageNet [21]. There are two types of prefix branches, one is the attribute branch and the other is the sentence branch. The detailed structures of each branch are shown in Tab. 12. Prefix dimension is set to 768. Attribute prefix length and sentence prefix length are and 5, respectively.
There are totally types of attribute prefix. For each type of attribute prefix, we send it to the generator GPT2 [73] and generate the attribute annotation accordingly. Also, we concat all the attribute prefixes and sentence prefix, and send them to types of style encoders for obtaining the stylized prefixes. Then the stylized prefixes will be used to generate complete textual descriptions of 2 different styles accordingly. The two style encoders are actually two linear transformation matrices with different weights.
A.7.2 Details of Noise Filter Strategy
To filter various noises in the LUPerron-NI. [30] dataset, we propose Seed Filter Strategy, which is illustrated in Algorithm 1. Specifically, it consists of three processes, each with its corresponding similarity threshold for filtering, reassigning or merging. 1) Filter-ou. For all samples with same identity, we cycle to calculate the similarity between the current sample and the center of other samples and exclude the samples whose similarity does not meet the filtering threshold a, until the similarity of all samples with same identity meets it. 2) Reassignment. According to the similarity and reassigning threshold or, we reassign the excluded samples in the excluded dataset to the correct identity within certain identity continual range. 3) Merger. We merge the samples that should belong to the same identity but are divided into different identities according to the merging threshold Om. Through these processes, the samples with incorrect identity label annotations can be well filtered out or included in their expected identity set.
The following is detailed explanation of the specific symbols in the Algorithm 1. Sin is the input dataset to be denoised while Sout the output denoised dataset. Sezolude the set of image samples to be excluded. Smerge is the set of identities to be merged and it contains lot of merged-identity pairs. 1D2 is the khth identity of the input dataset, with image samples att. FID& is the output final identity. Sim is a similarity calculate function. If the variable it contained is single identity, will calculate the similarity between each sample and other samples' center. If it contains image
22

---

Table 12: Structure of the attribute and sentence branch. The bias of all linear layers is set true. represents batch size. represents the channel size. h and w represent the height and width of the output feature map.
Layers
Paramiers (kcrncl.stride.pad) (3,2,1)
Output Size
(CNN-MLP)
(n.c.h.w)
Conv layer
(n.c.h/2.w/2)
Avgpool
(n,c)
Lincar
(n.768)
Dropout
rate=0.25
(n,768)
Lincar
(n,2304)
Dropout
rate=00225
(n,2304)
LeakyRel.JU
rate=1/5.5
(n,2304)
Rearange
(n.3,768)
Cony layers X
(3,211)
(n.c.h/4,w/4)
Avgpool
(n,e)
Linear
(n,1536)
Dropout
rale=0.25
(n,1536)
Lincar
(n,3840)
Dropout
rate=0.25
(n,3840)
LeakyRell.U
rate=1/5.5
(n,3840)
Rearange
(a,5,768)
sample and identity, it will calculate the similarity between the image sample and the center of all samples in the identity. If it contains two identities, it will calculate the similarity between the two centers the two identities. Merge is a function that merges the identities. We use cosine similarity to calculate the similarity between samples. For the byper-parameters, 0, is set to 0.6. o, is set to 0.65. om is set to 0.62. ra and r3 are both set to 2.
Algorithm Seed Filter Strategy
1: Input S.n (ID& where ID (xy,.... for ID₃ E S.n do
3: repeat
4: if min Sim(IDx) < then
at = arg min (imm)Da):
6:
ID = ID2 24 25 = Sencludei
end if
8: until min Sim(! 2 a.
end for
10: for 2t E Sexclude do
11: if (amx(Simaat; ID.) 2 ar then
12: ID, arg max (Six(x)
13: ID, ID, + +
14: end if
15: end for
E (ID,)N do 17: if {Siimii (D > om then
16: for ID
18: (ID [IDR, DD++s) Smerge
19: end
20: end for
21: for FID; E Smerge do iit,
22: for FID, E (FID,) J= do
23:
if FTD; n FID, # Ø then
24:
FID; Merge Merge((IDD,FIID) end if
25:
26: end for
27: FID; - Sout
28:
end for
29: Output Souti
23

---

A.7.3 Details of Data Distribution Strategy
For the image part, there are some bad pictures with blocking, blurring and multiple people. Due to the poor quality, these pictures are not a good choice for training. Also, the number of images per identity in LUPerson-NL. [30] is very unbalanced. Some identities have thousands of images, while others have only few images. To further improve the quality of our dataset, we have adopted the following strategies: 1) Ensure the qualities of generated attributes. In our setting, the attributes generated by poor quality pictures will contain a large number of unknowns. To filter these poor quality pictures, the pictures with three more unknown attributes will be directly filtered. Particularly, if the uphody and downbody attributes of a picture are unknown simultaneously, it will also be filtered. 2) Ensure the consistency of gender attribute. Ensuring the gender consistency of id plays a role in promoting the stability of the model training. However, for those pictures with poor quality, it is often very challenging to correctly predict their gender attributes. So we adopt a gender voting mechanism. That is, for all pictures in an identity, if the gender attribute with the maximum frequency of occurrence is greater than 0.7, the gender of all pictures in the identity will be automatically changed to this gender. Otherwise, the pictures in the identity will be considered as bad pictures, and will be filtered out. 3) Balance the distribution of identities. If the number of images of an identity is less than the minimum number of 5, this identity will be filtered. If the number of images of an identity is greater than the maximum number of 20, randomly select 20 images from all images of the identity and filter out the others.
For the text part, we aim to generate three sen- lences for an image. Two of them are directly generated by SPAC while the another is gener- ated by Attributes Prompt Engineering exploit- ing the attribute descriptions generated by SPAC. In many processes of our approach, we must ex- tract the attribute phrases from a textual descrip- tion. For example, in order to guild the SPAC to generate attribute descriptions, we have to ex- tract the attribute labels from the paired textual descriptions. The extracting process is as the fol- lowing: 1) Construct the keywords set for each attribute. Firstly, we extract all the nouns in the datasets' [52, 24] captions by nltk tools and sort them by frequency. Then, for each attribute, we manually select from the nouns and construct the keywords set. 2) Assign the noun phrases to relevant attributes. For textual description, we firstly extract the noun phrases by nitk tool. Then, according to the keywords set, we assign each noun phrase to the correct attribute. On this way, we finally finish extracting the attribute phrases from a a textual description. For Attributes Prompt Engineering, as shown in Fig. 8, we have created a mask standard sentence library with many diversities. To construct the library, by exploiting the nltk tool and the above attributes extracting method, we first mask all attributes phrases of the captions in CUHK-PEDES and ICFG-PEDES. Then We manually select sentences with good sentence structure from them to form the mask stan- dard sentence library. The library totally consists of 1664 sentences for different generated attribute missing situations.
Feraal::
1. The [person] has [head], She is wearing [up]. [down), and [shaes]. She carrying [others).
2. The [person] with [head] wearing a [up] and [down].
She also wearing [shoes] and carrying [others].
[person] with [head] and she is wearing [up] with
[down] paired with [shoes] carrying a [others].
Male:
1. The [person] is wearing [head] with a [up] and he is also wearing [down] paired with [shoes], carrying [athers].
2. A [person] with [head] is wearing [up] and [down]. He also wearing [shoes]. He carrying a [athers].
3. [person]
[head] is wearing [up] and a pair of [down], He is also wearing [shoes] and carrying a [others].
Person: 1. An individual with [head] and is wearing [up] with
[down) paired with [shoes] carrying a [others].
2. A [porson] with [head] and is wearing a [up] with (down) paired with [shoes] carrying a [others].
3. A [person] has [head], and wears [up] and pair of [down) a pair of [shoos), halding a [athers).
Figure 8: Some examples in the mask standard sen- tence library for the attributes all-known situation.
A.8 Experimental Setup
A.8.1 Implementation Details
During the training of SPAC, we adopt ResNet-101 [37] together with a Feature Pyramid Network (FPN) [58] as the prefix encoder and GPT2 [73] as the generator. The ResNet- 101 and GPT2 are both pre-trained on their respective pre-training tasks. All images are resized to 384 x 128 and normalized with mean and std of [0.485, 0.456, 0.406), [0.229, 0.224, 0.225], which are calculated from all images in ImageNet. We use the combination of CUHK- [52] and ICFG-PEDES [24] as the training dataset. We adopt horizontally flipping to augment data, where each image has 50% probability to flip randomly. We aim to generate the textual descriptions with two style type. The learning rate is fixed at 0.0001. The balance factor À is set as 0.15, There are six types of attributes
24

---

including gender. head, upper body, lower body, shoes and belongings. The prefix lengths of attributes and relation are and 5 respectively. We train it on x Geforce 3090 GPUs for 30 epochs with a batch size of 64 totally, which takes approximately 1. days. The optimizer is Adam [63] with the default setting.
During the training of PLIP, we adopt four types of backbone as the visual encoder, i.e., ResNet50, ResNet101, 15s and Swin Transformer Base. The pre-trained BERT [23] is utilized as the textual encoder and we only unfreeze the last 5 layers, keeping other parameters frozen. All images are resized 256 x 128 and normalized with mean and std of [0.357. 0.323. 0.328]. [0.252, 0.242, 0.239], which are calculated from our proposed SYNTH-PEDER to augment data, where each image has 50% probability to flip randomly. For ResNet50, we train our model on 4 x Geforce 3090 GPUs for 70 epochs with a batch size of 512 totally, which takes approximately 15.2 days. The base learning rate is set to 0.002 and decreased by 0.1 at the epoch of 30 and 50. Besides, the learning rate warm-up strategy is adopted in the first 10 epochs. The learning rate of BERT has 0.1 decay. For other types of visual encoder, there are some differences on the learning rate and batch size setting. The hyper- perameeaesssrss in the objective function are set to A1= = 0.02 and 22 = 0.1. The optimizer is Adan [100] with the default setting. We adopt the mixed precision training mode by Apex.
We adopt horizontally flipping
For each person-centric downstream task, we reproduce a range of state-of-the art methods as the baselines. If no special instructions are given, we perform the experiments by just replacing the backbone in each baseline to the models pre-trained by our proposed method. Meanwhile, for text-based person Re-IID, we adopt the standard metrics Rank-k (k=1,5,10) to evaluate the model performance. For image-based person Re-ID and person search, we follow the popular evaluation metrics: the mean Average Precision (mAP) and the Rank-k (k=1,5,10). For person attribute recognition, we adopt four evaluation metrics including accuracy (Acc), mean accuracy (mA), recall (Rec) and Fl value (FI). For human parsing, we use mean accuracy (mA) and mean intersection over union (mloU) as the evaluation metrics.
A.8.2 Benchmarks
To evaluate our proposed approach, we perform in-depth experiments eleven datasets for five downstream person tasks. For text-based person Re-ID, we conduct extensive experiments on two popular pubic datasets, i.e., CUHK- [52] and ICFG- [24]. For image-based
person Re-ID, we use three popular public datasets, i.e., Market1501 [114]. MSMT17 [94] and
DukeMTMC [120]. For person attribute recognition, three large-scale datasets PETA [22], PA-
100K [62] and RAP [45] are used. For person search, we adopt PRW [115] and BUHK-SYSU [99]
in our experiments. For human parsing, the LIP dataset [33] and PASCAL-Person- dataset [13] are used. Next is an introduction to representative datasets in each task.
CUHK-PEDES dataset [52] is the first and most commonly used benchmark for text-based person re-identification. It contains 40,206 images and 80,412 textual descriptions for 13,003 identities. Each person image has two corresponding textual descriptions on average. We dopt the same data split as [52]. The training set has 34,054 images of 11,003 identities. The validation and test set have 3,078 and 3,074 images of 1,000 identities, respectively.
Market1501 dataset [1 14] includes 32,668 images of 1,501 persons captured from 6 different cameras. The DPM detector is employed to crop the bounding boxes for these pedestrians. The dataset is divided into training set comprising 12,936 images of 751 persons and test set with 3,368 query images and 19,732 gallery images of 750 persons.
PA-100K dataset [62] contains 100,000 pedestrian images with resolutions ranging from 50 100 to 758 x 454 captured from 598 different cameras. Each pedestrian image is annotated with 26 commonly used attributes. As the official protocol, there are 80,000 images for training. 10,000 images for validation, and 10,000 images for testing.
PRW dataset [115] is widely used dataset for person search which contains 11,816 video frames captured by 6 different cameras, and 34,304 manually annotated bounding boxes. All people are divided into labeled identities and other unknown ones. The training set comprises 5,704 images and 482 identities. The test set contains 6,112 images and 2057 query people.
LIP dataset [33] is a large human parsing dataset containing 50,462 images with manual pixel-wise annotations for 19 semantic human parts. The dataset is collected from the real-world scenarios. which
contains human images with various appearances, heavily occlusions and
25

---

introduces many challenges. LIP is divided into 30,462 images for train set, 10,000 images for validation set and 10,000 images for test set.
A.9 Thorough Ablation Studies and Analyses
In this section, we perform in-depth ablation studies to analyze the effectiveness of each part of our work. If not specially stated, we use ResNet-50 as the visual encoder and pre-train on the sub-dataset of SYNTH-PEDES, which has 10,000 identities, 139,564 images and 353,617 textual descriptions. The evaluation is mainly performed on the benchmarks of text-based person Re-ID and image-based person Re-ID using the zero-shot evaluation protocol.
A.9.1 Effectiveness of Each Component
Table 13: Ablation study on the impact of each pretext task, all using default settings.
Impact of each pretext task. We have designed three pretext tasks to implement pre-training. To assess the impact of pretext tasks the general- izability of learned representations, we directly evaluate the performance of pre-trained models on the downstream datasets without fine-tuning. As indicated in Tab. 13, each task contributes the modell's zero-shot capability JUHK-PEDES and Market 1501. Combining all tasks leads to the optimal performance, highlighting the significant role each task plays in facilitating the learning of generic person representations.
Components
CUHK PEDES
Marketl500
WLC TIC IAP
REl
R85
RB10
R@l
R83
1010
30.2
53.3
64.0
62.3
79.9
85.7
4
314
55.2
65.9
62.9
80.7
86.2
30.5
54.4
64.3
62.7
80.5
86.1
39.3
61.4
70.4
32.5
56.3
66.6
63.1
$0.8
86.3
TIC is essential for transfer capability. After pre-training with different settings, we evalu- ate the model's performance on four datasets under the zero-shot setting. The metric DukeMTMC [120] is rank-5, while others [52, 24, 114] are all rank-1. As shown in Fig. 9, without TIC, the learned representation exhibits noticeably weaker performance on downstream datasets, averaging a 2.45% decrease compared to the default method on CUHK-PEDES and M5rket1501. However, the introduction of TIC significantly improves the transfer performance. We believe this is because TIC encourages the model comprehend textual semantics and accomplish body part localization and coloring, which holds significant implications for learning generic representations.
Text-based Person Ro-ID
Image-basd based Person Re
22
3013
38
60
260
363
20+
52-
2LD
BLIP
(b)
Figure 9: The effectiveness of TIC on transfer capability. TIC brings significant improvements.
IVLC contributes to stable representation learning. We utilize the IVLC loss to optimize the model at idevtity-level. We conduct an ex- periment to verify the effectiveness of IVLC in stable and meaningful representation learn- ing. The training samples in this experiment the first one million images from SYNTH- PEDES. We test the zero-shot performance of the trained models with different settings on the CUHK-PEDES and Market 501 datasets. As the results shown in Fig. 10, the model trained with instance-level InfoNCE loss significantly underperforms compared to the model trained with identity-level IVLC loss. Additionally, as the training process extends, the model trained with InfoNCE loss experiences notable decline performance, whereas the performance of the model trained with IVLC loss tends to stabilize. These results demonstrate that IVLCC contributes to stable and better representation learning.
0
25
80
100
MadertD011
Figure 10: The effectiveness of IVLC in stable representation learning, which brings much better performance.
26

---

A.9.2 Dataset Quality Evaluation
The quality of dataset is critical for large-scale representation learning. We have conducted quan- titative experiments to evaluate the quality of our dataset compared to three manual annotated datasets [52, 24, 122] and an existing synthetic dataset [97].
Manual Evaluation
We randomly select 500 image-text pairs from our SYNTH-PEDES dataset and other manually annotated or synthetic datasets, forming 2500 image-text pairs to be evaluated. The score of the evaluation is divided into five levels. The first is that the most part of the description of the image is incorrect. The second is that small part of the description of the image incorrect. The third is that the description is correct but rough. The fourth that the description cor- rect and generally detailed. The fifth is that the description is completely correct and very detailed. As shown in Tab. 14, our *NNTH-PEDE dataset is competitive in quality compared to these manually annotated dataset at about 200 times the amount of image-text pairs.
Table 14: Manual evaluation results. Our SYNTH- PEDES dataset is competitive with manual datasets and surpasses an existing synthetic dataset by large margin.
Score
Dataset
Average
2
3
4
CUHE- [52]
14
24
146
120
3.77
ICFG-PEDES [24]
12
44
124
150
170
3.84
RSTPReid [122]
11
26
149
146
168
3.87
FinGGPR [197]
41
435
2.82
SYNTH-PEDES
18
30
162
132
158
3.76
Automatic Evaluation
We utilize some multi modal pre-tratee pre-trained mod- els like CLIP [72], ALIGN [40], BLIP [49] and BLIP2 [48] to calculate the cosine similarity of image-text pairs in each dataset. Moreover, con- sidering the varied sensitivity of each model to image-text similarity, we employ the following normalization to ensure that the results are ap- proximately within the same range. For each model's 's evaluation result, we initially identify its maximum similarity across all datasets and assign it a score of 100 points. Subsequently, the score for each dataset is determined as the average similarity relative to this highest simi- larity. As shown in Fig. 1, under the evaluation of five models, our SYNTH- only gains competitive scores compared to man- ually annotated datasets, but also surpasses the existing synthetic dataset [97] by large margin.
CUP-0/32
CUP-8/16
BUF2-0/32
88.6
60.3
36.6
45.4
57.7
47.7
61.6
62.22
56.2
BUP-8/32
SYNTH PEDES
RneGPR-C
CUHK-PEDES
ICFO PEDES
ASTPReN
dataset not
Figure 11: Automatic evaluation results by 5 pre- trained models. Our dataset are about 200 times larger than manually annotated datasets, yet still mantains competitive data quality.
A.9.3 Different Pre-training Settings
SAT
ClipCap
SPAC
Low-quality R@1 R@5 R 10
High-quality
Downstream
Downstream
R@
@@
R
R@ 10
R@
R 10
R@
R@5 R 10
CUHK-PEDI
23.6
55.8
24.3
56.2
26.6
59.8
CUHK-PEDE
28.3
51.7
62.4
29.2
51.9
62.9
CCDG-S
17.9
48.8
18.4
49.2
20.9
52.3
22.2
43.6
53.4
23,8
44.9
55.5
RSTPReid
16.2
51.0
16.5
51.4
18.5
54.4
RSTPReid
19.3
41.7
54.7
20.0
42.5
55.2
500
57.3
81.8
57.4
81.8
58.4
82.8
M0r
57.9
77.2
83.3
58.6
77.4
83.4
Duk
37.8
59.4
38.2
59.6
39.3
61.0
DukeMTMC
39.6
56.0
62.8
40.9
57.2
64.4
MSMT17
16.9
36,2
17.1
36.3
18.3
37.9
MSMT17
19.3 32.6
39.1
19.6
33.0
39.3
(a) Different caption methods.
(b) Different datasct quality.
Table 15: Ablation studies on caption methods in (a) and dataset quality in (b).
Pre-training with other caption methods. We have compared our proposed SPAC with the previous representative caption methods SAT [102] and ClipCap [66] in generating captions for person images. The compared methods generate only one caption per image, lacking diversity. To ensure a fair comparison, exclusively utilize SPAC to generate single caption for the sub-dataset derived
27

---

IVI.C TIC
CUHK
Market
G(z)
CUHK
Market
Visual/Textun
CUHK
Market
R@1 R@@ 1 R@ 10
R@1
R@
R@1 1 R@
R@ I 10R@
R@@ 10
Ltd. Manh. 18.7 49.0
46.4
73.8
0.2
32.1
65.9
63.2
86.6
Aver. Pooler
26.6 60.8 55.5
81.0
Lal Manh. 17.6 45.4
44.0
71.9
0.5
30.9
65.3
62.9
85.2
Max. Pooler.
26.2 59.5 56.2
83.0
Lom. Manh. 19.1 48,5
44.4
75.4
0.8
31.6
66.1
62.7
86.9
Attn. Pooler. 24.3 57.1 57.3
81.9
Ltd. Eucl. 30.0 63.5
63.1
86.1
x²
32.3
66.0
63.4
85.9
Aver. Aver. 30.2 65.6 60.1
84.1
Cal. Eucl. 31.8 64.1
63.0
85.7
V
32.5
66,6
63.1
86.3
Max. Aver. 32.5 66.6 63.1 86,3
Com. Eucl. 32.5 66.6 63.1
86.3
31.7
65.5
63.7
86.5
Attn. Aver.
27.9 62.4 61.3 84.9
(a) Different training objectives.
(b) Different prediction difficulty.
(c) Different pooling methods.
Table 16: Ablation studies on training objectives in (a), prediction difficulty functions in (b) and pooling methods in (c).
from SYNTH- Totally, for cach method, there are 139,564 image-text pairs for pre-training. As shown in Tab. 15 (a), the performance of SPAC on six datasets is much better than the others, showing the superiority of our SPAC.
High-quality dataset leads to good pre-training. In this experiment, we consider the original LUPerson-NL dataset [30] and our dataset as low-quality and high-quality dataset, respectively. We conduct a performance comparison on downstream tasks following pre-training on each dataset. To ensure parity, an equal number of identities and image samples are randomly selected for each dataset, and captions are generated using SPAC. The pre-training process involved total of 21 214.053 image-text pairs for each dataset. As shown in Tab. 15 (b), the downstream performance of the low-quality dataset is markedly inferior to ours. These results collectively validate the efficacy of the strategies employed in our dataset gathering process.
The diversity of textual descriptions matters. Our dataset has textual descriptions with differ- ent styles for each image. To validate the effec- tiveness of this diversity, we assess it through pre-training with datasets exhibiting varying de- grees of textual diversity, followed by compre- hensive generalizability study on downstream tasks. Specifically, we have studied four dif- ferent degrees of textual diversity from weak to strong, while keeping the number of person images and identities consistent. As shown in Fig. 12, the fourth case, with the highest degree of textual diversity, has the best performance. Also, compared to the first case, the superior perfor- mance observed in the second case further substantiates the significance of diversity, as the prompt caption exhibits less diversity than the generated caption.
Text Iesed Person Be ID
Image hosed Pirsson Re
65
35
$51
Rosec10 (56)
55
25
65
1CPC- MEDES
CURK-FEDES
Marwet1003
Figure 12: The diversity of textual descriptions matters. PC and GC mean prompt caption and generate caption, respectively.
A.9.4 Optional Practices
Training Objectives of IVLC and TIC. The different combinations and implementations of the loss functions for IVLC and TIC are shown in Tab. 16 (a). For the IVLC task, we have implemented three identity-based loss functions for vision-language contrastive learning. Specifically. Cat, represents the alignment loss proposed in [93], Led. represents the SDM loss proposed in [42], and Lrm. represents the CMPM loss proposed in [112]. For the TIC task, to measure the error between the color-restored image and the original colorful image, we tried using Fuclidean distance and Manhattan distance. In fact, any function meeting the requirements can be used in our learning framework. However, as shown in Tab. 16 (a), the combination of CMPM loss and Euclidean distance leads to the best performance, and gains rank-1 32.5% and 63.1% % on each dataset accordingly.
Different prediction difficulty. Curriculum learning [3, 71] demonstrates that it could be useful to organize the training samples from easy to hard. It is inspired by the human learning process and has achieved success in a wide range of tasks. To explore the effectiveness of this, we consider the masked textual descriptions with different masking probabilities as samples with different difficulty levels. Specifically, we define G(z) as the difficulty function and explore a variety of optional functions, including constant, x, x², and Vz. By using the gradually increasing functions, we start the Gax) grows from 0.2 to 0.8, which means the training samples will change from easy to hard at different
28

---

rates. As shown in Tab. 16 (b), the results suggest that satisfactory performance on both datasets can be achieved with /z in pre-training. This means the gradual difficulty sample strategy truly plays role in encouraging the model to learn generic person representations better.
Pooling methods of global embeddings. We have studied the impact of different pooling methods of global embeddings, In the textual branch, we investigated two methods for obtaining the global embedding, namely pooler-out and average pooling. Additionally, in the visual branch, we explored three pooling operations on the last stage feature: average pooling, maximum pooling, and attention- based pooling. As shown in Tab. 16 (c), the combination of maximum pooling and average pooling consistently yields the best performance on both datasets, establishing it as the default setting.
Effectiveness of different sypper-prraeterr Since our objective function consists of three losses and each loss is critical to the overall performance, we further search for the optimal hyper-parameters to balance the loss weights. As shown in Tab. 17, PLIP achieves the best performance when Å₁ 0.02 and X2 0.1, surpassing the version of \1 = 1 by 3.8% and 5.5% rank-1 on CUHK-PEDES and Mar- ket1501, respectively. This is mainly attributed to the loss values of three pretext tasks. The
Table 17: Comparable results of different hyper- parameters in overall objective function. We show the best score in bold.
Settings
CUHK- PEDES
CURK-PEDES
R@5
R@1
Market|501
RØ5
A m==1 1, -==== =
28.7
51.9
62.3
57.6
17.1
K3.4
0.2.
30.3
53.6
64.1
61.7
79.2
85.5
55.1
62.2
79.9
-1-0.02, Ag
32.3
55.8
65.9
62.6
80.3
85.9
A1 0100,
31.7
55.3
65.1
624
80.1
85.7
A2
32.5
55.9
80,9
-002, A2 0.2
32.3
56.1
66.4
62.7
81.1
$6.1
A = 0.02, 0.1
32.5
56.3
66.6
63.1
80.8
$6.3
IVLC loss is the primary component of the objective function, aiming to achieve the visual and textual modality association. The TIC and IAP losses are employed to better assist in representation learning, and their loss values are relatively large. Therefore, adjusting the weights reasonably is also crucial for performance.
A.9.5 Learning with Other Pre-training Methods
PLIP exhibits clear superiority in person repre- sentation learning compared to other general pre- training methods. To validate this, we have pre- trained a series models on the entire SYNTH- PEDES dataset with different pre-training meth- ods, i.e. CLIP [72], BLIP [49], BLIP2 [48] and PLIP. For BLIP2, exclusively utilize its first- stage pre- teeiiraining method, aligning with the de-
Table 18: Comparable results of PLIP with other representative pre-training methods. We show the best score in bold.
Method
Backbone
CUHI-PEDES PEDES
Market
RBB-3
RG3
CLP
RX50
35.1
62.6
73.0
20.1
84.7
89.7
CLIP
VIT-B
41.6
65.2
74.8
72.9
85.66
90.9
BLJP
VIT.S
44.9
68.6
77.4
73.7
86.2
BLIP
VIT-B
48.2
72.7
80.6
76.3
88.6
92.5
fault setting in the original paper designed for
HIIP2
VIES
47.3
71.6
80.1
75.9
88.
92.4
BLIP2
VIT-B
48.8
72.6
81.2
77.3
89.4
93.3
image-text retrieval tasks. Subsequently, we
PLIP
RN50
52.9
74.7
82.5
81.1
91.4
94.6
compare their direct transfer performance downstream datasets. The results, as reported in Tab. 18, demonstrate that under the comparable setting of models with the roughly same parameters (ViT-Small vs ResNet50 and ViT-Base vs Swin-Base), PLIP outperforms all other pre-training methods significantly on downstream datasets. Specifically, PLIP with ReNet50 achieves 52.9% and 81.1% rank- on CUHK-PEDE ket1501, greatly surpassing CLIP with ResNet50 by 14.8% and 11.0%. Moreover, with Swin-Base as the backbone, PLIP achieves the best performance, with rank- reaching 56.3% and 83.7%, re- spectively. These experimental results demonstrate the superiority of PLIP in person representation learning.
PLIP
Swin-B
$6.3
77.6
84.8
83.7
93.6
95.9
and Mar-
A.10 More Evaluation on Downstream Tasks
Table 19: Comparisons of domain generalization
Domain Generalization for Image-based Per- son Re-ID, comparison to the SoTA in gen- eralizable mmage-based person re-identification shown in Tab. 19. The models trained on the source datasets are directly generalized to the target datasets without tuning. Several methods and datasets recently are compared, with methods including QAConv [56], Trans- Matcher [57], WePerson [46], and APTM [106].
performance on image-based person Re-ID.
501
MSMT17
Method
Train Set
mAP
R@1
MAP
RØ1
QACamy
RandPersun
46.9
74.3
14.0
403.6
QAConv
ClonedPerson
59.9
84.5
18.5
49.1
TransMatcher
RandPersun
49.6
77.6
16.4
45.3
TransMatcher
UnrealPerson
59.4
81.6
21.6
52.0
WePerson
WePersom
55.0
81.5
18.9
46.4
APTM
MALS
3.8
11.9
1.8
7.4
PLIP
SEDES
35.8
81.
22.1
52.3
29

---

and datasets including RandPerson [92], ClonedPerson [105], UnrealPerson [109], WePerson [46] and MALS [106]. As the results indicate, even without specific design tailored for this generalizable task, our PLIP trained on SYNTH-PEDES shows competitive performance when directly generalized downstream real-world datasets. These results verify the outstanding generalization ability of our pre-trained models for this task.
Improvement over Person Attribute Recog- nition Methods. Our model can also bring con- siderable improvement to person attribute recog- nition methods. We conduct experiments using two representative baseline methods [44, 41] by comparing the performance gain of different pre- trained ResNet50 models. We report the results in Tab. 20, where the two methods are based on traditional CNN structure and multi-label classi- fication loss. As we can sec, model improves their performance significantly on the two pop- ular datasets, and the improvements leave all other pre-trained models far behind. Particu- larly, in terms of mA, by using pre- pre-trained ResNet50, the average improvements of these methods are 2. 1% and 1.1% on PA100K and PETA respectively. These results demonstrate that our
Table 20: Improving two person attribute recogni- tion baseline methods. The results of DeepMAR are from a re-implementation by replacing the backbone with ResNet50, which are much better than the original.
Pre-trin
mA
Ace
PATOR
Rec
F1
BA
Acc
PETA
Rec
FI
Buseline
78.1
76.3
51.0
61.3
82.1
17.1
54.6
83
MaCov2
DospMar
LUP [29]
78.5
77.3
81.5
82.8
77.3
$1.5
85.3
LUP-NL [30]
99.3
77.4
84.7
84.5
83.5
71.9
$5.0
83.6
PLIP
80.5
78.8
86.0
M6.5
83N8
78.2
85.3
85.8
Rothink [-1]]
Buscline
50.1
791
57.0
57.4
81.0
75.7
55.6
McCov2
30.2
78.3
LUP-NL
[30]
$1.3
79.1
87.4
84.1
78.6
36.3
PLIP
82.2
81.1
88.6
88.6
85.1
79.5
86.3
86.9
framework helps to learn hetter person representations for fine-grained recognition.
Comparison with state-of-the-art methods on human parsing. Human parsing task re- quires excellent perception of spatial informa- tion. By training the TIC task, as shown the Figure 5, our models truly learn the corre- lation between attribute phrases and spatial re- gions, which guarantees the good performance of this fine ane-grained semantic segmentation task.
Table 21: Comparison with SoTA methods on hu- man parsing. We show the best score in bold.
LIP
PASCAL.
Methods
Backbone
mloU
mloU
Attention [7]
VGGI6
42.92
56.39
MMAN [64]
RN101
46.81
59.91
JPPNet [55]
RN101
51.37
59.36
CE2P [76]
RN101
53.10
In Tab. 21, compare results with existing SoTA human parsing methods on LIP [33] and PASCAL [13]. Under the ResNet101 setting, our method achieves 61.32% and 72.63% mloU
CNIF [91]
RN101
57.74
70.76
SCHP [50]
RN101
59.36
71.46
CDGNet [60]
RN101
60.30
SOLIDER [12]
Swin-B
60.50
PLIP
RN50
60.41
72.14
LIP and PSCAL, respectively, significantly surpassing previous SoTA methods. Also, by applying our pre-trained Swin-Base Base model on SCHP [54], we achieve the best performance each datasets. These results demonstrate that our pre- pre-trainn training framework shows good potential in learning discriminative person representation for this task.
PLIP
RN101
61.32
72.63
PLIP
RN152
62.44
73.51
PLIP
Swin-B
63.52
73.93
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction our paper accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We conduct an in-depth discussion of the limitations of our work, which can found Sec. A.3 of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification:
Our work focuses on computer vision applications, not including theoretical
results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
We have fully provided all the details of our proposed methods in the main text and appendix, to guarantee the reproducibility of our paper's main experimental results. Meanwhile, we will open-source all of our code, data, and models after our paper accepted. These details and open-source initiatives will ensure the reproducibility of our work and make significant contribution to the entire community.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
32

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: To address privacy concerns associated with our dataset containing persons and the application of person RelD technology. we will implement controlled release of our code, data, and models. Therefore, to ensure private information security, we would not provide open access to our data and code during the paper submission process. Notably, will publicly share the application link for all sources after paper is accepted. Applicants will be required to adhere to relevant guidelines and regulations to access our sources, and we will conduct strict review of their qualifications prevent any privacy violations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips co/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wtt.SSummmsiioooooioooo.ooooo for more details. / //nips. wwtpppccp/pppaccccccccccccccccee wtapiic/ggies//eeeeeeeeeeeeeeeet Stteeees/oooSSSmmeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Our paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
33

---

Due to pre-traine -reiiiinn technology is comparably computational expensive, the error bars are not reported the paper like nearly all related work.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: our paper provides sufficient information on the computer resources needed to reproduce the experiments, which can be found in Sec. A.8.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attpp:://eeursi wttpiippp...pppllpe httpppllic/thhissu.....e
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
34

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses both potential positive societal impacts and negative societal impacts of the work performed Sec. A.2 of the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our paper describes safeguards that have been put in place for responsible release of data and models in Sec. ?? of the appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of assets used the paper are properly credited, and the license and terms of use are properly respected.
35

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaaaaa com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: To address privacy concerns associated with our assets including code, data, and models, we will implement controlled release of our assets. Therefore, ensure private information security, we would not release our assets at submission time. We will publicly share the application link for assets after our paper is accepted. Applicants will be required to adhere to relevant guidelines and regulations to access our assets, and we will conduct strict review of their qualifications prevent any privacy violations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "U8S79zQU29", "reviewer": "Reviewer_Rk98", "summary": "This paper introduces a person pre-training framework PLIP, which consists of three pre-text tasks: text-guided image colorization (TIC), image-guided attributes prediction (IAP) and identity-based vision-language contrast (IVLC). Furthermore, a large-scale person dataset SYNTH-PEDES is constructed to facilitate the pretraining. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "In the IAP task, the paper employs attribute phrases masking, which differs from random masking. However, attribute phrases masking may overlook the fact that certain verbs can reflect relationships between entities. Have the authors conducted the experiment to make an experimental comparison between attribute phrases masking and random masking?", "limitations": "Please refer to Weaknesses and Questions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Good Writing**: This paper exhibits an overall clear storyline. I can easily catch the motivation and the contribution of this work.\n\n2. **Adequate Experiments**: The experiments are comprehensive and validate the importance of pedestrian pre-training data.", "weaknesses": "1. **Unfair Experimental Comparison**: My primary concern lies in the fairness issue of this paper. As mentioned in line 1046, SPAC uses CUHK-PEDES and ICFG-PEDES as training datasets, while the subsequent PLIP model leverages SYNTH-PEDES, constructed using SPAC, as the training corpus. **This results in information leakage from CUHK-PEDES and ICFG-PEDES into the PLIP model**. Therefore, the claimed zero-shot or domain generalization experimental comparisons are not fair, and such comparisons can not effectively demonstrate the superiority of the proposed PLIP method.\n2. **Undefined Abbreviations**: The abbreviations SIC, VLC, and VAP are not explained in Figure 2, making their significance unclear.\n3. **Lack of Comparison with Latest Works**: The compared methods are outdated. For example, the most recent methods referenced in Tables 6 and 7 were proposed in 2022.\n4. **Lack of Novelty**: The three pretext tasks proposed in PLIP are not novel. For instance, TIC from LapsCore, IAP from APTM, and IVLC from TBPS-CLIP have already been introduced. Therefore, these tasks are unlikely to provide valuable new insights for subsequent research in this field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. The concerns are answered as follows.\n\n**(i) Analysis of experimental comparison fairness.**\n\nWe believe that it might be due to some of our wording that led to your misunderstanding that our method would result in information leakage and issues of unfair comparison. In fact, our method **does not have an issue with information leakage**, and a series of fair experimental comparisons have also demonstrated the superiority of our approach. The detailed explanations are as follows.\n\n(a) We utilize the **training sets** of the CUHK-PEDES and ICFG-PEDES dataset to train our SPAC, ensuring that there is no information leakage from the test sets of these two datasets. \n\nSpecifically, in our method, the training sets from these two datasets are used to train our SPAC. Then, we employ SPAC to generate corresponding text descriptions for large-scale person images. Subsequently, we use these synthetic image-text pairs to train our PLIP model. Throughout this process, we do not utilize any information from the test sets of CUHK-PEDES and ICFG-PEDES. Therefore, when we validate the excellent performance of our method on the test sets of these two datasets, there is no issue of information leakage.\n\n(b) A series of experimental results demonstrate the superiority of our method in promoting various person-related tasks on many other datasets. \n\nFor example, on the unsupervised person ReID task, using PPLR as the baseline algorithm, our PLIP pre-trained model significantly boosts the mAP by 5.1% on Market1501 and 14.7% on MSMT17. It is worth noting that the Market1501 and MSMT17 datasets are not encountered during our pre-training process, hence there is no issue of information leakage. This further demonstrates that our PLIP can learn highly generalized person representations.\n\nIn conclusion, our pre-training framework avoids information leakage, and various fair comparative experiments have also confirmed its superiority. Meanwhile, thanks for pointing out the possible misinterpretations. We will clarify our use of the combined CUHK-PEDES and ICFG-PEDES training sets at line 1046.\n\n**(ii) Undefined abbreviations.**\n\nThank you for pointing this out. The undefined abbreviations in Figure 2 are an oversight. In fact, SIC should be TIC (Text-guided Image Colorization), VLC should be IVLC (Identity-based Vision-Language Contrast), and VAP should be IAP (Image-guided Attributes Prediction). We'll fix these and clarify them in the caption.\n\n**(iii) Comparison with latest works.**\n\nFirstly, the methods in Tables 6 and 7, though from 2022, are top pre-training models in person ReID. Secondly, to reproduce these methods for our experiments, they must have open-sourced their models and code, as we assess the impact of various pre-trained models on downstream unsupervised tasks, like in Table 6. Therefore, we compare key methods such as LUP, LUPNL, and PASS in these tables.\n\nIndeed, in many of the experiments, we have compared our PLIP with the latest works, such as SOLIDER (CVPR 2023) in Table 1, and IRRA (CVPR 2023), APTM (ACMMM 2023), RASA (IJCAI 2023) in Table 5. Notably, compared to the SoTA expert models for these different tasks, our PLIP has achieved consistently competitive or leading performance.\n\n**(iv) Novelty.**\n\nWe respectfully disagree with that the tasks of our PLIP are lack of novelty and unlikely to provide valuable new insights for subsequent research in this field. \n\nIt is worth noting that the three pretext tasks we proposed are not simply equivalent to the three works mentioned. In fact, there are significant differences between them.\n\n(a) The core motivation significantly differs from these works. The core motivation of all these works mentioned is to design various expert models specifically for the single task of text-based person Re-ID. However, our motivation is to propose a language-image pre-training framework to advance a series of person-related tasks rather than a single task. Alternatively put, our PLIP, unlike these works, is not confined to one specific task but is proposed to learn general person representations that can boost a wide range of person-related tasks.\n\n(b) The specific designs and implementation differ from these works. These works have comparabaly complex structures and loss function designs to maximize the performance on the single text-based person Re-ID task. These complex specialized designs are quite time-consuming and challenging to apply directly to large-scale pre-training. However, our PLIP not only deeply considers the characteristics of persons, but especially designs targeted, simple and efficient representation learning tasks, making it naturally adaptable to large-scale pre-training.\n\nFurthermore, our PLIP opens up a new research direction on how to utilize language information to learn general person representations suitable for various person-related tasks. We believe that our work can provide valuable insights for subsequent research from many aspects. For example:\n\n(a) How to generate more detailed fine-grained textual descriptions for images?\n\n(b) How to design more effective multi-modal pre-training tasks for person representation learning?\n\n(c) How can we add more modalities to pre-training? Does this improve model performance?\n\n(d) How to optimize pre-training data: balancing quantity and quality, and maximizing implicit knowledge extraction?\n\nThese are the directions our team will pursue. I believe researchers in our field will build on our work to explore and answer these questions.\n\n**(v) Masking strategies.**\n\nIn fact, in the IAP task, we randomly mask both attribute phrases and other non-attribute phrases, with attribute phrases being masked at a higher probability. We believe this approach encourages the model to focus more on extracting attribute information from person images. Meanwhile, we have conducted ablation experiments on different masking strategies in Section A.9.4 of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ezdsFTgg7e", "reviewer": "Reviewer_RKFb", "summary": "This paper introduces a Language-Image Pre-training framework, termed PLIP, designed for enhancing person representation learning. In order to better adapt to downstream person-centric tasks, three pretext tasks are designed to pay more attention to critical person-related characteristics, including Text-guided Image Colorization, Image-guided Attributes Prediction, and Identity-based Vision-language Contrast. In addition, a large-scale person dataset named SYNTH-PEDES generated by an image captioner named SPAC is employed during pre-training. And PLIP performs remarkable ability in various downstream person-centric tasks.", "questions": "The description in line 206 seems to violate the blind review rules. [such as the Cross-Modal Projection Matching (CMPM) loss [110] we adopt].", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors highlight a significant domain gap that exists between general pre-training technologies and person-centric tasks. To address this gap, the authors suggest a language-image pre-training framework, which represents a meaningful direction.", "weaknesses": "1.\tWhat is the core difference between the proposed pretext tasks and Lapscore [93]? The authors assert that the proposed IAP masks attribute phrases instead of color words, as done in Lapscore. While I agree with this distinction, it seems to be a minor difference based on certain operations.\n\n2.\tSome of the representations are unclear and lack specificity. For instance, the abbreviations in Figure 2 are not explained either in the caption or in the main text; the method/technology represented as Baseline in Table 1 remains unclear.\n\n3.\tThe writing should be improved. For example, the last paragraph in Section 2.1 appears to be somewhat absurd; Fig 4 and Fig 12 are same. \n\n4.\tThe experimental evaluation is incomplete, and certain comparisons lack logicality and fairness. The proposed framework comprises a methodology and novel datasets, which require separate meticulous verification. Otherwise, some comparisons may be unfair due to the proposed framework being trained on a larger dataset, SYNTH-PEDES. Additionally, there are existing works [1-2] that aim to construct large-scale text-image person datasets, and their performance should be compared with the proposed SYNTH-PEDES dataset.\n\n5.\tThe authors propose training an image captioner specifically for annotating person images. However, it begs the question of why not directly utilize the existing state-of-the-art MLLM technologies for image annotation? Contemporary MLLMs possess exceptional modeling capabilities. It is worth mentioning and discussing the existing person image captioning methods [1-3] that have been previously proposed.\n \n[1] 2024-CVPR-Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID\n\n[2] 2023-ICCV-Unified pre-training with pseudo texts for text-to-image person re-identification\n\n[3] 2023-ACMMM-Text-based Person Search without Parallel Image-Text Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. The concerns are answered as follows.\n\n**(i) Core difference between PLIP and Lapscore.**\n\nIn fact, there are many differences between our PLIP and Lapscore.\n\n(a) The different core motivation. The core motivation of Lapscore is to design some modules specifically for the single task of text-based person Re-ID. Unlike Lapscore, our PLIP is not limited to a single task; instead, it is designed to learn general person representations capable of boosting various person-related tasks.\n\n(b) The different core designs. Lapscore designs two color reasoning modules for text-based person Re-ID. The first uses LSTM to obtain the text embedding, which is then infused into each skip connection of the UNet architecture to achieve color restoration of grayscale images. The second utilizes MobileNet to extract features from color images, and then employs these features along with the Bilinear Attention Network to predict the masked color words in the text. These complex designs are challenging to apply directly to large-scale pre-training. However, our PLIP not only deeply considers the characteristics of persons, but especially make targeted, simple and efficient representation learning designs for each task, making it naturally adaptable to large-scale pre-training.\n\n\n**(ii) Some unclear representations.**\n\nThanks for pointing this out. The unexplained abbreviations in Figure 2 are an oversight. Specific explanations can be found in the second response to Reviewer Rk98. Notably, the methods as baselines in Table 1 are actually described detailedly in the table caption.\n\n**(iii) Writing.**\n\nThis paragraph dicusses the challenge of aligning fine-grained text with image regions, a future research direction for us. We will simplify the expression to make it more easily understandable. Also, we will polish the writing to avoid any issues such as repetitive images.\n\n**(iv) Experimental evaluation.**\n\nWe respectfully disagree with that the experimental evaluation is incomplete, and certain comparisons lack logicality and fairness. In fact, we have already conducted verification experiments for both the method and dataset.\n\n(a) We conduct comparative experiments on the pre-training method. In Table 18, to verify the superiority of our PLIP, we have pre-trained a series of models on the entire SYNTH-PEDES dataset with different pre-training methods, and compare their transfer performance on downstream datasets. The results show that our PLIP outperforms all other pre-training methods significantly on CUHK-PEDES and Market1501. Considering that the pre-training datasets are identical, these experimental results strongly verify the effectiveness of our method.\n\n(b) We perform extensive ablation studies to verify the effectiveness of the pre-training dataset. In Section A.9.3, through three experiments, we meticulously explore the effectiveness of various components of our dataset. A series of experimental results all indicate that pre-training on our dataset significantly enhances the learning of more generalized person representations. Considering that the pre-training methods are identitcal, these experimental results strongly verify the effectiveness of our dataset.\n\nAdditionally, thanks for pointing out these related works. Our work achieves comparable or leading performance compared to them, and we will include the comparative results in our future version.\n\n**(v) Advantages of specialized person image captioner and a discussion on existing methods.**\n\nIt is an interesting question. We believe that using a specialized person image captioner to annotate person images has the following clear advantages over using existing MLLMs.\n\n(a) The generated texts have **higher quality**. MLLMs are trained on extensive general-domain image-text datasets, which include a limited number of person images. The texts generated by MLLMs are often coarse, with low granularity, and are more likely to contain erroneous descriptions, which has been discussed in [4]. These low-quality texts are disadvantageous for model pre-training. However, by training on person-related image-text datasets, our image captioner is able to provide significantly more accurate descriptions of the appearance of persons. These accurate textual descriptions are a guarantee of our dataset's high quality.\n\n(b) The speed of text generation is **significantly faster**. Existing MLLMs generally have a large number of model parameters and slow inference speeds, which can lead to considerable time costs when generating text for large-scale images. In contrast, our captioner is a small expert model that generates text quickly and is well-suited for generating text for large-scale images.\n\nAdditionally, there is a discussion of the image captioning methods.\n\n[1] annotates images by having MLLMs fill in attribute words into human-designed templates. [2] uses CLIP to calculate the similarity between images and various attribute prompts to obtain person attribute information, which is then filled into a given template to produce final textual description. [3] first queries MLLMs for each attribute to obtain attribute information, and then either fills this information into a given template or further processes it with a language model to generate final textual description.\n\nWe can observe that these methods essentially leverage existing models to tag images with attribute labels. Since the utilized models, such as CLIP, have not been trained on specialized person data, there is a common issue of less accurate attribute tagging. In contrast, our captioning method is capable of generating more accurate and stylish textual descriptions. We will include a more detailed discussion in the paper.\n\n**(vi) Blind review rules.**\n\nAfter review, it has been found that this sentence does not violate the rules.\n\nReference\n\n[4] 2024-CVPR-UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ggWMukWvX", "reviewer": "Reviewer_cgBx", "summary": "A novel vision-language pre-training framework is proposed in this paper, termed PLIP, for person-centric downstream tasks: image-based Re-ID, text-based Re-ID, attributes recognition, human parsing and person search. To form PLIP, three pre-training tasks are designed: text-guided image colorization, image-guided attributes prediction and identity-based vision-language contrast. Also, the authors design a new caption model named SPAC to generate stylish textual descriptions for each person image. Utilizing SPAC, they construct a large-scale synthetic person dataset with image-text pairs to validate PLIP's effectiveness. A range of experiments have shown that the pre-trained models achieve state-of-the-art results on various downstream tasks, without the bells and whistles.", "questions": "1. For the IAP task, why don’t you use multiple binary classification heads instead of just one multi-classification head? Please explain.\n2. How does the TIC task help with learning person representations? I am looking forward a more detailed explanation.", "limitations": "The authors have discussed the limitations and broader impact in detail. I hold the view that they have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed PLIP is well-motivated and the combination of three pretext tasks explicitly helps to learn fine-grained cross-modal associations. That is to say, the design of the pretext tasks takes into good consideration the distinguish features of this research field, and it is simple and effective.\n2. The paper presents a new dataset called SYNTH-PEDES, which contains whopping 4.7M person images and 12M textual descriptions, making it by far the largest person dataset with image-text pairs. This dataset will undoubtedly become an important resource in this research field.\n3. The authors commit to open-sourcing the code, dataset and weight. This practice of open-source will have a positive impact on this research field.\n4. Extensive experiments provide strong support for the paper. Notably, the ablation studies and analyses are very comprehensive, investigating various settings and dataset quality evaluations.\n5. The improvements of PLIP models to downstream tasks are obvious. Compared with existing pretrained models, the PLIP models achieve consistent leading performance across a range of tasks and settings. Specially, PLIP brings a significant improvement to unsupervised ReID methods. For example, on the MSMT17 dataset, it improves the mAP of PPLR and ISE by 14.7% and 11.4%, respectively.", "weaknesses": "1. The proposed PLIP comprises three tasks, which seems somewhat complex. The popular CLIP, for example, only includes one contrastive learning task, whereas the method in this paper has three tasks, which should introduce greater computational overhead. I think it’s worth trying to make the pre-training method simpler.\n2. Since the identities in SYNTH-PEDES are obtained based on tracklets, and the textual descriptions are also synthetic, this dataset will contain some noisy data. \n3. There are some typo errors. For instance, the period in the ninth line should be changed to a semicolon. The authors should carefully review the full text to avoid these typo errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and appreciation of our work. The concerns are answered as follows.\n\n**(i) Somewhat complex designs.**\n\nConsidering that existing general-domain multimodal pre-training techniques do not take into account person-related characteristics, they are not well-suited for application in person representation learning. Therefore, we propose a novel language-image pre-training framework termed PLIP, with three elaborately designed three pretext tasks. These three tasks are obviously effective without introducing significant computational overhead.\n\n(a) Effectiveness. The effectiveness of these three tasks has been demonstrated across a series of experiments in this paper, with each task significantly enhancing the performance of the learned representations. For instance, as shown in Table 18, under a fair comparison setup with ResNet50 as the backbone, our PLIP outperforms CLIP by 14.8% and 11.0% in Rank-1 accuracy on the CUHK-PEDES and Market1501 datasets, respectively.\n\n(b) Computational overhead. Under the same experimental conditions, training the ResNet50 (or ViT-Small) for 70 epochs on the entire SYNTH-PEDES dataset using 4×Geforce 3090 GPUs takes approximately 12.4 days for CLIP, 18.1 days for BLIP, and 15.2 days for PLIP. It can be seen that our PLIP achieves significant performance improvement without introducing much additional training cost. Notably, when directly transferring to the CUHK-PEDES dataset, the rank-1 results of BLIP and PLIP are 44.9% and 52.9%, respectively, while the training time of PLIP is 2.9 days less compared to BLIP.\n\nCertainly, exploring how to simultaneously improve the efficiency and performance of pre-training methods is a highly meaningful research topic. Following your suggestion, we will continue to advance this in our future work.\n\n**(ii) Noisy data.**\n\nDue to the impracticality of manually annotating all images, noisy data is a common issue encountered by almost all large-scale image-text datasets. To minimize the amount of noisy data in the dataset as much as possible, we employed a series of denoising strategies to ensure the quality of the dataset to the greatest extent. The specific details of the denoising strategies can be found in Section A.7 of the paper. Meanwhile, we meticulously evaluated the quality of the dataset in Section 9.2 of the paper. The evaluation results indicate that the quality of our dataset is nearly on par with that of manually annotated datasets. \n\n**(iii) Typo.**\n\nThanks for pointing this out. Following your suggestion, we will carefully review the full text to avoid these typo errors.\n\n**(iv) About the prediction head of IAP.**\n\nThanks for your insightful question. The IAP task requires to predict the masked words in a textual description. Considering the entire vocabulary as the sum of categories, with each word as a separate category, the IAP task is akin to, given a masked position, selecting the most fitting word from the entire vocabulary. In essence, this is a multi-class classification task, and therefore we use the common multi-class classification head instead of multiple binary classification heads to predict the words.\n\n**(v) About TIC task.**\n\nThe pretext task of text-guided image colorization is first proposed in this paper and has been proven to be very effective for person representaion learning. It forces the model to understand the meaning of colors and attributes, and associates them with related visual regions, rather than simple memorization. This ability to distinguish between different parts of the person body leads to more discriminative person representation learning and guarantee the superior performance on many person-centric tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjBICJlw5N", "reviewer": "Reviewer_bKKs", "summary": "This paper presents a language-image pre-training framework and a large-scale synthetic image-text dataset for person representation learning. The proposed framework contains three elaborately designed pretext tasks: 1) Text-guided Image Colorization (TIC); 2) Image-guided Attributes Prediction (IAP); and 3) Identity-based Vision-Language Contrast (IVLC). Meanwhile, a large-scale person dataset with image-text pairs (SYNTH-PEDES) is constructed by automatically generating textual descriptions using the proposed Stylish Pedestrian Attributes-union Captioning (SPAC) method. The dataset contains over 4.7M images from 300K identities with 12M descriptions. Finally, the paper utilizes the pre-training framework to pre-train models on SYNTH-PEDES. Extensive experiments demonstrate the significant improvements brought by the pre-trained models to a range of downstream person-centric tasks on various settings. Also, the paper conducts extensive experiments to verify the superiority of the proposed pre-training framework compared with other SOTA pre-training methods.", "questions": "1. As shown in Table 4, for the proposed pre-trained model PLIP, CMPM/C achieves the best results, significantly surpassing the other two downstream algorithms. However, for all other pre-trained models, the LGUR achieves the best results. Can you explain the reason for this?\n2. In my opinion, the TIC task utilizes the textual descriptions with color words to restore the original color information of the gray-scale images, which could lead to learning comparably fine-grained information. Nevertheless, in some specific scenarios, like encountering a white T-shirt with a blue logo and vaguely described as “a white T-shirt,” could TIC result in learning suboptimal representations?", "limitations": "Since the research in this paper involves persons, it inevitably contains some privacy and security issues. However, the authors have discussed these concerns in detail in the “Broad Impact” and “Limitations” sections of the appendix and have taken measures to minimize the risk of privacy breaches. Therefore, I think the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper is well-written and easy to follow.\n2. The proposed pre-training framework considers the key characteristics of persons and effectively utilizes the correlations between images and texts. The learned representations through this framework show good generalization ability on extensive person-centric tasks.\n3. The SYNTH-PEDES dataset contains a large number of person images along with their corresponding text descriptions. The quality and diversity of the texts are quite good. This dataset can facilitate research on more cross-modal person-related domains.\n4. This paper conducts comprehensive experiments on many aspects, validates the powerful transfer and domain generalization capabilities of the pre-trained models, and evaluates the quality of the dataset.", "weaknesses": "1. The paper does not conduct pre-training on commonly used ViT.\n2. The entire content of the paper, including the appendix, seems somewhat verbose, totaling about 31 pages. The authors could be more concise in their presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and appreciation of our work. The concerns are answered as follows.\n\n**(i) Pre-training on commonly used ViT.**\n\nDue to the need for handling multi-scale image features in the TIC task of PLIP, we only consider the variants of vision transformers with hierarchical structure design, such as Swin Transformer, as the backbone in our PLIP. To verify the effectiveness, we perform a range of experiments with Swin Transformer as the backbone and achieve a series of results, shown in Table 1, Table 5, Table 7, Table 9, Table 18 and Table 21 of our paper. These experimental results all demonstrate the effectiveness of our pre-training method on the Swin-Transformer. Certainly, in our future work, we will explore more universal pre-training methods that impose fewer constraints on the model architecture. We hope the above response has addressed your concerns.\n\n**(ii) More concise presentation.**\n\nThanks for pointing this out. Due to the extensive experiments conducted to validate the effectiveness of each component in the proposed framework, as well as experiments on a wide range of downstream tasks, the length of this paper is considerable. Following your suggestion, we will refine our presentation with greater conciseness in the future version.\n\n**(iii) The reason for inconsistent improvements.**\n\nThanks for your insightful comment. Due to LGUR's high performance, other pre-trained models can achieve the best results on this method, yet our PLIP obtains the best results on CMPM/C, the reasons for which are explained below: \n\nA series of previous works in the field of multi-modal pre-training have demonstrated that, for pre-trained multi-modal models, excellent performance can be achieved with simple fine-tuning on downstream tasks, and additional complex model structure design is unnecessary. We must note that CMPM/C is a very simple algorithm that correlates the visual and textual global representation by designing only two loss functions, without any additional complex design for model structure. On the other hand, SSAN and LGUR are designed with highly complex additional structures for model training. In fact, through large-scale person-centric multi-modal pre-training, PLIP has learned a very discriminative multi-modal shared space for text-based person Re-ID. Therefore, simply fine-tuning PLIP with CMPM/C gives better results than SSAN and LGUR.\n\n**(iv) Fine-grained image with a vague text.**\n\nThanks for your valuable comment. In fact, due to the lack of sufficiently fine-grained texts in existing manually annotated datasets, there will inevitably be some relative vague texts in our synthetic dataset SYNTH-PEDES. We acknowledge that relatively vague texts naturally lead to the model not being good at distinguishing some very detailed features to a certain extent. However, we must note that our model has a preliminary understanding of the meaning of attributes and colors, and can associate them with related image regions. This ability to distinguish between different parts of the person body leads to discriminative person representation learning to some extent. We believe that not being good at distinguishing very detailed features will not have a significant negative impact on person representation learning. Obviously, if we want to further enhance the model's ability to perceive details, we can try manually annotating datasets with super fine-grained texts. We will continue to work towards this research direction with diligence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers' time and efforts in reviewing our paper. 

We are glad to find that reviewers generally recognized our contributions including the meaningful and novel pre-training framework (Reviewers cgBx, bKKs, RKFb), the significance of our proposed dataset (Reviewers cgBx, bKKs), the comprehensive experiments (Reviewers cgBx, bKKs, Rk98), the practice of open-source (Reviewer cgBx) and our well-written paper (Reviewers cgBx, bKKs, Rk98).

We will **open-source all the code, models and dataset**. We believe that our work will make great and timely contributions to this community. 

1) We have pioneered the introduction of language modality into general person representation learning and verified its effectiveness through comprehensive experiments. This learning paradigm offers a solid path to more powerful general person-centric models.
2) Our pre-trained models can directly improve existing person-related downstream methods to a much higher level without bells and whistles. These models will support various person-related task applications and promote the sustainable development of the technology.
3) Our SYNTH-PEDES dataset with high-quality person image-text pairs is the largest by now and can facilitate research on more cross-modal person-centric domains.

We have replied to each reviewer individually to address any concerns. I noticed that we can continue discussions on the OpenReview system for some time to come. If the reviewers has any questions, we would be happy to continue the discussion.

Once again we thank all reviewers and area chairs!

Meta-review: This paper presents a language-image pre-training framework and a large-scale synthetic dataset for person representation learning, with applications in person re-identification and search tasks. 

The initial reviews were mixed. Two of the reviewers recognized the innovation of the framework, the contribution of the datasets, and the strength of the empirical results. However, the other two reviewers raised concerns about potential information leakage during training and the technical novelty w.r.t. Lapscore (Wu et al., ICCV 2023).  The authors provided a rebuttal addressing these concerns. Despite this, the two critical reviewers maintained their stance and did not respond further to the rebuttal. Consequently, the final ratings remained polarized.

After a thorough review of the paper, the rebuttal, and the discussion, the AC believes that the main concerns raised by reviewers Rk98 and RKFb were adequately addressed in the rebuttal. The AC did not find compelling support in their reviews or the discussion to warrant rejecting the paper. Therefore, the decision is to recommend acceptance. The authors are encouraged to incorporate the reviewers' feedback in their final version.
Decision: Accept (poster)