ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
Authors: Jialong Zuo
Yongtai Deng
Mengdan Tan
Rui Jin
Dongyue Wu
Nong Sang
Liang Pan
Changxin Gao
Venue: NeurIPS 2025 poster ()
Keywords: Person Re-identification
Multi-Modality
Benchmark
Primary area: applications

Abstract: In real-word scenarios, person re-identification (ReID) expects to identify a person-of-interest via the descriptive query, regardless of whether the query is a single modality or a combination of multiple modalities. However, existing methods and datasets remain constrained to limited modalities, failing to meet this requirement. Therefore, we investigate a new challenging problem called Omni Multi-modal Person Re-identification (OM-ReID), which aims to achieve effective retrieval with varying multi-modal queries. To address dataset scarcity, we construct ORBench, the first high-quality multi-modal dataset comprising 1,000 unique identities across five modalities: RGB, infrared, color pencil, sketch, and textual description. This dataset also has significant superiority in terms of diversity, such as the painting perspectives and textual information. It could serve as an ideal platform for follow-up investigations in OM-ReID. Moreover, we propose ReID5o, a novel multi-modal learning framework for person ReID. It enables synergistic fusion and cross-modal alignment of arbitrary modality combinations in a single model, with a unified encoding and multi-expert routing mechanism proposed. Extensive experiments verify the advancement and practicality of our ORBench. A range of models have been compared on it, and our proposed ReID5o gives the best performance.

ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
Jialong Zuo
Yongtai Deng Nong Sang
Mengdan Tan"
Rui Jin Changxin Gao
Dongyue Wu
Liang Pan
National Key Laboratory of Multispectral Information Intelligent Processing Technology. School of Artificial Intelligence and Automation, Huazhong University of Science and Technology.
2 Shanghai AI Laboratory.
(jlongzuo, cgao) edu.
m Code & Dataset:
Abstract
In real-word scenarios, person re-identification (ReID) expects to identify person- of-interest via the descriptive query, regardless of whether the query a single modality combination of multiple modalities. However, existing methods and datasets remain constrained to limited modalities, failing to meet this requirement. Therefore, investigate new challenging problem called Omni Multi-modal Person Re-identification (OM-RelD), which aims to achieve effective retrieval with varying multi-moda queries. To address dataset scarcity, construct ORBench, the first high-quality multi-modal dataset comprising 1,000 unique identities across five modalities: RGB, infrared, color pencil, sketch, and textual description. This dataset also has significant superiority in terms of diversity, such as the painting perspectives and textual information. It could serve as an ideal platform for follow- up investigations in OM-ReID. Morcover, we propose ReID5o, novel multi-modal learning framework for person ReID. It enables synergistic fusion and cross- moosl alignment of arbitrary modality combinations in a single model, with a unified encoding and multi-expert routing mechanism proposed. Extensive experiments verify the advancement and practicality of our ORBench. A range of models have been compared on it, and our proposed ReIDSo gives the best performance.
1 Introduction
Person re-identification (ReID), fine-grained retrieval task, aims to search for person images of the target identity based on given descriptive query [51]. Due to the broad applications of Rell> in fields such as urban security, many researchers have conducted in-depth studies on it. It can be classified into single-modal retrieval [58, 45] and cross-modal retrieval [26, 46, 34]. The former involves the mutual search of RGB images, while the latter usually use queries from other modalities to retrieve RGB images. Considering the frequent lack of original RGB queries in real scenarios, the latter has increasingly drawn attention in recent years.
However, despite the considerable progress in this technology [61, 62, 24, 13], a challenging and practical problem, which aims to achieve effective retrieval with varying multi-modal queries and their combinations the RelD model, remains largely unexplored in previous researches.
We propose that exploring such a challenging problem, termed Omni Multi-modal Person Re- identification (OM-ReID), is of great significance. On the one hand, due the diversity of information acquisition, there is often a need for multi-modal queries in practical applications [50]. Users expect the model to be able to effectively process multi-modal information and provide comprehensive and accurate retrieval results. If the model can handle these varying modalities, it will significantly
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

boost its practicality. On the other hand, theoretically, different modalities provide abundant and complementary features [52, 5, 24]. For instance, RGB images are vulnerable environmental light variations [46], and infrared as well as sketch images lack essential color information critical for RellD tasks. Fusing multi-modal features akin to assembling an information jigsaw from diverse perspectives and dimensions. This enables a more comprehensive portrayal of person characteristics, thereby minimizing misjudgments stemming from single-modal limitations.
Considering the above aspects, and given that the re- lated researches are constrained by datasets involv- ing only few modalities and whose quality is not satisfactory [58, 46, 26, 34, 52], this paper makes the following contributions at the levels of both the dataset and the methodology.
Any coctraaioo:
Σ
1
We collect the first high-quality person RelD dataset with 5 modalities, named ORBench. As shown in
Slng Soxtaily
Fig. 2, for the same person, our dataset simultane- ously contains RGB images, infrared images, col- ored pencil drawings, sketch drawings, and textual descriptions, providing a comprehensive portrayal of the person. Moreover, our dataset exhibits out- standing characteristies in terms of diversity. In addition to the cross-camera perspectives inherent in the real-captured data [46, 56], the contained paintings simultaneously reflects the frontal, dorsal, and lateral appearance features of the same person. Furthermore, the text component adopts an highly detailed and unrestricted descriptive style. By per- forming elaborate manual annotation and recurrent corrections, ORBench has significant superiority in terms of quality and diversity, compared to existing datasets. It contains 45,1 13 RGB images, 26,071 I infrared images, 18,000 color pencil drawings, 18,000 sketches and 45,113 textual descriptions of 1,000 identities in total.
RelD5o
Ielaareaded
Exiiteg oxtiods
Seathh
Sama Kodilit
Moxiilly
Now-Moddbam
Figure 1: retrieval with any combinations of five modali- ties, adapting to various queries with different uncertain modalities in real scenarios. How- ever, existing methods [46, 26, 52, 5, 24] constrained to few modalities and are unable to achieve arbitrary retrieval with five modalities.
Our ReID50 can effectively conduct
In addition, as shown in Fig. 1, to tackle the new challenge of retrieving with multiple modalities, we further propose novel unified model named ReIDSo. by which the portrayals of the same person across different modalities and their combinations are explicitly correlated to deeply mine the identity-invariance ReID5o first encodes the inputs from diverse modalities into a shared embedding space via a multi-modal tokenizing assembler. Subsequently, a multi-expert router is devised to facilitate effective modality -speciii representation learning within unified feature extractor. Then, a feature mixture and a simple aligmment strategy [16] are utilized to implement efficient fusion of mudti-modal representations and their correlation learning, respectively. Compared with existing methods, RelD5o achieves the best performance and can serve as a simple but effective baseline method for our ORBench.
We conduct comprehensive experiments to verify the advancement and effectiveness of our dataset and method. It can be concluded that multi-modal combination querying brings significant performance improvements, and serves as a more practical approach in real scenarios. As a research focusing on person ReID with arbitrary modalities and their combinations, we believe that our first high-qualit dataset, ORBench, and our method, ReID5o, can inspire the subsequent development in this field.
2
ORBench Dataset
2.1 Construction Motivation
Multi-modal learning [44, 60, 22] has become hot topic in the computer vision field. However, when directing to person RelD, researchers often confine themselves to datasets with few modalities and unsatisfactory quality [26, 34, 52, 5, 24]. Therefore, we construct ORBench, the first high-quality dataset with five modalities, driven by the following two major motivations.
Filling the modal gap and improving data quality. In the field of person ReID, previous datasets only covered few modalities, making it difficult to comprehensively depict the characteristics of datasets [52, 5] is obviously poor, and only
persons. In addition, the quality of some

---

RGB
Infrared
One Person
Five Modalities Sketch
Text
Color Pencil
bright
pmspectaned plopiaua Sh hears patr of Mark-rtemed glasem with bet bair apod is hish moarsal coler whrch deep Nace stit adiand bie denin that mukiilisl Ske rolorn smnkes on her ter, with wille
shostskkets
light
Cross-modal images collected Irom real scenes, Including a total of 13 RGB cameras and 11 IR cameras.
Highl detailed text descesptions The average word number of each descrigtion maches 71.2
soseensiinnn coseenaiin High-quiitt perntings miltyle perspectives, persan
3 perspectives color pencil drawings and 18 sketch drawings.
Figure 2:
The overview of our proposed ORBench dataset. Our dataset is remarkable for containing rich, high-quality and diverse five-modal descriptive data for the same person, offering comprehensive and in-depth resource for person ReID research.
rough generation strategies are utilized to obtain unrealistic multi-modal data. Therefore, to fill such a gap, we are determined to construct a new high-quality dataset that encompasses five modalities in a pioneering way, faithfully covering the characteristics of persons.
Promoting the development of multi-modal technologies and driving innovation. Multi-modal fusion and retrieval technologies [59, 42] have great potential in person RelD and there is an urgent need for their practical applications. However, due to the lack of suitable multi-modal datasets, their development is severely restricted. Therefore, to build an ideal platform for the development of this technology and drive innovation, we construct our ORBench, serving as the first high-quality multi-modal dataset in this field.
2.2 Dataset Collection
Based on the existing RGB-infrared datasets [46, 56], our ORBench dataset, with the cooperation of manual annotation and image generation experts, achieves the supplementation of additional modalities including color pencil drawings, sketch drawings, and textual descriptions. In addition, a manual correction and review mechanism has been incorporated to ensure the high quality.
For the RGB and infrared data, we manually selected persons with representative appearances from the existing datasets SYSU-MM01 [46] and LLCM [56]. That is to say, we filtered out those person images with poor imaging conditions and weak clothing representativeness. For example, if the appearance features of person can hardly be seen in both the dual-modal images, the data of this person will be removed. Eventually, we obtained 45,113 13 RGBs and 26,071 IRs of 1,000 persons.
For the color pencil data, we utilized an online image generation model [1] as our painting agent, and incorporated extensive manual supervision to ensure the drawing quality. Specifically. for each person, we first selected some mulli-persppectin them with detailed textual descriptions. Then, we required the agent refer these descriptions and images to paint color pencil drawings of the person's 's front, back, and side views. In addition, with many color pencil drawings painted, manually selected six most satisfactory drawings for each view, to further ensure the quality. Eventually, we obtained 18,000 color pencil drawings.
RGB images with high quality, and manually annotated
For the sketch data, considering the maturity of the sketch synthesis technology, after conducting extensive investigations, we decided to utilize the Meitu application software [2] to directly convert the obtained colored pencil drawings into realistic sketch drawings. For the text data, we hired few unique workers to be involved in the text annotation task and instructed them to describe the important characteristies for each RGB person image in detail. Through the above two procedures, we obtained 18,000 sketch drawings and 45,113 textual descriptions in total.
2.3 Dataset Statistics
Benefiting from our meticulous and labor-intensive manual annotation, ORBench is full of rich and high-quality multi-modal data. Compared with existing ReID datasets, our ORBench enjoys the following advantages:
Most Modalities to Date. Most previous multi-modal datasets [46, 26, 34] have merely two modalities, while for few tri-modal datasets [52, 5], the extra modalities are acquired via rough data synthesis strategies. As shown in Tab. 1, ORBench is the person RelD dataset boasting the greatest number of modalities so far, offering more comprchensive and detailed portrayal of persons.
3

---

Table I: Comparisons with other ReID datasets. ORBench is the first to have rich five-modal data and is the only dataset covering color pencil drawings. Only relatively high-quality datasets with manual annotations are listed, and low-quality synthetic datasets are excluded.
Datasets
Modality
#ldentities
#RGB Imgs
Imgs
RCP Imgs
RSK Imgs
#Texts
Murket SSI [58]
2015
1,501
32,668
CUHK- PEDES [26]
2017
R.T
13,003
40,206
80,412
MSMT17 [45]
2018
PKU.Sketch [34]
2018
R.S
200
400
200
SYSU-MMO1 [46]
2020
R.I
491
30,071
15.792
ICTG-PEDES [8]
2021
R.T
4.102
54,522
54,522
TrikelD [52]
2022
R.S.T
200
5,600
200
5,600
LLCM [56]
21,141
MaSkIK [30]
2023
R.S
1,501
32,668
4,763
UHine6926 [63]
2024
R.T
6.926
26,206
52,412
ORBench
R.I.S.T
45,113
26,071
18,000
18,000
45,113
High Quality, Existing tri-modal datasets often have poor quality due to rough data synthesis. For instance, C-CCU---EDDE [6] just converts low-res RGB images sketches, resulting in blurriness, missing parts, and distortion. TriRelD [52] utilizes limited-capacity caption models for rough texts. However, ORBench ensures high-quality data through careful, labor-intensive manual annotation. More examples are shown in the supplement.
Competitive Dataset Scale. As shown in Tab. 1, compared with existing datasets, the scale of our dataset is highly competitive, containing 45,113 RGB images, 26,071 infrared (IR) images, 18,000 color pencil (CP) drawings, 18,000 sketch (SK) drawing and 45,113 texts of 1,000 persons. Notably, our scale especially outshines those datasets with paintings. For example, compared with the previous largest sketch dataset, MaSk1K [30], we have approximately 4 times the number of sketches.
2.4 Evaluation Protocol
There are 1,000 valid identities in ORBench dataset. We have fixed split using 600 identities for training and 400 identities for testing. During training, all multi-modal data for the 600 persons the training set can be applied.
During the testing phase, the samples from the RGB modality are regarded as the gallery set, while the samples from the remaining four modalities and their combinations are treated as the corresponding query sets. Specifically, we design four search modes: angle-modal search (MM-1), dual-modal search (MM-2), tri-modal search (MM-3), and quad-modes search (MM-4). For single-modal search, the samples of each modality form their respective query sets. For the remaining three multi-modal searches, we first select a primary modality. Then, for each sample of this modality, we randomly select samples of the same identity from the remaining modalities as supplementary ones to form query tuple. Totally, there are 4 query sets for MM-1, 12 for MM-2, 12 for MM-3, and 4 for MM-4. The performance each mode measured as the average of retrieval results across its query sets.
As common practice. we adopt Cumulative Matching Characteristic (CMC) and mean average precision (mAP) evaluate the performance. Given query, all gallery images are ranked according to their affinities with the query. A successful search is achieved if any image of the corresponding person is among the rank-k images.
2.5 Discussion on Realism and Idealized Assumptions.
It is important to acknowledge the idealized assumptions underpinning dataset construction. We operated under the premise that comprehensive eyewitness information enables the creation of a complete textual description and, subsequently, a high-fidelity visual rendering of the target. Consequently, the identity consistency across modalities in ORBench is intentionally high.
In the taxonomy of [20], sketches and color paintings best characterized as "viewed precccs depictions derived from ample information rather than incomplete or inac-
sketches" precise curate "forensic sketches" that result from the fallibility of human memory. This design choice was necessitated by the feasibility of large-scale dataset construction. While ORBench provides a foundational, high-quality henchmark for studying multi-modal alignment in controlled setting, it does not fully capture the noise and variance inherent in real-world forensic scenarios. We posit that this clean dataset serves crucial starting point and powerful pre-training resource.

---

3 RelD5o: Method
In this section, we propose a unified multi-modal learning framework for person ReID, named RelDSo. We utilize multi-modal tokenizing assembler to encode the shared embedding space. Then, a unified encoder is used to extract modality-share features, and a multi-expert router is designed to facilitate modality-specific representation learning. Also, a feature mixture fuses the multi-modal features. Finally, we use the SDM [16] and identity classification losses to implement cross-modal alignment. The whole schematic is shown in Fig. 3.
inputs into
3.1 Multi-modal Tokenizing Assembler
Festure Misture
Coosi Algaacs
lneettty Chosott
To encode varying-modal data inputs into shared embedding space, we design a straight- forward multi-mode tokenizing assembler. This assembler is composed of five sub-tokenizers, which are respectively used to tokenize the RGB, IR, CP, SK, and text data, In addition, it will also provide discrete control signals for the sub- sequent multi-expert router according to the modality difference.
Multi-modal Encoder
Mutii-modal Tokenidng Assmnler
Multi-Expert Router
Specifically, for these four types of visual inputs, we use non-shared visual tokenizers [38] with the same structure to encode them respectively. This visual tokenizer consists of simple con- volutional layer and an LN layer. addition, we add an additional class token and positional embeddings as a common practice. For text in- put, following standard convention [38], we first add special flags at the beginning and end of the text respectively. Then, we employ vocabu- lary mapping layer and positional embeddings to project the input text into a nigh-dimensional the process of obtaining its corresponding embeddings can be formulated as follows.
1
Actencad Sired
Kociall Specia Toir
ORflreck
Wailally Pachh Tikeos
Inge
Disuat
Figure 3: The schematic of our proposed ReID5o framework. As the unified multi-modal encoder extracts the modality modality-shared features, specially designed multi-expert router can effectively pro- mote the modality-speciffe representation learning.
embedding space. Formally, for the input data X'mod
E Emdd mod mad
X mmdd)
(I)
representing that the input X mod encoded into n tokens with dimension D emnod represents a binary signal for modality activation, which is used to control the subsequent multi-expert router. Here, y ymd represents the tokenizer of each modality and mod E (R, I, C,S,T) which respectively represent the RGB, Infrared, Color pencil, Sketch, and Text modalities.
where Emodd E DR
3.2 Multi-Expert Router
We utilize unified multi-modal encoder to extract modality- shared features, which inherits the rich multi-modal pre-trained knowledge from CLIP [38] to ensure a good initial starting point for subsequent training. As a Transformer, it contains multiple linear layers. For certain linear layer W E Rddk its function can be reduced as y = W(x) (x). where x Rd and y E IR* represent the input and output feature for this layer, respectively. It can be seen that if the same linear layer is used directly for different modalities, is hard to extract modality-specific representations.
Therefore, we propose a simple yet effective multi-expert router to explicitly promote the modality- specific representation learning based on the modality categories. Specifically, we incorporate multiple modality-speciff low-rank matrices [15] into each linear layer as efficient feature extraction experts. In the absence of data input, these experts stay inactive. Once the assembler encodes data from particular modality, the router will receive the associated control signal and activates the corresponding expert, enabling modallit--speciffi parameter updates. Formally, for the input feature xodd of certain linear layer W, its output feature can be formulated as follows.
mod
mod + mod A Ammdd mod
(2)
W mod + mod Booo A mod x mod
5

---

where B E Rd A E R and r < min(d, k). cmnod is a binary control signal from the assembler, indicating whether the expert AWmod is activated. We can observe that for any modal input x mod, while W used to extract the modal shared features, AW will be employed extract the modal unique features. In this way, we can effectively uncover the modal invariance and discrimination.
3.3 Feature Mixture and Learning Strategy
Now, for the input data X through the aforementioned tokenization and expert routing mech- anisms, we can obtain its output features Zmnd E RTXD. Then, to achieve full complementary information interaction between different modalities, we employ efficient feature mixture to fuse the features of varying modality combinations. The feature mixture consists of a multi-head self attention (MSA) layer, 1-layer transformer block and a multi-layer perception (MLP). Then, the com- binatorial traversal is performed on the entire set of multimodal query features {z', zC, Z )² We generate all possible combinations from single- modal to quad-modal. These combinations are then concatenated and processed by the Feature Mixture module.
Taking the combination of Z' and Z" as example, the feature fusing process can be formulated as follows.
z MMPCTFMMSLL
z") ()))),,
where z - E RR,, indicating the IR and Text fused representation. CAT(-), LN(-)) and TF(-) denote the concatenation, layer normalization and transformer block, respectively. The last layer of the MIP is an average pooling layer.
In addition, we directly regard the output of special token the single-modal representation z mod As a simple learning strategy, we adopt the commonly used SDM loss [16] and identity classification (IC) loss supervise the learning process. Considering that RGB modality usually utilized as the gallery set and contains more original and complete information, we take it as the core target for feature alignment. We explicitly conduct the alignment of diverse modality combinations. To realize this, the overall learning objective can be written as follows.
C
Σ SDM(2 20t)+ + EIC(Z)) Rec
where & E C S, S, and S = (R, S,T) a a manually set fixed which is used to control the importance of the identity classification loss.
Experiments
There is existing person RelD) methods specifically designed to achieve the retrieval of any combinations of these five modalities. We investigate a wide range of possible solutions based on some leading general multi-modal models [38, 54, 11] and cross-modal person ReID models [5, 61, 24, 16, 36], and compare these solutions with our proposed method. We also conduct experiments to verify the data quality of our proposed ORBench dataset. Extensive experiments and comparisons demonstrate the superiority of our ORBench dataset and ReID5o method.
4.1 Experimental Settings
Datasets. The main experiments are conducted on our proposed multi-modal ORBench dataset. This dataset is divided into training set with 600 persons and a testing set with 400 persons. If there is no special statement, the performance is evaluated with the four proposed search modes. In addition, CUHK-PEDES [26), ICFG-PEDES [8] and UFine6926 [63] are used for comparisons.
Implementation Details. We employ a pre-trained multi-modal encoder [38], i.e., CLIP-B/16. as the unified encoder, which is pre-trained on billions of image-text pairs with contrastive learning. In the actual implementation, to fully leverage pre-trained knowledge, we employ an independent text encoder for the textual modality, while adopting the aforementioned strategy for other visual modalities. For the multi modil tokenizing assembler (MTA), the tokenizers for all modalities are initialized simultancously with the pre- pre-trained parameters in CLIP [38], and will not be shared during the subsequent training. For the multi-expert router (MER), each modality will be assigned an independent expert, and the low-rank matrices [15] of a certain expert will be installed on each linear layer of the encoder, where r is set to For each layer of the feature mixture (FM), the hidden size and number of heads are set to 512 and 8. The papemmprrretter for the ID loss is set to 1.0. During

---

training. all images are uniformly resized to 384 x 128 and the maximum length of the textual tokens 77. Also, random erasing. horizontally flipping and crop with padding are employed for image augmentation. Random masking and replacement is employed for text augmentation. Our RelD5o is trained with Adam [19] for 60 epochs with an initial learning rate 1 We spend warm-up epochs linearly increasing the learning rate from le to 1e-6. For the random- iandom-iiitialized experts and feature mixture, the initial learning rate is set to 5e We use a single A100 80GB GPU.
4.2 Ablation Study and Analysis
Effectiveness of Each Component. To verify the contribution of each component ReIDSo, conduct ablation experiment our OR- Bench. The results are reported in Tab. 2. In the baseline No.0, all visual modalities use the same shared visual tokenizer, and there is no expert router extract modality-speciiff features. In addition, simple average pooling is employed for multi-modal feature fusion. As evident from Tab. 2, compared with the baseine method, each introduced component proves crucial for the overall performance of our ReID5o model, and combining all of them leads to the best performance.
Table 2: Ablation study on each component of ReID5o. We report the mAP results for each search mode, with the best in bold.
Components
ORBench
No.
MIA MER FM
MM-I
MM-2
MM-3
MM-4
0
49.24
66.36
74.21
78.42
51.85
67.43
75.43
79.81
2
4
56,99
73.58
81.42
85.14
52.80
69.23
77.70
81.87
58.09
75.26
82.83
86.35
Analysis of the Multi- Multi-Expert Router. To ana- lyze the effecctiveness and efficiency of the multi- expert router, we conduct ablation experiments on the setting of in the low-rank matrices. In Tab. 3, we present the performance of each set- ting, as well as the introduced additional pa- rameters and FLOPs compared to the baseline method (w.t. MER). We also analyze simple and straightforward approach, that is, utilizing unique encoders encode each modality sepa-
Table 3: Different settings of the multi-expert router. We report the mAP results for each search mode, with the best in bold.
Method
Params. (M)
FLOP..
ORBecch
(G)
MM-1
MM-2
MM-3
MM-F
WA
MER
52.30
69.23
77.70
81.87
3.64
58.09
75.26
86.35
r-8 8
4.72
7.29
57.59
74.59
82.33
85.71
14.57
r- 32
18.87
29.14
57.56
74.89
82.39
85.72
UE
256.4
52.42
69.13
76.36
80.15
rately (w. UE). We can observe that UE fails to bring significant improvement while introducing extremely large additional parameters. However, when 4, our multi-expert router achieves the best performance, and maintains a relatively low additional computational cost.
Analysis of Multi-modal Complementarity. Taking the text modality as the primary modal- ity, explore the impact of the introduction of different modalities on its retrieval results, so to investigate the complementarity among multi-modal data. As shown in Tab. 4, the in- troduction of any additional modality into the query can directly enhance the retrieval perfor- mance. Among them, the promoting effect of the C modality is the most obvious. Compared with the T modality, when using T and C modal- ities, the mAP has increased by 26.26%. In addition, regarding the C modality, the complementarity effect of the modality is significantly better than that of the S modality. Compared with T + C, T I C can bring about 5.61% increase in mAP, while + C +S only brings a 1.91 % increase.
Table 4: Queries with different modality combi- nations. The text modality is the primary modality. The best results are in bold.
ORBench
Modality
Rank-1
Rank-5
Rank- 10)
mAP
63.15
77.94
82.92
54.88
T+I T+C
81.19
92.19
95.19
70.77
91.01
96.83
98.04
81.14
T+S
84.71
93.39
95.55
74.01
T+I+C
95.85
99.25
99.68
86.75
T+l+S
92.85
98.01
98.94
81,98
T+C+S
93.19
97.80
98.83
83,05
T+I+C4S
96.79
99.37
99,78
87.46
Different Learning Strategies. The differ- ent implementations of the loss functions for the learning strategy are shown in Tab. 5. We have implemented one instance-level loss func- tion [38] and three identity-leve loss func- tions [17, 53, 16] for the cross-modal alignment.
Table 5: Ablation study on different learning strategies. We report the mAp results for each search mode, with the best in bold.
ORBench
Strategy
MM-1
MM-2
MM-3
MM.4
ITC [38]
53.44
67.35
74.62
78.25
As we can see, compared with other losses, the SDM loss [16] leads to the best performance. In addition, introducing the identity classifica- tion (IC) loss will further improve the model's performance to 58.09% mAP for the single-modal search mode.
SupITC [17]
48.16
64.84
73.03
77.34
CMPM [53]
48.02
67.27
75.41
79,51
SDM [16]
57.65
74.51
82,14
85.60
SDM+IC [16]
58.09
75.26
82.83
86,35
7

---

Table 6: Performance comparison with existing methods on ORBench. These methods can be divided into cross- modal methods and multi-modal methods. The former performs alignment between two modalities, while the latter performs alignment among multiple modalities. Each method has been fine-tuned on our ORBench. The results* are the outcomes of our reproduction efforts.
Single-modal Search
Dual-modal Search
Tri-modal Search
Quad- Search
Method
R@1
R@10
mAP
R@1
R@
mAP
R@1 1
R@ 10
mAP
R@1
R 10
mAP
Cross-modal Models
CLIP [38]
59.84
77.88
48.73
80.31
92.80
65.79
87.67
96.78
73.17
90.71
98.25
76.69
PLIP [61]
60.57
78.24
48.79
80.78
92.82
65.63
88.14
96.98
74.60
90.95
98.
77.87
IRRA [16]
61.17
79.30
51.42
82.50
94.17
68.83
89.70
97.39
75.86
92.65
98.49
79.15
RDE [36]
60.91
79.20
48.82
82,36
93.95
67.52
89.94
97.50
75.27
92.75
98.63
78.64
Multi-modal modal Models
Meta-TF* [54]
6,25
19.46
4.24
10.81
31.26
6,68
14.82
40.45
8.65
18.32
47.66
10.45
ImageBind" [11]]
60.18
78.32
48.61
81.78
93.18
67.02
88.73
97.23
74.20
91.05
98.36
77.36
UNIReID [5]
59.49
77.59
47.98
80.91
93. 16
66.83
88.71
97. 01.
74,86
91.34
02
78.12
AIO* [24]
50.44
70.88
40.14
58.82
81.67
49.34
69.72
89.34
58.58
75.33
92.52
63.48
ReID5o
68.12
86.36
58.09
86.15
96.76
75.26
93.53
99.15
82.83
96.25
99.70
86.35
ORDaad
CUEX PEDES
ICFC PIDES
4.56
SSaniion Hhiepy
Figure 4: Comparisons of the Shannon entropy per textual description with existing person ReID datasets containing the text modality. The average entropy of our dataset reaches 5.53, representing the highest level of textual information richness among current datasets.
4.3 Comparison with Existing Methods
As shown in Tab. 6, we compared the performance of extensive possible methods on our ORBench. These methods can be divided into multi-modas methods in the general domain and cross-modal methods in the field of person ReID. The former usually aligns data from multiple modalities into the same feature space, such as images, audio, videos, and texts. For the former methods, we treat each visual modality in ORBench as unique image modality for them. That is, we will utilize four different image tokenizers and then perform training on ORBench. For the latter, since most methods only achieve image-tex alignment, for this type of methods, we directly regard the visual data of the four modalities as the image modality of these methods. However, for the AIO method [24] that achieves alignment among RGB, IR, SK, and Text, additionally add CP tokenizer to its framework and conduct fine-tuning on our ORBench using its default settings. In addition, since none of these methods achieve explicit alignment between any modality combinations and RGB, implement an equivalent multi-modal search mode by superimposing the similarity lists of queries and gallery for each modality. We observe that, compared with the existing methods, RelD5o achieves the best performance in all four search modes. These results validate the effectiveness of our method in leveraging arbitrary modality combinations for person ReID, and can serve the first strong baseline method for our ORBench.
4.4 Dataset Quality Assessment
(thelly
2.4
Considering that we additionally anno- tate data of three modalities to the ex- isting RGB-infrared datasets, and the sketch data is converted from the color pencil data, we conduct experiments to explore the data quality of the text modal- ity and the color pencil modality. For the text modality, we use Shannon entropy to measure the amount of information contained the texts. Specifically, we can treat each word as an event, calcu-
1.8
1729
1587
1.2
0.6
19
127
70
113
134
0.0
0.0
Figure 5: Public evaluation to assess the identity con- sistency and perspective comformity of the color pencil drawings in ORBench.
8

---

late the probability of each word occurring, and finally calculate the entropy of the entire text. We conduct statistical comparison of the Shannon entropy per text in our ORBench with those in other datasets [63, 26, 8], as illustrated in Fig 4. We can see that our ORBench has richer textual information than other datasets, verifying the high quality of the text modality. For the color pencil modality, we adopt the method of public evaluation assess the identity consistency and perspective conformity of the color pencil drawings. Specifically, we randomly pick 2,000 samples from the color pencil drawings and split them into 20 groups. Then, 20 evaluators are asked to objectively score these samples on identity consistency with the original RGB images and conformity of drawing perspectives. Higher scores mean better evaluations. Detailed evaluation processes be found in the supplement. As shown in Fig 5, the overall quality the color pencil modality is quite high.
4.5 Dataset Generalization Evaluation
We conduct experiments com- pare the generalization perfor- mance of various models trained on different existing multi-modal datasets to other manually an- notated datasets. The subtasks divided into text-based trieval and sketch-base based retrieval. For the former, we use CUHK- PEDES [26] as the target dataset. For the latter, we used PKU- Sketch [34] as the target dataset. To ensure a fair comparison, we
Table 7: Evaluation comparison on dataset generalization. "T", "C", "T"", "P", """""""""""""" "O" denote Tri-CUHK-PEDE PEDES [5], CUHK- PEDES [26], ICFG-PEDES [8], PKU-Sketch [34], AIO [24] and our ORBench, respectively. The target datasets are all manually annotated woaldworld datasets. We report the rank-1 results
Method 0
T-1
C->l
OP P P
Cross-modal Models
PLIP [L1P]61]
52.61
49.23
49.23
IRRA [16]
38.52
32.57
32.57
SketchTrans [6]
70,30
66.30
62.70
Multi-modal Models
UNIReID [5]
36.62
33.64
77.80
69.80
65.40
ImageBind [I]]
38.58
35.33
75.40
68.60
61.80
ReID50 43.27
38.65
80.20
71.80
67.00
employ the same number of train-
ing samples for each training dataset, i.e., 40,000 training samples are randomly selected from each dataset. As the results shown in Tab. 7, for all models, compared with other multi-modal datasets, training on our ORBench dataset can achieve better generalization performance. This strongly demon- strates the high quality of our dataset and its generalizability to real-world application scenarios. Researchers can have full confidence in using our high-quality ORBench dataset.
4.6 Visualization
To demonstrate the role of multi-modal queries more clearly, we visualize re- trieval results on ORBench using diverse modality combinations as queries, rang- ing from single-modal to multi-modal inputs. As shown in Fig. 6, the addition of supplementary modalities such as integrating text with infrared or sketch sign-signifycaatly improves retrieval ac- curacy, with top-ranked results becom- ing more relevant. This highlights that multi-modal queries, by capturing com- plementary information (c.g., semantic descriptions and visual details under var- ied conditions), form a more complete depiction of search targets. For practical applications, this im- plies that leveraging multi-modal descriptive queries-ratie than relying single modalities can enhance retrieval precision, enabling more accurate identification of targets in complex scenarios.
II
Sk
IR
Retrioval Results
Figure 6: Visualization the top-10 ranking lists for four queries with different modalities ORBench.
5 Related Work
Multi-modal Learning. The goal of multimodal learning is to fully integrate heterogeneous in- formation from multiple modalities to obtain more representative representations of features [29]. With the popularity of Transformers, researchers have proposed unified architectures to process multimodal inputs end-to-end [18, 9, 25]. Although most models [3, 43, 35] do not face the problem

---

of missing modalities, this problem frequently occurs in real-world applications. Recent works [4, 33, 40, 11, 32] have introduced various strategies to mitigate this impact. Zhang et al.[57] align information from different modalities by mapping them into a shared feature space. Lee el al. [21] employ
prompt learning to deal with modality absence. Lian el al. [28] propose a graph-completion network to jointly optimize classification and reconstruction tasks in an end-to-end framework.
Cross-modal Person ReID. Compared to single-modal person ReID tasks [23, 31, 55, 7, 14], cross-modal ReID tasks [5. 46] typically use RGB images as retrieval targets while employing text descriptions [26, 39, 10, 48, 27]. infrared images [37, 49, 47], attributes [41], or sketches [12, 6] as queries to simulate real-world scenarios with uncertain query modalities. Li et [l.[ pioneered text-based person RelD. Wang et [l.[41] proposed a joint attribute-identity method to learn attribute- semantic and identit-diiscrminatii features. Qiu et al.[37] introduced a high-order structure-based intermediate feature learning network to fully exploit structural information in IR features. Lin er al. [30] designed mechanisms to mitigate the impact of sketches painter's subjectivity on Rell) performance. Subsequent work combines some modalities address practical constraints in obtaining target modalities and using complementary information. Chen et al.[5] fused features of RGB, sketch, and text modalities for modality-agnoste handling RGB, infrared, sketch, and text modalities. However, existing works are still constraint to scenarios with few modalities and overlook the rich cross-domas information contained in different modality combinations.
retrieval. Li et al. [24] developed unified framework
Limitation
While the proposed RelD5o framework and the ORBench dataset establish strong foundation for OM-ReID research, we acknowledge several limitations that point to future directions. First and foremost, ORBench is constructed under idcalized data generation assumption. The high fidelity of its sketch and color painting queries, which resemble "vicwed sketches" rather than imperfect "forensic sketches," " ensures tight cross-modal alignment but does fully encapsulate the noise and variance (e.g., inaccuracies, omissions) prevalent in real-world scenarios. Consequently, while our model demonstrates robust performance this clean benchmark and shows promising generalization in cross-dataset tests, its performance in truly noisy, in-the-wild settings requires further investigation. Secondly, our work is currently limited relevant modalities, such as thermal infrared or radar, are not included, and our framework is not yet designed for incremental learning of new modalities. Finally, the study focuses closed-set identification task, leaving open- open-world scenarios with unseen identities as an open challenge. We believe addressing these limitations-bu by introducing realistic noise, extending modality coverage. and moving towards open-set recognition- will be crucial steps for transitioning OM-ReID from a controlled benchmark to robust real-world applications.
predefined set of five modalities. Other practically
7 Conclusion
This paper investigated the problem of Omni Multi-modal Person Re-identification (OM- ReMR, where the goal to retrieve a person using a query from any single modality or their arbitrary combinations. To support this research, we constructed ORBench, the first comprehensive dataset for OM-RelD, featuring 1,000 identities across five modalities (RGB, infrared, sketch, color pencil, and text). We also developed RelD5o, a flexible framework that enables unified encoding and effective retrieval for any modality input. Our experiments demonstrate that leveraging multi-modal queries substantially improves retrieval performance compared to single-modality approaches, underscoring their practical value. We believe that the ORBench dataset and the RelD5o framework establish a solid foundation and strong baseline for future research in this emerging field.
8 Acknowledgments.
This work was supported by the National Natural Science Foundation of China No.62176097, and the Hubei Provincial Natural Science Foundation of China No.2022CFA055. We would like to sincerely thank the HPC Platform of Huazhong University of Science and Technology for providing computational resources.
10

---

References
[1] Doubao, text-to-image model,
general
intelligent
drawing.
[2] Meitu, structured sketch. https://ai.meitu.comm/
[3] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-End Referring Video Object Segmentation With Multimodal Transformers. CVPR, 2022.
[4] Lei Cai, Zhengyang Wang. Hongyang Gao, Dinggang Shen, and Shuiwang Ji. Deep Adversarial Learning for Multi-Modality Missing Data Completion. In KDD, 2018.
[5] Cuiqun Chen, Mang Ye, and Ding Jiang. Towards Modality-Agnostii Person Re-Identification With Descriptive Query. In CVPR, 2023.
[6]
Cuiqun Chen, Mang Ye, Meibin Qi, and Bo Du Sketch Transformer: Asymmetrical Disentan- glement Learning from Dynamic Synthesis. In ACMMM, 2022.
[7] Jiyuan Chen, Changxin Gao, Li Sun, and Nong Sang. Cesd: cross-camera self seff-distillation for unsupervised person re-identification. Visual Intelligence, 1(1):27, 2023.
[8] Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao. Semantically self-aligned network for text-to-image part-aware person re-identification. arXiv preprint arXiv:2107. 126666 2021.
[9] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal Transformer for Video Retrieval. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, 2020.
[10] Chenyang Gao, Guanyu Cai, Xinyang Jiang, Feng Zheng, Jun Zhang, Yifei Gong, Pai Peng, Xiaowei Guo, and Xing Sun. Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search, 2021.
[11] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, pages 15180-15190, 2023.
[12] Shaojun Gui, Yu Zhu, Xiangxiang Qin, and Xiaofeng Ling. Learning multi-level domain invariant features for sketch re-identification. Neurocomputing, 2020.
[13] Weizhen He, Yiheng Deng, Shixiang Tang, Qihao Chen, Qingsong Xie, Yizhou Wang, Lei Bai, Feng Zhu, Rui Zhao, Wanli Ouyang, et al. Instruct-reid: A multi- nuuli--pupoose person re-identification task with instructions. In CVPR, pages 17521-17531, 2024.
[14] Jiahao Hong, Jialong Zuo, Chuchu Han, Ruochen Zheng, Ming Tian, Changxin Gao, and Nong Sang. Spatial cascaded clustering and weighted memory for unsupervised person re- identification. Image and Vision Computing, 156: 105478, 2025.
[15] Edward Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[16] Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. In CVPR, 2023.
[17] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurIPS, 33:18661-18673, 2020.
[18] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-Language Transformer Without Convolution Region Supervision. In JCML, 2021.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[20] Brendan Klare, Zhifeng Li, and Anil K Jain. Matching forensic sketches to mug shot photos. TPAMI, 3333)663, 2010.
[21] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yi Yu Lee. Multimodal prompting with missing modalities for visual recognition. In CVPR, 2023,
11

---

[22] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Graphics and Vision, 16(1-2):1-214, 2024. 2024.
[23] He Li, Mang Ye, Cong Wang, and Bo Du Pyramidal Transformer with Conv-Patchify for
Person Re-identification. In ACMMM, 2022.
[24] He Li, Mang Ye, Ming Zhang, and Bo Du. All in one framework for multimodal re-identification in the wild. In CVPR, pages 17459-17469, 2024.
[25] Jumnan Li, Ramprasaath Selvaraju. Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In NeurIPS, 2021.
[26] Shuang Li, Tong Xiao, Hongsheng Li. Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural language description. In CVPR, pages 1970-1979, 2017.
[27] Siyuan Li, Li Sun, and Qingli Li. CeIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels. In AAAI, 2023.
[28] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation. TPAMI, 2023.
[29] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and Trends in Multi- modal Machine Learning: Principles, Challenges, and Open Questions, 2023.
[30] Kejun Lin, Zhixiang Wang. Zheng Wang, Yinqiang Zheng, and Shin ichi Satoh. Beyond Domain Gap: Exploiting Subjectivity in Sketch- Person Retrieval. In ACMMM, 2023.
[31] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of Tricks and A Strong Baseline for Deep Person Re-identification. In CVPRW, 2019.
[32] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. SMIL: Multimodal Learning with Severely Missing Modality. AAAI, 2021.
[33] Yongsheng Pan, Mingxia Liu, Yong Xia, and Dinggang Shen. Sheease-Image-ppeee Learning for Stagnosis-Orienter Neuroimage Synthesis With Incomplete Multi-Modality Data. TPAMI, 2021.
[34] Lu Pang. Yaowei Wang. Yi-Zhe Song, Tiejun Huang, and Yonghong Tian. Cross-domand adversarial feature learning for sketch re-identification. In ACM MM, pages 609-617, 2018.
[35] Petra Poklukar, Miguel Vasco, Hang Yin, Francisco Melo, Ana Paiva, and Danica Kragic. Geometric Multimodal Contrastive Representation Learning. In ICML, 2022.
[36] Yang Qin, Yingke Chen, Dezhong Peng. Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. In CVPR, pages 27197- 27206, 2024.
[37] Liuxiang Qiu, Si Chen, Yan Yan, Jing-Hao Xue, Da-Han Wang, and Shunzhi Zhu. High-Order Structure Based Middle-Feature Learning for Eisible-Infraree Person Re-Identification, 2024.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, pages 8748-8763, 2021.
[39] Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, and Changxing Ding. Learning Granularit-Unnified Representations for Text-to-Image Person Re-identification. In ACMMM, 2022.
[40] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi-Modal Learning With Missing Modality via Shared-Specific Feature Modelling. In CVPR, 2023.
[41] Jingya Wang. Xiatian Zhu, Shaogang Gong, and Wei Li. Transferable Joint Attribute-Identity Deep Learning for Unsupervised Person Re-identification. In CVPR, 2018.
[42] Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. A comprehensive survey on cross-modal retrieval. arXiv preprint arXiv: 0060000.2211. 15, 2016.
[43] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: BERT Pretraining of Video Transformers. In CVPR, 2022.
12

---

[44] Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang. Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447-44 2023.
[45] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In CVPR, pages 79-88, 2018.
[46] Ancong Wu, Wei-Shi Zheng, Hong- Xing Yu, Shaogang Gong, and Jianhuang Lai. Rgb-infrared mrdslimy person re- re-identification. In ICCV, pages 5380-5389, 2017.
[47] Bin Yang, Jun Chen, Xianzheng Ma, and Mang Ye. Translation, Association and Augmentation:
Learning Cross-Modality Re-Identification From Single-Modality Annotation. TIP, 2023.
[48] Shuyu Yang. Yinan Zhou, Yaxiong Wang, Yujiao Wu, Li Zhu, and Zhedong Zheng. Towards Unified Text-based Person Retrieval: Large-scale Multi-Atrribute and Language Search Benchmark. In ACMMM, 2023.
[49] Mang Ye, Cuiqun Chen, Jianbing Shen, and Ling Shao. Dynamic Tri-Level Relation Mining With Attentive Graph for Visible Infrared Re-Identification. TIFS, 2022.
[50] Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng. David Crandall, and Bo Du. Transformer for object re-identification: A survey. IJCV, pages 1-31, 2024.
[51] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEEE TPAMI, 44(6)2287222893, 2021.
[52] Yajing Zhai, Yawen Zeng. Da Cao, and Shaofei Lu. Trireid: Towards multi-modal person re-identification via descriptive fusion model. In ICMR, pages 63-71, 2022.
[53] Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching.
In ECCV, pages 686-701, 2018.
[54] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-tran reta-transformer: A unified framework for multimodal learning, arXiv preprint 1xXiv:2307.1002. 2023.
[55] Yizhen Zhang, Minkyu Choi, Kuan Han, and Zhongming Liu. Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning. In NeurlPS, 2021.
[56) Yukang Zhang and Hanzi Wang. Diverse embedding expansion network and low-light cross- modality benchmark for visible-infrared person re-identification. In CVPR, pages 2153-2162, 2023.
[57] Yunhua Zhang, Hazel Doughty, and Cees Snoek. Learning Unseen Modality Interaction. In NeurlPS, 2023.
(58) Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, pages 1 1116-1 11 124, 2015.
[59] Lei Zhu, Chaoqun Zheng, Weili Guan, Jingjing Li, Yang Yang, and Heng Tao Shen. Multi-modal hashing for efficient multimedia retrieval: A survey, IEEE TKDE, 36(1):239-260, 2023.
[60] Ye Zhu, Yu Wu, Nicu Sebe, and Yan Yan. Vision+ x: survey on multimodal learning in the light of data. IEEE TPAMI, 2024.
[61] Jialong Zuo, Jiahao Hong. Feng Zhang, Changqian Yu, Hanyu Zhou, Changxin Gao, Nong Sang, and Jingdong Wang. Plip: Language-image pre-training for person representation learning. In NeurlPS, 2024.
[62] Jialong Zuo, Ying Nie, Hanyu Zhou, Huaxin Zhang, Haoyu Wang. Tianyu Guo, Nong Sang, and Changxin Gao. Cross-video identity correlating for person re-identification pre-training. In NeurIPS, 2024.
[63] Jialong Zuo, Hanyu Zhou, Ying Nie, Feng Zhang, Tianyu Guo, Nong Sang, Yunhe Wang, and Changxin Gao. Ufinebench: Towards text-based person retrieval with ultra-fine granularity. In CVPR, pages 22010-22018 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction our paper accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We conduct an in-depth discussion of the limitations of our work, which can found Sec. of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
Our work focuses on computer vision applications, not including theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully provided all the details of our proposed methods in the main text and supplementary materials, to guarantee the reproducibility of our paper's 's main experimental results. Meanwhile, we will open-source our work after our paper is accepted. These details and open-source initiatives will ensure the reproducibility of our work and make significant contribution to the entire community.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
15

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: To ensure the proper use the technology, we would not provide open access to our data and code during the paper submission process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
hggiiiiggg//////eeeeeeeeic gwwiee//ooeeuubeeeeeeeeeeeoeeeee "CodeSubmissionP oniolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips wttpccccppccpppptttctccccccctttt http//uudeseeeeeeeeee Cateee///oSSSuuiiseeeecaa ssi asiioPoolcyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due comparably expensive computational cost, the error bars are not reported in the paper like nearly all related work.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our paper provides sufficient information the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tt////eeuup.e//ee ittbll/c//thcccccideiinee
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
17

---

Justification:
Our paper discusses both potential positive societal impacts and negative societal impacts of the work in the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Our paper describes safeguards that have been put in place for responsible release of data and models in the appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We communicated with the dataset creators writing, obtained permission to use and extend their data, and strictly complied with their licenses. Additionally, we have specified the research license (CC BY-NC-SA 4.0) on OpenReview.
Guidelines:
The answer NA means that the paper does not existing assets.
18

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our work does introduce new assets by extending the existing dataset with new annotations. We can confirm that all relevant guidelines have been followed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Our external manual annotation work did involve human participation. We confirm that all annotators were fairly compensated, and our research strictly adhered to all relevant ethical guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer: [Yes]
Justification: Our external manual annotation work did involve human participation. We confirm that all annotators fairly compensated, and our research strictly adhered to all relevant ethical guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : 225 for what should or should not be described.
20

---

A Appendix
A.1
More Specific Samples
We have shown and compared some specific typical samples of our proposed ORBench dataset and two existing tri-modal datasets in Fig 7. As we can observe, due to the adoption of some rough data synthesis strategies to achieve the supplementation of additional modalities, the quality of existing tri-modal datasets is extremely low. For Tri-CUK--PEDES [5], since it is based on the existing CUHK- [26] and directly stylizes its RGB image data into a sketch style, we can notice that the sketch images in this dataset have very low resolution and are significantly different in style from actual sketch drawings. For AIO [24], it is based on the existing S SYNTH- [61] and directly uses some data augmentation methods to obtain the infrared and sketch modality. We can observe that the infrared and sketch images in this dataset are very rough and coarse- graieeedd,ddd failing to effectively cover the appearance characteristics of persons. However, for our proposed ORBench dataset, benefiting from meticulous manual annotation and repeated corrections, it possesses very high-quality five-modal data. The color pencil drawings and sketch drawings can maintain a high degree of identity consistency with the persons, and they have diverse painting angles and outstanding detail presentation capabilities. The text descriptions are extremely detailed and can be with the existing finest text-based person ReID dataset UFinc6926 [63]. Our ORBench, with its excellence in aspects such as the number of modalities. data diversity, and data quality, can make a very prominent contribution to the person ReID community and profoundly inspire subsequent researches.
par
A.2 Public Evaluation Procedure
In order to further evaluate the quality of our color pencil drawing data, we adopt the method of public evaluation to assess the identity consistency and perspective conformity. We randomly pick 2,000 samples from the color pencil drawings and split them into 20 groups Then, 20 evaluators are asked to objectively score these samples on identity consistency with the original RGB images and conformity of drawing perspectives.
Regarding its identity consistency, the scores can be divided into five levels. The higher the score, the better the consistency. The meanings represented by each score are as follows:
There are two or more features in the color pencil drawing that are inconsistent with the original RGB image, such as the color of the shoes, whether wearing hat or not, etc.
Score 1:
Score 2: There is one feature in the color pencil drawing that is inconsistent with the original RGB image.
There are no features the color pencil drawing that are inconsistent with the original RGB image, but there are relatively obvious painting errors, such as the orientation of the feet, etc.
Score 3:
Score 4: There are no features in the color pencil drawing that are inconsistent with the original RGB image, but there are few less obvious painting errors, such as the unnatural twisting of the hands, etc.
Score 5: There are features in the color pencil drawing that are obviously inconsistent with the original RGB image, and there are no painting errors at all.
Regarding its perspective conformity, the scores can be divided into five levels. The higher the score, the better the conformity. The meanings represented by each score are as follows:
Score 1: The perspective does not match. The features of a certain perspective cannot be reflected in the color pencil drawing at all. For example, the label of the color peneil drawing "back view", but the content of the drawing is actually front view.
Score 2: The perspective partially does not match. The features of a certain perspective cannot be partially reflected in the color pencil drawing. For example, the label of the color pencil drawing is "front view", but the content of the drawing is side view that shows only small part of the front features.
Score 3: The perspective basically matches. The features of certain perspective can be basically reflected in the color pencil drawing. For example, the label of the color pencil drawing "front view", but the content of the drawing is side view that basically shows the front features.
21

---

Sketch
RGB
Infrared
Sketch
Text
Ville deck
Tia ese warte
call bak akl Easas
digeo dorts
alais daee
(a) typical synthetic sample Tr-CUHK.
(b) typical synthetic sample AIO.
RGB
Infrared
Color Pencil
Sketch
Text
sit dant, je tiuk for thi growe
Naarii line shieel clostied
Mirk craal theaes nack cocpiral vnto pat ot Muad shai
young sadics
croppet
botyy dooeddeddd
sbutt dai
ras in lind
bus, aggur becte ba mein
Boni baa un reperned liwc 3t siles lokinn with theaser
tild and beglt heuatig
judbec reclia Eat be:
mourtr par your legal
(c) Some typical samples our ORBench
Figure 7: A visual overview of some typical samples in our proposed ORBench dataset and two existing mooti- datasets [5, 24]. As we can see, the quality of existing datasets is quite rough. However, our ORBench not only encompasses greater number of modalities but also demonstrates significant superiority in the data richness, quality and diversity.
Score The perspective mostly matches. The features of a certain perspective can be mostly reflected in the color pencil drawing. For example, the label of the color pencil drawing is "front view", but the content drawing is side view that mostly shows the front features (such as turning slightly sideways about 10 degrees).
Score 5: The perspective completely matches. The features of certain perspective can be completely reflected in the color pencil drawing. For example. if the label of the color pencil drawing is "front view". then the content of the drawing a completely frontal image of the person.
A.3 More Experimental Results
We have presented the retrieval results of our ReID5o in Tab for 32 different query sets on our ORBench dataset. It can be seen that the supplementation of each additional modality can effectively improve the retrieval performance. This inspires us that when actually deploying the RelD technology in real-world scenarios, we should focus on the development in the direction of multi-modal retrieval.
A.4 Controlled Release
To address the privacy concerns associated with the application of person ReID technology, we will implement& a controlled release of our dataset and code, thereby preventing privacy violations and ensuring information security. We will require that users adhere to usage guidelines and restrictions to access our dataset and code. We have drafted the following regulations that must be adhered to, which will be refined and elaborated in subsequent releases:
22

---

Table 8: The performance of our RelDSo via the queries with different modality combinations. The flst modality of each group regarded as the primary modality. The best results are bold.
Modality
CRRBench Rank-10
Bank-1
Rank-5
mAP
mINP
87.222
95. 1.9
96.905
75.091
37.599
69.833
55.389
89.944
58.719
23.935
I+C
93,909
98. 475
99 348
84.062
43.937
C+l
93.514
99.347
82,962
44.188
72.157
S41
84, 264
93. 991
96.694
72.540
33.701
1+T
79,530
91.453
94, 12.
68, 991
30.017
C48
86.875
95. 250
97.250
75, 156
37.562
S4C
86.319
72.
96, 96.986
75. 107
37.349
T:C
91.014
96.831
98.099
81.141
38.563
S+T
80,431
91.319
94.155
68, 396
30,848
IiCSS
92.127
99.090
99,693
83 D0C
44.344
C41+S
94,778
98 875
99. 597
83 941
45.096
ICT
95.031
98.
99 39.597
85 0665
45.728
C41+T
95.056
09.083
99.639
641641
46.331
I+++
91.799
57.640
98.791
80963
38.979
S41+T
91.764
97. 486
98, 9444
79.991
40.322
Ts1eS
006
98.942
81.975
37.373
90.903
98.250
40
S+C+T
90.792
97.014
98.191
007
10.167
TeCyS
93.191
98.825
83 048
40.155
1CCS
C4++S+T
96.444
99.236
99.694
85 686
47.192
Sei+C+I
96.000
99.083
99.694
85.742
47.156
Tal:C1
MM-1 Aver.
68.123
81. 9009
86.360
58, 065
24.320
MM-2 Aver.
86.154
M4.604
96.759
75.158
35.581
MM-4 Aver.
96.247
99. 205
99.704
86.347
46
Privacy: All individuals using the ORBench dataset and ReID5o model should agree to protect the privacy of all the subjects in it. The users should bear all responsibilities and consequences for any loss caused by the misuse.
2. Redistribution: The ORBench dataset and RelD5o model, either entirely or partly, should not be further distributed, published, copied, or disseminated any way or form without prior approval from the creators, no matter for profitable use or not.
The ORBench dataset, RelD50 model, in full, part, in derived formats, is not permitted for commercial use. Derivative works for commercial purposes are also prohibited. If commercial entity wants use them, separate licensing agreement with the creators must be negotiated.
3. Commercial Use:
4. Modification: The ORBench dataset, in whole or part, cannot be modified. This includes changes to data elements, visual content, textual descriptions, and metadata, as well as structural changes. Modification is restricted to maintain dataset integrity for research. In rare cases where modification needed, a request to the creators must be made, specifying the reasons and changes.
In parallel, we will require users to provide relevant information and will rigorously screen the submitted details to restrict access to our dataset and models by institutions individuals with history of privacy violations.
A.5 Broad Impact Discussion
This paper explores multi-modal person re-identification technology, which holds promise for appli- cations smart retail, intelligent transportation, and public safety systems by enabling cross-camera identification and tracking of individuals in urban environments.
However, the deployment of such technology must be approached with careful consideration of its ethical and societal implications. First, we explicitly acknowledge that the term "bias" in this context encompasses not only labeled sensitive attributes but also imbalances data distribution. Our dataset, ORBench, is sourced primarily from university settings and does not reflect globally uniform
23

---

demographic distribution. Certain demographic groups such as the elderly. individuals with visible disabilities, or specific racial groups are likely underrepresented. As result, model performance may vary across populations, and end-users should validate the fairness and generalization of the system their specific application contexts prior to deployment.
Furthermore, while we adopted a structured annotation protocol to maximize objectivity and consis- lency-focusing on descriptive visual attributes such clothing and accessories- wee recognize that the design of such guidelines inherently introduces form of methodological bias. By prescribing which features are salient, the annotation process shapes the model's 's attention and may omit other potentially relevant characteristics.
The potential for misuse of this technology also warrants serious attention. Although our research is intended for academic and public safety purposes, the capability to retrieve individuals via textual descriptions could be repurposed target people based attributes associated with protected categories-such as disability, religious attire, or age. Like many dual-use technologies, person re-identification systems require strong governance, ethical oversight, and clear usage policies to prevent discriminatory or privacy-infringnng applications. We oppose any such misuse.
In light of these considerations, emphasize the importance of legal and regulatory frameworks to govern the application of ReID systems. Training often relies on surveillance data collected without explicit individual consent, raising privacy concerns. The research community should avoid using ethically problematic datasets and adopt responsible data practices. To mitigate potential harm, we will release BRnehch under gated access, with usage restricted non-commercial research purposes and subject to striet guidelines.
By addressing these issues transparently, we hope to encourage the responsible development and deployment of multi-modal person re-identification systems.
24


Reviews:
[{"review_id": "G3CEljmuSC", "reviewer": "Reviewer_h3Ed", "summary": "The paper introduces a new dataset for multi-modal person re-identification, ORBench, and a novel multi-modal reidentification framework, ReID5o.\n\nORBench is constructed by combining images from two existing public person re-ID datasets,\nSYSU-MM01 [46] and LLCM [57], and extending these with additional multi-modal annotations:\npencil sketches, color sketches and text descriptions. The pencil and color sketches are created from the original images using (commercial) image-to-image AI software tools. The text descriptions have been manually created by paid workers. Additional manual curation has been applied to remove unreliable samples, resulting in 1000 people, 45k RGB, 45k text, 26k IR and 18k color/grayscale sketches. Compared to other Re-ID datasets, ORBench is unique in its inclusion of all of these modalities together, and its large amount of color and grayscale images. \n\nReID5o is a multi-modal transformer trained to retrieve a person from a reference set using multi-modal observations/descriptions of that person. The multi-modal inputs are first tokenized by modality-specific encoders, and then processed by a transformer. The transformer uses low-rank (LoRA) decomposition for feature extraction, where the LoRA transformation is specific to the input modality associated with each token. Finally, a Feature Mixing stage fuses information from all procided input modalities to produce a single descriptor.\nThe paper presents experiments to show ReID5o outperforms various baseline methods on ORBench. It also shows strong dataset generalization of ORBench-trained models compared to models trained on other datasets, and includes ablation studies on ReID5o components, and a study on the strength of the different modalities.", "questions": "See my discussed Weaknesses for more detail:\n\n- Technical: how are the features exactly fused in the Feature Mixture module?\n- What are the use cases / motivation for adding the selected modalities?\n- How laborious was creating this dataset, and what design choices were considered?\n- To what extent have you actively checked if the public datasets, on which ORBench is built, comply with the NeurIPS ethical guidelines?", "limitations": "Some limitations are discussed in the Appendix, which is good.\nStill, I list sev3eral ethical concerns in the Weaknesses above, which also touch on societal issues.\nI would encourage the authors to discuss bias & fairness of the data, and if this research could lead to discrimination e.g. by searching for people with particular race, sex, age, etc.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The proposed ORBench dataset combines more diverse modalities (rgb, ir, text, grayscale & color pencil) compared to other ReID datasets, allowing to test new multi-modal settings.\n-  A new multi-modal architecture is proposed, ReID5o, which combines existing insights in a sensible manner. A single ReID5o model can perform retrieval on a single modality, a few modalities, or all modalities. It outperforms multiple other multi-modal retieval methods on ORBench.\n- An extensive ablation study is included to validate the components of ReID5o, and also inspect the impact of the different modalities in ORBench.\n- Additional experiments show improved generalization of the ORBench-trained model compared to the model trained on other datasets; The paper also includes qualitative results to support the findings. Overall, the evaluation on the new dataset is thorough.\n- In general the paper is clear and easy to follow.\n\n# Weaknesses\n\n## Methodological details are unclear\n\n- the proposed Feature Mixture module\nline 218: \"For each multimodal features Z^mod, we perform a sequential combinatorial traversal.\"; I don't understand this line. As I understand it Z^mod here is the feature vector from a single modality (since mod \\in {R,I,C,S,T}). I don't understand how a single feature can be processed combinatorically.\n\n- As I understand it the network is used to generate feature vectors from the input (multi-modal) inputs; I assume the setup is as follows  as in a standard image retrieval task: the network turns a query into a descriptor, and compares it to descriptors of an available reference dataset. I did not find this explicitly stated in Method section 3. Instead, the section refers to \"identity classification\" which made me think the network was trained to classify a fixed set of person identities known at train time (which would not generalize to new identities). Only at the experimental setting was it made explicit that training identities and test identities are different, hence I believe this is not a classification task but a metric learning task. \nIf it is indeed an retrieval task, then the used retrieval procedure should be stated explicitly (e.g. Euclidean or cosine comparison, any top-k results voting, etc.). Also, the size of the resulting feature size used for retrieval ($D$ from Section 3.4, I believe) should be stated for reproducibility.\n\n## Unclear how laborious creating the dataset was\n\nPapers that introduce new datasets often represent a significant engineering and data curation effort, and sharing such data represents a valuable service to the research community. In return, as reviewer we should expect less methodological novelty for such papers.\n\nAt first glance, this paper appears to introduce a new large dataset representing significant labour that allows the community to better study some real-world phenomena. But upon closer inspection, the new dataset mostly combines and extends existing data sources. The novel annotations that have been added are generated by \"off the shelf\" AI models.\n\nThe most laborious efforts seems to be the text descriptions. It is stated that 45k text descriptions were created from each RGB image by hired workers (line 119), but Figure 7 shows pretty similar description formats. Did the annotators use a tool to help them with this, and did this not introduce other biases and repeated phrasing? Might the workers not produce lower quality descriptions for the last 5k compared to the first 5k descriptions? More details on the annotation setup should be provided. I also wonder why AI tools were used for the drawing modalities, but why no LLMs were used to generate these text descriptions. \n\nIt might sound pedantic, but understanding the effort to build such a dataset is important to judge the significance and value of the dataset towards the community, and also how critical to review the dataset's realism and design choices.\n\nThis brings me to my next concern:\n\n## How realistic is the dataset, and have meaningful choices been made?\n\nI am missing this higher-level motivation for adding new modalities as has been done in this work:\n- What real-world setting(s) are exactly being simulated?\n- Were these AI-generated annotated added simply because they appear to look good, or do these type of sketches reflect the intended real-world use cases?\n- In the intended use cases, can we expect all modalities to have the same level of detail and completeness? What kind of variance within and across modalities can we expect?\n\nCurrently, it is unclear what practical use cases the dataset exactly targets, and if the new annotations represent the these use cases well, also in terms of data variance. My overall impression is that main motivation for the design choices is pragmatic: The experiments show the new dataset improves generalization to other ReID datasets.\n\nThere is some discussion on the realism in the appendix (A.7 \"Limitation\"), but I believe the paper should be explicit in its goals, assumptions, and simplifications when introducing the data in the paper itself.\n\n## Ethical concerns\n\nFirst off, let me acknowledge that the authors address several concerns in the appendices:\n- Appendix A.5 \"Broader Impact Discussion\" discusses risk of such surveillance technology, and encourages researchers using their data to follow the law; It is also explained the data is only released for research purposes, not commercial use.\n- Appendix A.6 \"Controlled Release\" indicates that the authors plan to release the data in a controlled manner, requiring users to \"adhere guidelines and restrictions\" to access the data and code.\n\nNevertheless, I have several concerns that might require closer inspection by an ethics expert:\n\n1. **Privacy of people in the benchmark.**\nThe paper relies on already public benchmarks, and does not make claim about the privacy of the individuals by itsself. However, by extending and updating the benchmark, and publishing it at a highly visible venue as NeurIPS, I believe the paper should make explicit how it established that the used data does follow all guidelines. This may require explicitly stating how the used public benchmarks protected privacy, and how those policies are consistent with the NeurIPS Ethics Guidelines. If those public datasets did _not_ make clear how such privacy concerns were addressed, then extending those datasets might not be suited for a NeurIPS publication. I find the current argument in de appendices A.5 & A.6 too weak: all responsibility to avoid privacy violations is delegated to the end users of the dataset and models complying with law. But the paper does not make any attempt itself to check for such violations.\n\n2. **Bias and discrimination** This research is clearly about identifying human subjects, I did not find any discussion on potential bias in the dataset, and risks for discrimination, risk of predicting protected categories, etc. These areas (bias & fairness, discrimination, predicting protected categories in surveillance) are explicitly mentioned in the ethics guidelines, but not discussed. I expect a minimum would be to indicate what kind of population the data represents, what type of variations in people it contains. This too might require checking the information from the original public datasets, and here too I wonder if those datasets should be extended in a NeurIPS paper if their compliance with its guidelines cannot be verified. Furthermore, I am not sure if the human text annotations would enable to look for \"protected categories\" as the ethics guideline states. Could this research enable surveillance technology that retrieves individuals with a particular ethnicity, or some age, or a disability using a simple text query?\n\n3. **Answers in questions in the NeurIPs checklist do not align with what can be read in the paper submission**\n  - Question 12 \"Licenses of existing assets\"  -> authors answered \"Yes\",\nbut there is no mention of the license of the datasets on which their proposed ORBench builds. There is also no (intended) license mentioned for ORBench.\n  - Question 13 \"New assets\" -> the authors answered \"[N/A]\".\nThe paper _does_ introduce new assets by extending the dataset with new annotations. None of the guidelines to Question 13 have been considered: No details on the license of the dataset, no discussion on how consent was obtained from people in the dataset, no anonymization.\n  - Questions 14 & 15, the authors reply \"Our paper does not involve crowdsourcing nor research with human subjects\", \nBut the paper is about having descriptions of human subjects; Also, the paper stated that external manual labour has been used to generate textual descriptions (line 119) and to perform a public evaluation of the dataset quality (line 348)\n\nI am not saying these necessarily _are_ ethics violations, but my impression is that the answers do not acknowledge what the checklist and NeurIPs guidelines are for. I believe it would be good if an ethics expert reviews the submission.", "was_revised": true, "final_justification": "In the discussion in the rebuttal phase have answered some of my questions, but also confirmed some of my concerns, so I stick to the lower ratings on significance and originality: there are some limitations and key discussions missing in the original submission. The methodology is also not particularly innovative, but it does appear effective.\n\nI can still see arguments for a reject: (1) a lot of necessary updates to the text have been identified and promised, but this could warrant a new review round to see if they are properly implemented; (2) the involved expert ethical reviewers might still find the rebuttal insufficient, in which case I believe the paper should also be rejected and resubmitted.\n\nNevertheless, I have updated my rating one step now, as I expect the necessary improvements will address my main concerns sufficiently.", "rebuttal": "Thanks for your valuable comments. Now we respond to your concerns as follows.\n\n---\n## **Methodological details are unclear**\n> `Q1`: *\"How are the features exactly fused?\"*\n\n**A:** Thanks for identifying an ambiguity.\n\nTo clarify, the combinatorial traversal is performed not on individual modal features $Z^{mod}$ but on the entire set of multimodal query features $\\{Z^I, Z^C, Z^S, Z^T\\}$. Specifically, we generate all possible combinations from single-modal to quad-modal (i.e., $C_4^1=4$, $C_4^2=6$, $C_4^3=4$, and $C_4^4=1$ combinations). These are then concatenated and processed by the Feature Mixture module.\n\nThe example in Lines 221-225 (IR+Text fusion) illustrates this process for one dual-modal combination. We will revise the text to resolve this confusion.\n\n> `Q2`: *\"Detailed explanations on task setup, retrieval procedure and feature size.\"*\n\n**A:** The task is essentially a **fine-grained multimodal image retrieval task** that generalizes to unseen identities (not included in the training set). This is a **metric learning** task rather than an identity classification task.\n\nIn training, we follow common practice by using training-set identity labels to guide the representation learning, with two losses:\n\n* **The primary loss is the similarity distribution matching (SDM) loss**, a typical metric learning loss that aligns cross-modal features by minimizing the KL divergence between cross-modal similarity distributions and normalized label-matching distributions.\n\n* A **supplementary identity classification loss** serves as an auxiliary objective to enhance feature discriminability, using identity labels for multi-class classification of each multimodal sample.\n\nIn inference, the classification layer is entirely discarded. Retrieval is performed by computing **cosine similarity** between the query feature and all gallery features.\n\nFor the feature size, we use CLIP-B/16 as the encoder (consistent with prior methods) with its default size of **512**.\n\nAs suggested, these details will be explicitly stated in the revised version.\n\n---\n## **Unclear how laborious creating the dataset was**\n**It took months of intensive effort to ensure quality and diversity of ORBench**—no mere crude combination/extension of existing resources. Key workflow details:\n\n* **Manual filtering of existing infrared-RGB datasets.** Some datasets (notably LLCM) contain noisy samples with extremely poor imaging quality, where facial features are barely distinguishable to the naked eye—unsuitable for accurate annotation and lacking sufficient learnable features. Initial attempts at automatic filtering via resolution thresholds failed, as resolution does not strictly correlate with facial clarity. Thus, we adopted manual screening: for each identity, all images were reviewed individually, and low-quality samples were removed.\n\n* **Manual fine-grained text annotation for RGB images.** We developed strict annotation guidelines specifying requirements for objective, unbiased and explicit appearance descriptions, followed by training annotators (including trial sessions) to ensure compliance. The task was completed by 26 annotators over approximately one month. Our team then rigorously reviewed all annotations, with inappropriate descriptions returned for revision.\n\n* **AI-assisted generation of paintings and sketches with human oversight.** AI assistance was chosen for two reasons: manual drawing demands high expertise, and large-scale manual creation is time-prohibitive. After testing multiple tools, we selected Doubao's artistic painting tool. The workflow involved: manually selecting the clearest RGB images per viewpoint; adding detailed manual text descriptions; feeding these to the AI to generate color paintings; and having human reviewers inspect outputs (with substandard ones recreated). Sketches were derived via style conversion from these paintings. This hybrid approach ensured efficient production of high-quality data.\n\n> `Q3`: *\"Did annotators use tools, and did this introduce biases or repeated phrasing?\"*\n\n**A:** Annotators strictly followed guidelines for manual annotation without tools. Specialized reviewers mitigated potential biases. While some repetition is inevitable, the 26-annotator team enhanced linguistic diversity, reducing redundancy.\n\n> `Q4`: *\"Might later descriptions be lower quality than earlier ones?\"*\n\n**A:** Specialized reviewers manually evaluated all annotations, with substandard entries returned for revision—effectively preventing quality degradation.\n\n> `Q5`: *\"Why use AI for drawings but not LLMs for text?\"*\n\n**A:** Text descriptions require detailed references to RGB images, which pure LLMs cannot process. MLLMs, while capable of image description, struggle with low-resolution person images and fine-grained details, failing to generate satisfactory results. Thus, we opted for manual annotation for text. For drawings, AI (with human oversight) efficiently produced high-quality outputs, making it the practical choice.\n\n---\n## **How realistic is the dataset, and have meaningful choices been made?**\n> `Q6`: *\"What real-world setting(s) are exactly being simulated?\"*\n\n**A:** Our five-modal dataset and task setup align with practical needs, explicitly simulating real-world scenarios. A key example is public safety: when searching for a target, investigators often use multi-modal information—such as low-resolution surveillance footage, witness descriptions, and sketches/portraits derived from these descriptions. Effectively leveraging their complementarity to characterize targets for accurate identification remains a highly practical research topic. Additionally, real-world scenarios naturally involve variable numbers of modalities (from single to multiple). Our work addresses retrieval under arbitrary modal combinations, enhancing practical relevance.\n\n> `Q7`: *\"Were these AI-generated annotations added just to look good, or do such sketches reflect intended real-world uses?\"*\n\n**A:** These modalities were included based on practical value rather than superficial visual appeal. In real scenarios, after witnesses provide textual descriptions, more precise drawings or sketches are often created to improve search accuracy, with subsequent retrieval combining all available modalities. Our dataset faithfully reproduces such practical cases.\n\n> `Q8`: *\"In intended use cases, will all modalities have the same detail and completeness? What variance can we expect within and across them?\"*\n\n**A:** In intended use cases, variations in detail and completeness across modalities are inherent—this is a core challenge in real-world multimodal ReID, which our dataset deliberately incorporates.\n\nRegarding intra-modal variance: RGB/infrared data exhibit fluctuations in color, contour, and clarity due to lighting and scene conditions; Sketches/paintings show variability from AI generation randomness and multi-perspective design; \nTexts vary in descriptive content due to stylistic differences among 26 annotators.\n\nInter-modal differences appear in information type and granularity (Figure 7): visual modalities focus on spatial features (RGB with color, infrared with temperature, paintings more abstract), while texts are semantic-symbolic with lower detail granularity. \n\nThese differences reflect real-world realities and highlight the challenges our dataset presents to practical algorithms.\n\n---\n## **Ethical concerns**\n> `Q9`: *\"Privacy of people in the benchmark\"*\n\n**A:** We clarify that both the original benchmarks and our extended data strictly adhere to the guidlines with explicit privacy safeguards.\n\nFor SYSU-MM01, we verified its creators obtained written privacy licenses from all captured pedestrians, explicitly permitting use of their images for scientific research and academic publication.\n\nFor LLCM, its creators used MTCNN for facial blurring to anonymize personal identifiers. We confirmed this meets standard anonymization criteria, aligning with NeurIPS'  privacy risk minimization requirements.\n\nBoth datasets have usage agreements with privacy protection provisions; access requires strict adherence and formal signing. Additionally, we obtained written permission from both datasets' creators to modify and extend them.\n\n> `Q10`: *\"Bias and discrimination\"* \n\n**A:** Our data comes from public datasets complying with local laws, with measures ensuring fairness and avoiding bias or discrimination. Sourced from Sun Yat-sen and Xiamen Universities, it covers their campus communities, with balanced age and gender distribution. The datasets contain no biases or specific protected groups and comply with NeurIPS guidelines. Our human text annotations, under strict review, are objective and unbiased, unable to identify protected categories. Moreover, due to rigorous data review and compliance, our work will not retrieve individuals by ethnicity, age, or disability.\n\n> `Q11`: *\"Answers in questions in the NeurIPs checklist do not align with what can be read in the paper submission\"*\n\n**A:** For Question 12: We communicated in writing with dataset creators, obtained permission for data use and extension, and strictly complied with their licenses. Additionally, we've specified the research license (CC BY-NC-SA 4.0) on OpenReview. We will also follow your suggestion to explicitly note the licenses in the supplement.\n\nFor Question 13: Since we did not directly submit the asset (e.g., via an anonymized URL) for this submission, we marked [N/A]. For further details, please see earlier statements.\n\nFor Question 14&15: We initially understood these two questions to refer to large-scale, direct human experiments (e.g., distributing mass questionnaires). As our research is unrelated, we marked [N/A]. We confirm strict adherence to relevant guidelines and that all workers were paid adequately. For further details, please see earlier statements.\n\n---\nWe would like to sincerely thank Reviewer `h3Ed` again for the valuable time and feedback."}, {"review_id": "1oYer4DsiM", "reviewer": "Reviewer_AoA7", "summary": "This work addresses the largely overlooked problem of Omni Multi-modal Person Re-identification (OM-ReID), which involves retrieving a person using any combination of five modalities: RGB, infrared, sketch, color pencil, and text. To support this task, the authors present ORBench, a high-quality dataset featuring all five modalities across 1,000 identities. Additionally, they propose ReID5o, a unified framework that employs a multi-modal tokenizing assembler and a multi-expert routing mechanism to facilitate flexible modality fusion and alignment.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1 The paper presents ORBench, a novel five-modality dataset characterized by its high diversity and quality, providing a valuable foundation for advancing research in more practical and versatile multi-modal ReID settings.\n\n2 ReID5o employs a simple yet effective feature mixture strategy with multi-head self-attention and transformer blocks to combine inputs from arbitrary modality combinations. This design facilitates interactive information exchange, allowing the model to harness complementary cues without complex architectural modifications.\n\nWeaknesses:\n\n1 Most modalities in the dataset introduced by this paper are artificially generated rather than captured from real-world sources. Specifically, the Sketch and Color Pencil images are derived from the original RGB images through repeated transformations. This raises questions about the model’s ability to train effectively and generalize well, as the dataset may not fully capture the complexity of real-world multimodal perception.\n\n2 Although the authors introduce a new dataset, ORBench, most of the baseline methods are re-implemented and fine-tuned on this dataset by the authors themselves. However, the paper lacks cross-dataset transfer evaluations on established benchmarks, making it unclear whether ReID5o's superior performance stems from genuine model effectiveness or potential biases inherent in the dataset design.\n\n3 ReID5o incorporates several modality-specific tokenizers, a shared transformer encoder, and a multi-expert routing mechanism, resulting in increased model complexity. However, the paper lacks a thorough discussion of the trade-offs between efficiency and performance during training and inference, as well as the model's scalability under constrained hardware resources—factors that are critical for real-world deployment.\n\n4 To construct multimodal query combinations, the authors randomly sample data from different modalities but of the same identity. This approach can result in overly consistent intra-identity pairs, potentially reducing the realism and difficulty of the evaluation. In particular, when the sketch, color pencil, and text are all derived from the same RGB image, the model may exploit superficial shortcuts instead of truly demonstrating cross-modal generalization capabilities.", "was_revised": true, "final_justification": "Thank you for the author's response. The author has addressed some of my concerns. However, after carefully reviewing the comments of other reviewers, I agree with reviewer h3Ed regarding the dataset. I also remain skeptical about whether constructing a dataset based on artificially generated modalities, rather than raw image modalities (visible/infrared), can fully capture the complexity of real-world multimodal perception.\nTherefore, I will maintain my original score.", "rebuttal": "Thanks for your valuable comments of our work. \n\nFirst, we would like to explain the research motivation and contributions of our work.\n\nIn real-world practical applications, when searching for a target person, the actual witness data used as queries is usually multi-source and multi-modal. Data from different modalities typically focus on different aspects of the target person's features. In practice, **the number of modalities** in such witness data is **often uncertain**, which raises a thought-provoking question:\n\n*For queries with an uncertain number of modalities, how can we facilitate feature complementarity between any different modal data to achieve an accurate portrayal of the target person and thereby improve the accuracy of the search?*\n\nTo promote in-depth research on this issue, our work makes the following contributions:\n\n1. We propose **a new task** with practical value and challenges, namely omni multi-modal person re-identification (ReID). This task requires achieving effective retrieval with varying multi-modal queries and their combinations in the ReID model, which remains largely unexplored in previous research.\n\n2. We construct **the first high-quality five-modal person ReID dataset** named ORBench, where each person is accurately and comprehensively characterized by data from five modalities. Considering that existing datasets usually contain only a small number of modalities and are of poor quality, we have supplemented three additional modalities of data—namely painted, sketch, and text modalities—based on existing RGB-infrared datasets through extensive manual annotation and intervention. This dataset can serve as a valuable research platform to advance the development of the field.\n\n3. We propose **a simple and efficient multi-modal person ReID framework** for retrieval with any combination of modalities named ReID5o, serving as a baseline method for the task and dataset. Through a multi-expert routing and multi-modal feature fusion mechanism, this method can effectively achieve feature information complementarity between any modalities, thus obtaining more accurate retrieval results.\n\nWe believe that the task, dataset, and method, as an integrated whole, complement each other. They simulate real-world practical application scenarios and can provide a foundation and inspiration for subsequent research on multi-modal person ReID.\n\n---\nNow, we would like to address your specific concerns through the following detailed responses.\n\n> `Q1`: *\"Most modalities in the dataset introduced by this paper are artificially generated rather than captured from real-world sources.\"*\n\n**A:** We acknowledge that some modalities in our dataset are artificially generated, but **we have taken rigorous measures to ensure all modalities closely align with real-world scenarios and capture their inherent complexity.**\n\n* For RGB and infrared modalities: The data in ORBench are filtered from two existing public datasets, LLCM and SYSU-MM01. These datasets contain RGB and infrared images directly collected from real-world scenes, with diverse distributions in illumination, weather, and scenarios, ensuring the pedestrian appearance features are real and representative of real-world variations.\n\n* For the text modality: The descriptions are manually annotated by 26 human annotators following strict guidelines, ensuring rich language styles, accurate reflection of real pedestrian attributes, and consistency with real-world semantic expression.\n\n* For sketch and color pencil modalities: These data are AI-generated with heavy human intervention, not via repeated RGB transformations. Instead, they mimic real creation: AI produces multi-view sketches and drawings by referencing real RGB images (appearance) and detailed text (semantics). Humans refine outputs to match real sketch/pencil traits (e.g., preserving key features, simplifying non-essentials, matching textures) rather than replicating RGB content. This labor-intensive process captures real-world sketch/pencil perception complexity.\n\nOur experiments confirm the dataset's effectiveness and generalization. As shown in **Figure `6`**, the model trained on our dataset leverages multimodal complementarity to achieve accurate retrieval, demonstrating adaptability to real-world multimodal variations. Additionally, **Table `7`**'s dataset generalization experiments show our dataset outperforms others, indicating our dataset helps to learn robust representations transferable to real-world complexity.\n\n---\n> `Q2`: *\"Most of the baseline methods are re-implemented and fine-tuned on this dataset by the authors themselves. The paper lacks cross-dataset transfer evaluations on established benchmarks.\"*\n\n**A:** Regarding the re-implementation and fine-tuning of baseline methods, it is worth noting that this study focuses on the novel task of person ReID across arbitrary modalities and their combinations, with ORBench being the first five-modality dataset for it. Since both are new, existing baselines lack training/evaluation under such multimodal settings and have no reusable weights. Therefore, to verify the effectiveness of ReID5o within a unified experimental framework, we re-implement existing methods and perform fine-tuning on ORBench. **This is a necessary step in verifying the effectiveness of new methods in pioneering work in this field**, ensuring fair and relevant comparisons.\n\n***Regarding cross-dataset transfer evaluation, we have actually conducted relevant experiments specifically in the paper (see Table `7` for details)***. We transferred the ReID5o model and baseline methods trained on ORBench to multiple existing cross-modal benchmarks for evaluation. The results clearly show that ReID5o still maintains significant advantages in cross-dataset scenarios, and the models trained on the ORBench dataset significantly outperform those trained on other datasets in terms of transfer performance. This result not only demonstrates the generalization ability of ReID5o but also verifies the rationality of the ORBench dataset, ruling out the impact of dataset design biases on performance.\n\n---\n> `Q3`: *\"The paper lacks a thorough discussion of the trade-offs between efficiency and performance during training and inference, as well as the model's scalability under constrained hardware resources.\"*\n\n**A**: ***Regarding the trade-off between model efficiency and performance, we have actually conducted a systematic consideration in our research and verified it through experiments.*** The specific explanations are as follows:\n\nIn fact, in terms of the trade-off between efficiency and performance, we specifically compared the impact of different multi-expert routing designs on model performance and computational overhead in **Table `3`.** \n\nThe experimental results show that the selected trade-off parameters not only achieve the best performance improvement but also only bring a small increase in params and FLOPs (please refer to Table 3 for specific values and experimental explanations). This result directly reflects the importance we attach to the balance between efficiency and performance in the design.\n\nFrom the perspective of model architecture design, the efficiency of ReID5o is mainly reflected in three aspects:\n\n* The shared transformer encoder significantly reduces redundant computations in multi-modal scenarios and avoids parameter explosion caused by designing separate encoders for different modalities;\n\n* The lightweight tokenizer controls additional overhead while ensuring modal adaptability by simplifying the modal feature mapping process;\n\n* The LoRA-based multi-expert routing mechanism realizes parameter-efficient fine-tuning through low-rank matrix factorization, which significantly reduces the computational complexity of the routing module compared with traditional multi-expert models.\n\nThe above designs enable the model to maintain high performance while having good hardware adaptability, and it can still run in environments with limited computing resources.\n\n---\n> `Q4`: *\"To construct multimodal query combinations, the authors randomly sample data from different modalities but of the same identity, resulting in overly consistent intra-identity pairs. The model may exploit superficial shortcuts instead of truly demonstrating cross-modal generalization capabilities.\"*\n\n**A:** First, our task focuses on omni multi-modal person ReID, whose core goal is accurate retrieval via any modality combinations. **This task inherently requires multi-modal query combinations to maintain identity consistency**—a necessity, not an artificial constraint. The aim is to enable the model to leverage complementary information across modalities, constructing more comprehensive pedestrian representations for real-world cross-modal search. This aligns with practical scenarios where users query based on diverse modal data of the same target.\n\nSecond, **our experimental design rigorously verifies cross-modal generalization**. We separated the dataset into 600 training and 400 test identities, ensuring independent division to evaluate performance on unseen data. Additionally, cross-dataset experiments in **Table `7`** explicitly demonstrate the model learns essential inter-modal associations (not dataset-specific superficial features), confirming strong cross-scenario generalization.\n\nFinally, significant modal differences in our dataset prevent over-reliance on superficial shortcuts. Sketch and color painting modalities include multi-perspective samples with **visual characteristics distinct** from RGB images; text descriptions use **varied styles to ensure richness and diversity**. These designs force the model to learn intrinsic inter-modal connections, instead of relying on superficial similar features, thus effectively improving the model's cross-modal generalization ability.\n\n---\nWe would like to sincerely thank Reviewer `AoA7` again for the valuable time and feedback provided."}, {"review_id": "sK2CGIAfDW", "reviewer": "Reviewer_FsmM", "summary": "For the real-world challenge of arbitrary single modalities or modality combinations queries in Person ReID, which existing methods hard to deal with, this paper proposes a diverse and practical benchmark: ORBench, with text, RGB, infrared, colored pencil drawing, and sketch modality. In addition, the authors also propose a new method ReID5o for this task, which contains a unified encoder and an expert routing mechanism and achieves flexible queries with arbitrary modality combinations. Extensive experiments demonstrate the advancement and practicality of the proposed benchmarks and indicate the SOTA performance of ReID5o against previous works.", "questions": "* Please discuss whether there is a conflict between learning the complementary information of modalities for fusion queries and fully mining the effective information for single modality queries, and if there is, how to conduct the trade off?\n\n* Please report the the space complexity, and the time cost of training stage and reference stage in different baselines. \n\n* When applying this foundation model on ReID tasks, would more parameters achieve better performance?\n\n*  The authors removed unclear samples from constructed dataset, but didn't literally or empirically analysis the robustness of the proposed model against possible noise in the real world.", "limitations": "**1.** The possible approaches of enhancing the robustness of the model to real-world noise should be discussed in detail in A.7.\n\n**2.** The computing cost is relatively large, which may limit its faster and wider application.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The experiments are sufficient to demonstrate the advancement of the proposed benchmark and method.\n2. The proposed benchmark has great practical and academic significance, which is able to provide a high-quality multi-modal data platform for the community.\n3. The presentation is easy to follow.\n\n**Weaknesses:**\n\nSee questions and limitations.", "was_revised": true, "final_justification": "This work contains a novel and practical task setting, a high-quality dataset that could effectively generalize on practical datasets with noise and a simple but effective method better than other counterparts. I think it provided a valuable data resource for the whole ReID research community thought it may not perfectly reproduce the readlity cases. After carefully reading comments from reviewers and corresponding rebuttals, I acknowledge the authors' efforts to revise the manuscripts and agree that this work could serve as a great start for this new area, so I will raise my score to show my support for it.", "rebuttal": "We sincerely thank Reviewer `FsmM` for the positive and constructive review. We are grateful for your recognition of our sufficient experiments, proposed benchmark, and the presentation. Below, we respond to your thoughtful questions and suggestions in detail.\n\n----\n> **`Q1`:** *\"Please discuss whether there is a conflict between learning the complementary information of modalities for fusion queries and fully mining the effective information for single modality queries, and if there is, how to conduct the trade off?\"*\n\n**A:** Thank you for your insightful comments.\n\nIn fact, **there is a potential conflict between them**: On one hand, to achieve complementarity in multimodal fusion (e.g., semantic-visual complementarity between texts and sketches), the model needs to learn cross-modal aligned representations, which may lead to the \"dilution\" of modality-specific information (e.g., unique color textures in paintings or night features in infrared images). On the other hand, over-emphasizing effective information mining for single modalities may widen feature space differences across modalities, hindering alignment during fusion and weakening complementarity.\n\nWhen designing our method, **we took this conflict into account** and achieved a trade-off through the below strategies:\n\n* Multi-modal Tokenizing Assembler. To encode data from different modalities into a shared embedding space, we designed a straightforward multi-modal tokenizing assembler. It consists of five lightweight sub-tokenizers, which are used to independently tokenize RGB, IR, CP, SK, and text data respectively. This operation can, to a certain extent, preserve the differences and uniqueness of the specific information of each modality.\n\n* Multi-Expert Router. We utilized a shared encoder to extract modality-shared features. Meanwhile, to extract modality-specific features, we designed this multi-expert routing mechanism. This mechanism allocates a respective LoRA-based expert to each modality, promoting the mining of independent and effective information for each modality and ensuring the discriminability of features from different modalities.\n\n* Feature Mixture. To achieve the fusion and interaction of complementary information between different modalities, we utilized this efficient feature mixture to fuse features from different modal combinations. The input of this mixture is the features of each modality with independent information, and the output is the multimodal fused feature.\n\nIn our training: for single-modal retrieval, we directly use independent features of each modality for alignment; for multimodal combined retrieval, we use fused features after mixture. This approach balances learning multimodal complementary information and mining effective single-modal information to some extent, as verified by ablation experiments in Table `2`.\n\nAdditionally, we propose that **future research in this area could focus on a \"query type-aware mechanism\".** This would enable the model to dynamically adjust feature extraction focus based on whether the input is a single-modal or multimodal query, balancing the two more precisely. We will pursue this direction in subsequent work.\n\nWe will follow your suggestions and add this part of the discussion to the supplement in the future version.\n\n---\n> **`Q2`:** *\"Please report the the space complexity, and the time cost of training stage and inference stage in different baselines.\"*\n\n**A:** Thank you for your suggestion. We report the space complexity (parameters), training time cost, and single inference forward time of each baseline in the table below.\n\n| Method |Params (M) |Training Time (h) |Average Forward Time (ms) |\n| --- | --- | --- | --- | \n| CLIP | 149.6 | 4.79 | 50.37 |\n| PLIP | 147.3 | 4.21 | 45.83 |\n|IRRA| 189.2 | 7.02 | 50.42 |\n|RDE| 152.8 | 7.78 | 50.46 |\n|Meta-TF| 127.2 | 3.86 | 47.24 |\n|ImageBind| 1200.8 | 18.64 | 152.2 |\n|UNIReID| 149.6 | 5.42 | 50.45 |\n|AIO| 166.8 | 4.92 |47.38 |\n|**ReID5o**| **159.2** | **5.99** | **76.13** |\n\nThe listed parameters include all module parameters of each baseline model during training, some of which may be discarded during inference. Training time is based on a unified setting of 60 epochs. \n\nFor inference time, to ensure efficient and fair comparison, a single forward inference is defined as: the model sequentially infers one sample of RGB, infrared, painted, sketch, and text, then fuses the features of the latter four into a multimodal feature (consistent with the quad-search process in the benchmark). This process was repeated 1000 times, with the average taken as the result.\n\nAs shown in the table, **ReID5o** not only maintains significantly leading retrieval performance but **also has advantages in terms of parameter count, training time, and inference speed**.\n\nFollowing your suggestions, we will include comparisons of results in this regard in the supplement of future versions.\n\n---\n> **`Q3`:** *\"When applying this foundation model on ReID tasks, would more parameters achieve better performance?\"*\n\n**A:** When increasing the parameters of our model (especially the pre-trained multi-modal encoder within it), it can **indeed achieve** better performance on the ReID task. To verify this, we replaced the original CLIP-B/16 encoder in ReID5o with CLIP-L/14, conducted training under the same settings, and tested its performance on ORBench. The results are shown in the following table.\n\n| Model |Params (M) |MM-1 |MM-2 |MM-3 |MM-4 |\n| --- | --- | --- | --- |--- | --- |\n|ReID5o (CLIP-B/16)| 159.2 |58.09| 75.26 |82.83| 86.35|\n|ReID5o (CLIP-L/14)| 443.8 | 62.14 | 78.32 |85.03 |87.64|\n\nConsidering that the ReID task is essentially a fine-grained representation learning task, we believe that the performance improvement brought by the increase in the number of parameters is mainly attributed to the **stronger and more robust representation ability** of larger-scale pre-trained multimodal encoders. This phenomenon has also been demonstrated in many other papers.\n\n---\n> **`Q4`:** *\"The authors removed unclear samples from constructed dataset, but didn't literally or empirically analysis the robustness of the proposed model against possible noise in the real world. The possible approaches of enhancing the robustness of the model to real-world noise should be discussed in detail in A.7.\"*\n\n**A:** Thank you for your insightful comments.\n\nFirst, please allow us to elaborate on two reasons why we removed unclear samples from the existing datasets:\n\n* Clear, high-quality RGB samples **aid in annotating** accurate, feature-rich data across color paintings, sketches, and text. Since we rely on original RGB images to draw and describe individuals, unclear ones hinder the creation of high-quality, appearance-consistent data—making it hard to fully and accurately capture people's appearance features.\n\n* Many prior studies indicate that **high-quality data enhances representation learning**, and filtering unclear samples directly improves data quality. For instance, unlike CLIP, which trains on noisy samples, BLIP uses a data filtering mechanism to remove poorly matched image-text pairs—achieving better performance with fewer training samples.\n\nWe acknowledge that the aforementioned sample removal measures will make our high-quality benchmark somewhat fail to reflect real-world noises, which is discussed in our `A.7` Limitation.\n\nHowever, **our cross-dataset generalization experiments in Table `7` validate the robustness of our proposed method, as most datasets here contain obvious noise** (e.g., CUHK-PEDES has many rough, identity-mismatched text descriptions). The results show that training on our ORBench yields better generalization than other datasets, with our ReID5o method also exhibiting significant superiority. This indicates that ORBench and ReID5o complement each other to enhance robustness against real-world noise.\n\nNaturally, improving robustness against real-world noise remains an urgent research focus. We believe the following directions are worthy of research:\n\n* **Physics-inspired noise modeling and synthesis**. The generation of real-world noises often follows physical laws, but existing data augmentation methods may deviate from the real distribution. We can construct noise models that conform to physical laws to better generate realistic noisy data. For example, by simulating the scattering effect of rain and snow on RGB images, we can enhance the model's robustness to weather-related noises.\n\n* **Dynamic noise-aware multimodal fusion**. In real-world multimodal person ReID applications, different modal data are affected by noise to varying degrees. How to dynamically allocate fusion weights based on the uncertainty of modal noise to obtain robust multimodal representations is also a research direction worth exploring.\n\n* **Progressive denoising curriculum learning**. By imitating the human learning process, we can first let the model learn basic representations from relatively clean data, and then gradually introduce noisier data. Alternatively, we can design some mechanisms to first recover a clean representation and then perform matching based on it, forcing the model to learn invariant representations in the presence of noise.\n\nFollowing your suggestion, we will discuss the robustness to real-world noise detailedly in Section `A.7` in our revised version.\n\n---\n> **`Q5`:** *\"The computing cost is relatively large, which may limit its faster and wider application.\"*\n\n**A:** Thank you for your concern regarding the computing cost. We have taken it into account during the model design process, utilizing a **lightweight** multimodal tokenizing assembler, multi-expert router, feature mixture, and a **shared** encoder. For the results and discussions on this part, please refer to the `Q2`.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `FsmM` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "T2a3VmdCiq", "reviewer": "Reviewer_akx3", "summary": "This paper focuses on person re-identification and proposes a novel task, omni multi-modal person re-identification (OM-ReID), which has been underexplored. The task enables person image search via any modality or their combinations. Based on this, the authors construct ORBench, the first high-quality five-modality dataset (RGB, infrared, color pencil, sketch, text). In addition, the authors also propose a unified model named ReID5o that can effectively achieve multi-modal combined retrieval. Extensive experiments validate the task's rationality, dataset's quality/generalisability, and method's superiority. Overall, the paper makes solid contributions.", "questions": "1. In the process of constructing the ORBench dataset, it seems that you filtered out a large number of low-quality RGB images from existing datasets. What was your motivation for doing so? Because low-quality photos often appear in real-world ReID applications.\n2. In Table 1, those cross-modal methods also seem to perform well. Considering that these methods only support RGB and text, how did you train and test these methods on your dataset?\n3. Even though some visualized qualitative comparison examples, such as Figure 7, effectively demonstrate the high quality of ORBench compared to existing multi-modal datasets, could you conduct comparisons using quantitative metrics? This would make the findings more convincing.\nI would greatly appreciate it if you could address these concerns in the rebuttal.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a novel and underexplored task, omni multi-modal person ReID. Aiming to enable person search via any combination of modalities, this new task holds both research and practical application value. \n2. This paper presents a high-quality five-modality person ReID dataset, ORBench. With rich diversity, it is the first dataset to feature five high-quality modal representations of the same individual, effectively promoting research development in this field.\n3. This paper proposes a simple yet effective multi-modal joint learning framework, ReID5o. If open-sourced, this framework would be of some significance for the community to follow up on this new task.\n4. The authors conducted extensive ablation experiments to verify the effectiveness of each component of the ReID5o method, and performed in-depth evaluations on the high quality and cross-domain generalizability of the proposed dataset.\n5. The paper is fluently written, with clear elaboration of research motivations and technical details, making it easy for readers to follow.\nWeaknesses:\n1. The RGB and infrared modality data in this paper are filtered from existing datasets. If the authors could independently collect new data for these modalities, it might have greater community influence.\n2. To ensure data quality, the data construction process in this paper appears to require substantial manual involvement, which somewhat limits the scalability of the dataset.\n3. The painting styles in ORBench seem somewhat monolithic. Enhancing the diversity of painting styles might better improve generalization ability in real-world scenarios.\nOverall, this paper makes significant contributions, with its strengths outweighing the weaknesses.", "was_revised": true, "final_justification": "Thank you for your detailed reply. I have carefully read the comments from other reviewers and the authors' responses, and most of the concerns and questions that I was interested in have been addressed. I believe the contributions of this research are solid, and both the dataset and methodology will have significant influence. I think this paper meets the acceptance criteria of NeurIPS, so I will upgrade my rating to show my support. Of course, I also hope that the authors can add some discussions to the revised version.", "rebuttal": "We sincerely thank Reviewer `akx3` for the positive and encouraging review. We are grateful for your recognition of our ORBench dataset, ReID5o method, extensive experimental results, and the quality of writing. Below, we respond to your thoughtful questions and suggestions in detail.\n\n---\n> **`Q1`:** *\"The RGB and infrared modality data in this paper are filtered from existing datasets. If the authors could independently collect new data for these modalities, it might have greater community influence.\"*\n\n**A:** The RGB and infrared data of our ORBench are manually filtered from the existing datasets. Then, through significant efforts, we have additionally expanded three modalities: color painting, sketch, and text.\n\nSince collecting a large amount of real-world video data containing both RGB and infrared information of the same person **takes a long time and involves some privacy issues**, we have decided to use the existing datasets LLCM and SYSU-MM01 for these two modalities, **without affecting the core purpose of the research**.\n\nWe have obtained permission for the use and expansion of these two datasets through written communication with their creators, which complies with relevant regulations and requirements.\n\nOf course, as you mentioned, collecting new RGB and infrared data ourselves will indeed increase the diversity of data in the field. We will further improve our work in this regard in future research.\n\n---\n> **`Q2`:** *\"To ensure data quality, the data construction process in this paper appears to require substantial manual involvement, which somewhat limits the scalability of the dataset.\"*\n\n**A:** Considering that **the development of existing technologies**, especially those for fine-grained image caption generation and artistic style painting targeting persons, **is still immature**, we have **indeed invested significant manual involvement**. This is to maximize the quality and diversity of the dataset, minimize noise contamination within it, and ensure that data across various modalities can accurately depict a person from multiple aspects. For further details on the efforts we have made in constructing the dataset, please refer to our response to the second comment from Reviewer `h3Ed`.\n\nCertainly, as you have mentioned, we also believe that large-scale expansion of the dataset is equally important. It will enable models to learn more robust features and facilitate practical application deployment. We will strive to achieve this in our future research endeavors.\n\n---\n> **`Q3`:** *\"The painting styles in ORBench seem somewhat monolithic. Enhancing the diversity of painting styles might better improve generalization ability in real-world scenarios.\"*\n\n**A:** Certainly, using more different AI painting tools would enhance the diversity of painting styles, which, to a certain extent, would be more conducive to generalization in real-world scenarios. However, **most AI art painting tools** tend to produce hallucinations and **perform poorly** when generating paintings based on low-resolution person RGB images. After multiple investigations, we had to select **the only AI tool that meets the requirements**, namely Doubao Art Painting.\n\nTo increase the diversity of paintings, we have made **the following efforts**: On the one hand, during the painting process, we manually required and supervised the model to paint the same person from three perspectives, thereby introducing **perspective diversity**; on the other hand, we generated multiple samples for the same person, with 18 color paintings and 18 sketches per person. Due to the inherent randomness in AI generation, this also brings about a certain degree of **sample diversity**.\n\nOur aforementioned efforts have increased the diversity of the dataset to a certain extent, and the experiments in Table `7` have also demonstrated that our ORBench has better generalization ability compared to other datasets. We will attempt to use more AI painting tools in the future to enhance the diversity of painting styles in the dataset.\n\n---\n> **`Q4`:** *\"In the process of constructing the ORBench dataset, it seems that you filtered out a large number of low-quality RGB images from existing datasets. What was your motivation for doing so? Because low-quality photos often appear in real-world ReID applications.\"*\n\n**A:** The question you raised has been addressed in our response to Reviewer `FsmM`'s fourth question. We have included it below for your convenience.\n\nFirst, please allow us to elaborate on two reasons why we removed unclear samples from the existing datasets:\n\n* Clear, **high-quality RGB samples aid in annotating** accurate, feature-rich data across color paintings, sketches, and text. Since we rely on original RGB images to draw and describe individuals, unclear ones hinder the creation of high-quality, appearance-consistent data—making it hard to fully and accurately capture people's appearance features.\n\n* Many prior studies indicate that **high-quality data enhances representation learning**, and filtering unclear samples directly improves data quality. For instance, unlike CLIP, which trains on noisy samples, BLIP uses a data filtering mechanism to remove poorly matched image-text pairs—achieving better performance with fewer training samples.\n\nWe acknowledge that the aforementioned sample removal measures will make our high-quality benchmark somewhat fail to reflect real-world noises, which is discussed in our `A.7 Limitation`.\n\nHowever, **our cross-dataset generalization experiments in Table `7` validate the robustness of our proposed method, as most datasets here contain obvious noise** (e.g., CUHK-PEDES has many rough, identity-mismatched text descriptions). The results show that training on our ORBench yields better generalization than other datasets, with our ReID5o method also exhibiting significant superiority. This indicates that ORBench and ReID5o complement each other to enhance robustness against real-world noise.\n\n---\n> **`Q5`:** *\"In Table 1, those cross-modal methods also seem to perform well. Considering that these methods only support RGB and text, how did you train and test these methods on your dataset?\"*\n\n**A:** Considering that RGB, infrared, color painting, and sketch data in the dataset essentially still belong to the visual modality, and these cross-modal methods all adopt a visual-text dual-branch architecture, we have adopted the following simple and direct approach to endow them with more multimodal retrieval capabilities: \n\n* For RGB, infrared, painted, and sketched modalities, we directly use a visual encoder to extract features from each image sample. Most of these models use pre-trained CLIP as the encoder, and since CLIP has been pre-trained on hundreds of millions of image-text pairs, its visual encoder is not limited to the RGB modality. Therefore, this method is simple and feasible. \n\n* In the training phase, similarly, we take RGB as the core of alignment and align the features of other modalities with RGB features to learn a unified multimodal space mapping. In the testing phase, for unimodal retrieval, it is only necessary to calculate the cosine similarity between each query and each RGB image in the gallery; for multimodal combined retrieval, we use the sum of the similarity ranking lists of each modal query as the combined similarity ranking list. \n\nThrough the above simple and direct method, we can train and test these cross-modal methods on our ORBench.\n\n---\n> **`Q6`:** *\"Even though some visualized qualitative comparison examples, such as Figure 7, effectively demonstrate the high quality of ORBench compared to existing multi-modal datasets, could you conduct comparisons using quantitative metrics?\"*\n\n**A:** Thank you for this question. A quantitative comparison can indeed better illustrate the high quality of our dataset. \n\nRegarding the quantitative comparison of the text modality, in fact, in Figure `4`, we verified the high quality of the text modality in our dataset by calculating and comparing the text information content across various datasets. The average Shannon entropy of the text data in our dataset is **5.53**, which is higher than that of previous datasets. For example, CUHK-PEDES only has a Shannon entropy of **4.13**. \n\nConcerning the quantitative experiment on painting data, in Figure `5`, we verified the high quality of the painting data in our dataset through manual evaluation, and it excels in both identity consistency and perspective conformity. The proportions of samples with identity consistency and perspective conformity scoring 3 or higher are **97%** and **91.7%** respectively. \n\nConsidering the current lack of effective automated evaluation methods for assessing the quality of painting data, we acknowledge that it is difficult for us to conduct a direct quantitative comparison of painting quality with other painting datasets at this stage. However, it can be clearly seen from Figure `7` that the painting quality of our dataset is significantly better than that of other datasets. \n\nWe guarantee that our dataset is of high quality, and we will make it publicly available to facilitate verification by the research community.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `akx3` again for the valuable time and constructive feedback provided during this review."}]

Author rebuttal:

Meta-review: The paper received two accept, one borderline accept and one borderline rejection ratings. Reviewers praised about the dataset contribution while criticizing the realism of the dataset. Reviewers also raised the ethic problems during review. The authors provided a rebuttal, which addressed most of those concerns (generalization on real-world data etc). AC acknowledges that the ethic problem exists in the whole person ReID community and is not specific to this paper. The dataset contribution can facilitate more research along this direction: apply more advanced deep learning framework for traditional problems. Thus AC recommends to accept the paper.
Decision: Accept (poster)