Unbiased Prototype Consistency Learning for Multi-Modal and Multi-Task Object Re-Identification
Authors: Zhongao Zhou
Bin Yang
Wenke Huang
Jun Chen
Mang Ye
Venue: NeurIPS 2025 spotlight ()
Keywords: Object Re-identification
Mulit-modal
Cross-modal
Primary area: deep_learning

Abstract: In object re-identification (ReID) task, both cross-modal and multi-modal retrieval methods have achieved notable progress. However, existing approaches are designed for specific modality and category (person or vehicle) retrieval task, lacking generalizability to others. Acquiring multiple task-specific models would result in wasteful allocation of both training and deployment resources. To address the practical requirements for unified retrieval, we introduce Multi-Modal and Multi-Task object ReID ($\rm {M^3T}$-ReID). The $\rm {M^3T}$-ReID task aims to utilize a unified model to simultaneously achieve retrieval tasks across different modalities and different categories. Specifically,
to tackle the challenges of modality distibution divergence and category semantics discrepancy posed in $\rm {M^3T}$-ReID, we design a novel  Unbiased Prototype Consistency Learning (UPCL) framework, which consists of two main modules: Unbiased Prototypes-guided Modality Enhancement (UPME) and Cluster Prototype Consistency Regularization (CPCR).
UPME leverages modality-unbiased prototypes to simultaneously enhance cross-modal shared features and multi-modal fused features. Additionally, CPCR  regulates discriminative semantics learning with category-consistent information through prototypes clustering.
Under the collaborative operation of these two modules, our model can simultaneously learn robust cross-modal shared feature and multi-modal fused feature spaces, while also exhibiting strong category-discriminative capabilities. Extensive experiments on multi-modal datasets RGBNT201 and RGBNT100 demonstrates our UPCL framework showcasing exceptional performance for $\rm {M^3T}$-ReID. The code is available at https://github.com/ZhouZhongao/UPCL.

Unbiased Prototype Consistency Learning for Multi-Modal and Multi-Task Object Re-Identification
Zhongao Zhou 1- Bin Yang!* Wenke Huang' Jun Chen
Mang Ye!
School of Computer Science, Wuhan University
Abstract
In object re-identification (RelD) task, both cross-modal and multi-modal retrieval methods have achieved notable progress. However, existing approaches are de- signed for specific modality and category (person or vehicle) retrieval task, lacking generalizability others. Acquiring multiple task-specific models would result in wasteful allocation of both training and deployment resources. To address the practical requirements for unified retrieval, we introduce Multi-Modal and Multi- Task object RelD (M (M'--D The M M'T-ReID task aims to utilize unified model simultaneously achieve retrieval tasks across different modalities and different categories. Specifically, to tackle the challenges of modality distibution divergence and category semanties discrepancy posed MDT-RelI, we design novel Unbi- ased Prototype Consistency Learning (UPCL.) framework, which consists of two main modules: Unbiased Prototypes-guie Modality Enhancement (UPME) and Cluster Prototype Consistency Regularization (CPCR). UPME leverages modality- unbiased prototypes to simultaneously enhance cross-modal shared features and multi-modal fused features. Additionally, CPCR regulates discriminative semantics learning with category-consistent consistent information through prototypes clustering. Under the collaborative operation of these modules, model can simultancously learn robust cross-modal shared feature and multi-modal fused feature spaces, capabilities. Extensive exper-
while also exhibiting strong sneeory-ddicrimmnnte iments on multi-modal datasets RGBNT201 and RGBNT1O demonstrates our UPCL framework showcasing exceptional performance for M'T-RelID. The code is available at attps:/// htt///gttia thub. com/ ZhouZhongao/UPor
Introduction
Object re- identification (ReID) [75, 49, 65, 45, 18, 60, 3, 75, 73, 69, 30, 4] leverages computer vision techniques to identify specific objects (such as persons or vehicles) in videos and still images. ReID technology has been widely applied in intelligent video surveillance, public security, and other related fields. Traditional RelD predominantly focuses on single-modal scenario, where both the query and gallery consist of RGB images. However, RGB cameras highly sensitive to illumination variations. making difficult to accurately capture target information under low- light overexposed conditions. To address the above challenges, Near-Infrared (NI) and Thermal Infrared (TT) modalities have been introduced into ReID tasks, enabling robust imaging in challenging environments [72, 15, 29, 55, 17, 71, 63]. Depending on the retrieval scenarios, the existing ReID methods can be broadly categorized into cross-modal RelD [58, 53, 36, 70, 41, 19, 2] and multi-modal ReID [47, 50, 68, 67]. Specifically, cross-modal ReID focuses on retrieval between two different modalities (c.g..NI-RGB, TI-RGB), whereas multi-modal ReID utilizes RGB, NI, and TI fusion achieve feature matching.
"quual contribution.
I Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multi-Modal Space
Cross-Modal Space
Modality-Unbiisee
Category-Consistent
Prototypes Person
Prototypes
Small Gap
Large Cap
O
Person
Distribution
Divergence
Cluster
A
L Update
O
Large Gap
Small Gap
O
Vehicle
Vehicle
Fused Feature!
Cluster
Clothing
Color
Semantics
Gender
Logo
Discrepancy
Poses
Lights
LUPME
ECPCR
Person
Vchicle
(a) Limitations
(b) Our method
Figure 1: Illustration limitations in M'T-RelD and our method. (a) Distribution divergence: In multi-modal feature space, the multi-modal fused features exhibit smaller gap while cross-modal gap remain larger. Conversely, the opposite characteristic holds in cross- modal feature space. Semanties discrepancy: Different categories of objects possess distinct discriminative semantics. (b) We introduce modality-unbiased prototypes and cluster-derived category-consistent prototypes to enhance the model's comprehensive retrieval capability from both modality and category perspectives.
As illustrated Figure 1, although existing cross-modal and multi-modal approaches have achieved remarkable results, they still suffer from two significant limitations: 1) Real-world surveillance environments present extreme complexity. where targets may appear scenarios captured by either single-modality cameras aligned multi-modal imaging systems. While cross-modal ReID primarily focuses on learning a shared cross-modal feature space, multi-modal ReID emphasizes the effective fusion of different modalities to obtain more robust fused features. The cross-modal and multi-modal approaches follow different optimization directions, thereby resulting in distribution divergence. Therefore, existing ReID models cannot simultancously handle both cross- mross-modal and multi-modi retrieval. 2) The retrieval tasks are confined to a specific category, necessitating separate model training for either person vehicle RelD tasks. In practical scenarios such as criminal investigations, surveillance systems require the capability to simultaneously retrieve both suspects and vehicles. Due to the discrepancy in semantics between diverse categories, existing ReID methods lack the generalization capability to perform unified retrieval across categories. While training dedicated retrieval models for distinct modalities and categories may serve feasible solution, this approach inevitably leads to substantial redundancy in both training and deployment resources.
To meet the real-world demands for retrieval across diverse modalities and categories, propose the Multi-Moda and Multi-Tash object ReID (-'I-ReID). The M'T-ReID task aims to achieve a unified model for simultancous retrieval across multiple modalities and diverse categories. However, achieving bigh-peeformance of M introduces several challenges. Firstly, since cross-modal and multi-modas retrieval models optimize fundamentally different objectives, this leads to challenge I: How to jointly learn both a robust cross-modal shared feature space and an effective multi-modal fusion feature space. Secondly, discriminative features vary significantly across object categories. For instance, person RelD primarily focuses on attributes like pose and clothing while vehicle RelD) emphasizes vehicle type and color, which raises the challenge II How enable model to simultaneously learn category-specific discriminative representations for heterogeneous objects.
To address the aforementioned chanllenges in M'T-ReID, propose Unbiased Prototype Consis- tency Learning framework (UPCL) which comprises two key modules: Unbiased Prototypes- guided Modality Enhancement (UPME) and Cluster Prototype Consistency Regularization (CPCR). For chal- lenge I, UPME enhances both cross-modal shared features and multi-modal fused features through modality unbiased
buballtt-uubiased prototypes, thereby bridging the discrepancy across heterogeneous modalities and simultaneously improving robustness of the overall feature space. For challenge II, CPCR derives category-consister features through prototypes clustering, thereby regulating the model stably acquire category-specific discriminative semantics from diverse categories.
The main contributions of this paper can be summarized as follows:

---

To address the practical demands for retrieval of diverse modalities and categories, we propose novel Multi-Modal and Multi-Tasks object ReID (M'T-ReID).
To address the challenges in M'T-ReID, we propose UPCL which comprises two main components: UPME and CPCR. UPME leverages modality-unbiased unbiased prototypes to simultaneously enhance cooss-modal shared features and
fused features, CPCR regulates the learning of discriminative semantics across diverse object categories through prototypes clustering.
Extensive experiments on the public multi-modal ReID benchmarks RGBNT201 and RGBNT 100 have verified the advantage of our methods, achieving significantly higher accuracy compared to existing counterparts in both cross-modal and multi- modal retrieval scenarios.
2 Related Work
2.1 Cross-modal and Multi-modal Re-identification
Cross-modal re-identification [23, 14, 20, 5, 8, 62, 31, 59, 11] aims to retrieve target RGB images across heterogeneous modalities. The cross-modal retrieval capability of a model primarily depends on the robustness of itscross-modal shared feature space. Wu et al.[52] utilize zero-padding one- stream network with grayscale inputs to learn the shared feature between RGB andNI images. Ye et al.[61] introduce a Channel Augmentation (CA) mechanism that mitigates the modality gap by generating iolee-tttelevant person representations. Liu al.[ propose the Aleorry-Agmeent Unidirectional Metric (MAUM) method to enhance the cross- modality correlation by utilizing two unidirectional metrics. Liang et al. [20] make early attempt at unsupervised cross-modal ReID with a two-stage framework. Yang et [l.[57] design an Augmented Dual-Contrastive Aggregation (ADCA) learning framework for Unsupervised Learning Visible-Infrared Person RelD.
Multi-modal re-identification [47, 15] jointly leverages complementary information from multiple modalities to extract more robust fused features, thereby improving retrieval accuracy. Zheng et [l.[ propose PFNet which bierarchically fuses RGB,NI, and TI features to obtain more robust rep- resentations. Wang et [4.[49] design a Cross-Modal Interacting Module to enhance modality-specific information during feature fusion. Wang el al.[50] utilize the relationship among heterogeneous modalities to fine-tune the network prior to inference, thereby improving generalization to unseen data. Zhang et al.[68] propose a general PromptMA framework, which employs learnable prompts to aggregate modalities and bridge the modality distribution gap. Wang et al. [46] introduce the token permutation to enhance inter-modal interaction and facilitate multi-spectral feature alignment. Zhang al.[67] propose EDITOR framework, which obtains ppatillfrrqueen masks refine multi- modal features. Wang et al.[48] adaptively balances decoupled features using a mixture-offeexeers mechanism to produce more robust multi-modal representations.
Due to the divergent optimization objectives between cross-modal and multi-modal, a single model cannot be effectively applied to both retrieval scenarios simultaneously. Furthermore, significant semantic discrepancies exist among objects of different categories, models trained on one category cannot achieve generalized to retrieval of other categories. To address diverse retrieval requirements in real-world scenarios, we propose the UPCL framework, which trains a unified model capable of performing retrieval across multiple modalities and object categories.
2.2 Prototypes Learning
Prototype is calculated as the mean feature of the instances belonging to the same ID [38]. Due to its simplicity and scalability, prototype plays an indispensable role across various domains, such as few-shot learning [56, 38, 13, 27, 42], unsupervised learning [54, 7, 16, inn.eeentttt learning [74, 76, 66, 43, 37]. and federated learning [21, 12, 26, 22, 28, 25, 40, 64, 34, 35]. In object ReID research [24, 57, 58, 53], prototypes and other feature-centric concepts have also demonstrated significant effectiveness. Luo et al. [24] introduce Center Loss to enhance the feature similarity of same-ID samples in the model. Yang et al.[57] assigns pseudo-labels to unannotated data through clustering and dynamically update the centers of samples corresponding to each pseudo- ID as supervisory signals for network optimization.
Current applications of prototypes or similar concepts primarily leverage their statistical properties at ID-level to enhance feature compactness within the same category, thereby improving network robustness. In this work, we innovatively utilize modality-unbiase prototypes from modality-
3

---

Instances
Features
Prototypes
Memory
Unbiased Memory
Category Memory
prgb
Person
Person
P
Cluster
A
Vehicle
Vehicle
Update
Cluster
P rg
UPME
CPCR
Pni
Jpii
Person-Consiste
Conssse-on Prototype
Prototype
RGB/NI/TI Features
ModalityUnbiased Prototypes
RGB/N//TI Prototypes
Category-Consistent Prototypes
Figure 2: Overview of the proposed UPCL framework. It consists of two core components: Unbiased Prototypes-guided Modality Enhancement (UPME) and Cluster Prototype Consistency Regularization (CPCR). We dynamically update the multi-modal prototype memory for each during every training iteration. UPME aggregates modality prototypes via Equation (9), leveraging their identity-consistent information to strengthen both cross-modal and multi-modal representation learning. CPCR further utilizes category- prototypes clustered from modality prototypes, exploiting their
semantics to regularize the model and achieve more discriminative category- decision boundaries.
consistency perspective and employ clustering strategies obtain category- (rather than identity- level) discriminative features. This approach significantly enhances the unified ReID model's capability to perform robust retrieval across multiple categories and modalities.
3 Method
In this section, we present the Unbiased Prototype Consistency Learning framework (UPCL) which consists of two main modules. The Unbiased Prototypes-guides Modality Enhancement (UPME) module leverages modality-unbiased prototypes to simultaneously enhance both cross-modal shared features and multi-modal fused features, thereby improving the model's performance across different retrieval modes. The Cluster Prototype Consistency Regularization (CPCR) module utilizes modality- unbiased prototypes via clustering derive category-consisten prototypes, which are then utilized to regulate the model's discriminative semantic learning process for different categories. The overview of UPCL is illustrated in Figure 2 and the details are discussed in the following subsections.
3.1 Overall Architecture
Our method utilizes the pretrained CLIP [32] model as the visual encoder which is shared with RGB, NI and TI modalities. Specifically, for the i-th multi-modal instance V; (Vrrb V V the images of three modalities are cropped into equal-sized patches and then mapped embedding vectors with fixed dimensions. Next, we feed these embedding vectors into the visual encoder to obtain their corresponding class token f D and patch tokens y!" E ND where m E {rgb, ni, Li}, Np denotes the number of patch tokens and D is the embedding dimension.

---

Consistent with existing multi-mode methods [72, 49, 50, 46], we employ the label smoothing cross-entropy loss C.n [39] and triplet loss Liri [10] to supervise the learning of visual encoder:
Lg = Lee + Ltri-
To further encourage cross-modal feature alignment, we introduce an additional cross-modal align- ment loss function:
exp (1.7TH (1T) /T) 1==1 exp :
C ((m-n)(() (1) -llo
exp (1f f.T) /T)
Cln-mm) (i) log
13 Σド= exp ((f. fft")
where ( represents the cosine similarity function. f. and f denote the features of i-th instance with different modalities m, n E {rgb, ni, tij. B is the batch size, and r is the temperature parameter. Then the cross-modal loss function between modalities and n can be formulated as:
Lmetn
D ( (1) ((n-m)) (1)
(4)
2B
Building upon this, we derive the cross-modal loss function Leross:
Loross =
Ergbesni + Crgbesti + Emissti-
(5)
Finally, we obtain the base objective La of the framework:
La Lg Lerosss
(6)
3.2 Unbiased Prototypes-guided Modality Enhancement
To enable the model to achieve high performance in both cross-modal and multi-modal retrieval, is essential to design optimization strategies that maintain directional consistency between these two paradigms. Inspired by the successful application prototypes other domains, we leverage the identity- connitt- information encapsulated in prototypes guide the feature learning.
RGB
RGB
ID-Consistent Semantic
ID-Consistent Semantic
Modality- Unbiassd
GlobaL
Large Gapl
Small Gap
Large Gap
A
A
ID- In-Consistent Semantic
A
TI
A
(a)
(b)
(e)
Figure 3: Comparison of different prototypes. (a) Global prototypes; (b) Modality-speciffc prototypes; (C) Modality-unbiant unbbased Prototypes (Ours). It is obviously observed that our proposed Modality- Unbiased Prototypes exhibit superior ID-consistent semantic representation in the feature space.
We compare two commonly used prototypes (global prototypes and modality -eseaiff-- prototypes) in multi-modal learning alongside our newly proposed unbiased prototype in Figure 3.
Global Prototypes. The features of all samples across three modalities under the same ID fused to obtain the global prototype:
pglobal
Σ (ET())
Tefused prgb oni s s!:)
[[e[
where I(c) denotes the indices of all instances with identity Lfused is an MLP designed to fuse the features from the three modalities, producing global feature in DDD
5

---

Modality-Specifie Prototypes. We compute modality- prototypes for identity c:
P
Σ iET(c)
, E {rgb, ni, li}.
||(c))|
(8)
Then this yields set of prototypes P. = ( , for three modalities.
Modality-Unbassed Prototypes. Global prototypes tend to incorporate semantic features biased toward the dominant modality, thereby deviating from identity semantics. Conversely, -ppecity- prototypes primarily focus intra-modal identity information, which inherently carries coddlltt--specifii bias relative to identity consistency. Therefore, in order to optimize the features towards identity-consistee semantics, we comprehensively derive a modalidy-unbiased prototype from Pa = (P Pei p
Ue = (P + Pr + P
As illustrated in Figure 3, compared with global prototypes and modality-specific prototypes, the modality-unbiased prototypes mitigate both inter-modal and intra-modal distribution discrepancies, thereby thereby capturing more robust identity representations.
Under the guidance of modality-unbiasse prototypes, we enhance features of modality m by:
Σ log exp ((f," Ue. /T) B
1
L LUPME
(10)
Σド=1 exp (1 Mej) /T)
where e; represents the identity of the i-th sample. Finally, we formulate the Unbiased Prototypes- guided Modality Enhancement loss through summation:
LUPME = Cir CUrmE + LUPME + LUPME.
(11)
By enhancing semantic consistency across multi-modal features, the discriminative capability for identities in any modality is improved, thereby boosting the cross-modal retrieval performance. Simultancously, the approach reduces inter-modal divergence and improves fusion efficiency, yiclding more robust multi-modal representations.
3.3 Cluster Prototype Consistency Regularization
Current RelD methods typically train dedicated models for specific object categories (e.g., persons or vehicles). However, since discriminative semantic features vary significantly across different object categories, models trained on specific category lack generalization capability. To develop a unified model capable of retrieving diverse object categories, we need to enable the model to accurately capture category-consistern semantic information. As detailed in Section 3.2, the modality-unbiast prototypes inherently encapsulate identity-consisted semantic information that is strongly correlated with sample categories. This intrinsic correlation motivates us to explore a novel approach for extracting category-consistee semantics directly from these prototypes.
Assuming that there are Np and N₂ identities belonging to persons and vehicles, respectively, we NN utilize 4n ) and Lt = {Me c=N,t1 to denote the sets of modality-unbiased unbiaased prototypes. To obtain dntegory-consisten information, we aim to fully integrate prototypes from all identities within a specific category. While the identities of any two prototypes in 21 or 2/ are distinct, they may share highly similar semantic characteristics (e.g. males wearing short sleeves). Such semantic similarity naturally forms clusters among these identities, where the dominant clusters contribute more representative information within 212 or l
To better capture discriminative semantic features of specific categories, we propose to utilize cluster-level statistics instead of identity-level information. The acquisition of reliable cluster-level statistics fundamentally depends on achieving proper clustering of UP and U!". Unlike conventional clustering algorithms K-means [1] and HAC [51]. FINCH [33] operates in completely parameter- free manner, automatically determining the optimal number of clusters based on the inherent similarity relationships among all prototypes. Therefore, we employ FINCH to cluster pu. and pu.r, obtaining the sets of multiple clustered prototypes C' and C":
u PV
Oluster C [C
I1=1
Lp
(12)

---

ur (=)) N=,11
Cluster C [C] J1=1"
(13)
where Lp and L. denote the number of elements in C and C", The I-th cluster prototypes for person and vehicle are denoted C and CT which are calculated by averaging all modality-unbiast unbiased prototypes in 1-th cluster. Then we obtain the category-come
prototype through:
Cate
Σe Lp
c
(14)
Cale
(15)
L 1=1
Then, we introduce the CPCR loss function with Cate? and Cate" for specific category:
B2
exp "atte") \/T/ /T) exp (ffTh Cater) /T) + exp (1ftm Cate") /T)
COPOR =
Σ
log
(16)
|
m
B"
exp (1fTT Cate") /T) exp (ffT Cater) /T) + exp (1ftm Cate") /T)
CCPOR
log
(17)
where and I" denote the index sets all plason-class and vehicle-class instances within batch, respectively, is natural to derive the total CPCR loss:
CCPCR CCPER CCPCR.
(18)
Under the regularization of category-consistent prototypes, the model effectively learns discriminative semantic features that represent object categories, thereby achieving robust performance across diverse retrieval tasks involving different object categories.
Finally, we integrate all components to formulate a comprehensive optimization objective:
Liotal Lb + aLUPME + BCCPCR.
(19)
where a and 3 are the the hyper-parameters to balance the contributions of each loss function.
4 Experiments
4.1 Experimental Settings
Datasets and Evaluation Protocols. To evaluate the performance of our UPCL, we combined RGBNT201 [72] and RGBNT100 [15] to construct multi-modal dataset with diverse object cate- gories. Specifically, RGBNT201 is the first multi-modal person RelD dataset, where each pedestrian ID contains RGB, NI and TI modalities. RGBNTI00 contains the same modalities as RGBNT201, but collects vehicle images instead. We evaluate the performance of our proposed UPCL with Rank-k matching accuracy, mean Average Precision (mAP) which are the commonly utilized metries in object RelD tasks. The evaluation protocol encompasses six cross-moda testing scenarios (R - N. - R, R + T, T + R. N T, and T + N) along with one multi-modal testing configuration (RNT + RNT).
Implementation Details. The implementation platform is Pytorch with a NVIDIA 3090 GPU. We utilize the pre-trained CLIP as the visual encoder. Images of all modalities are resized to 256x128. For data augmentation, we apply random horizontal flipping, cropping and erasing. The batch size is set to 64, sampling images per identity. The training process is conducted with the Adam optimizer for 50 epochs, and the initial learning rate set to 3.5e-4. We select 0.03 as the temperature and & are set as 2.0 and 0.5 respectively. During the testing
parameter The syper-parameters phase, cross-modal retrieval directly computes similarity using features from two modalities, while multi-modal retrieval concatenates features from three modalities for matching.

---

Table I: Comparison with the state-of-the-ar methods. Each model is trained combined dataset consisting of RGBNT201 and RGBNT100, and evaluated separately.
RGBNT201
Methods
R
N
N
R
R
T
T +
R
N
T
T N
RNT +
RNT
Harm_ Meen
mAP
Rank-/
mAP
Rank-/
mAP
Rank-I
mAP
Rank-I
mAP
Rank-1
mAP
Rank-
mAP
Rank-I
mAP
Rank-l
HTT [50]
4.43
3.31
3.59
3.26
2.73
0.48
3.26
0.36
3.26
2.27
4.25
4.31
9.16
4.67
3.83
1.05
TOP-ReID [46]
10.50
8.61
10.71
9.33
3.55
1.32
3.54
1.20
5.40
4,31
6.10
4.67
63.74
64.95
6.27
3.07
PrompiMA [68]
20.37
17.70
19.39
14.35
11.80
7.06
13.77
12.32
11.36
8.49
10.28
6,34
65.71
68.18
15.32
10.95
EDITOR [67]
3,70
2.03
3.38
2.27
3.83
2.39
3.72
0.60
4.37
2.51
3.32
0.96
56.03
56.70
4.26
1.56
DeMo [48]
3.98
2.27
4.33
2.63
3.30
1.44
4.09
3.10
0.60
3.44
2.75
64.35
63.76
4.22
1.82
UPCl.(Ours)
18.54
16.77
18.54
17.56
20.75
19.97
RGBNTI00
Methods
R
N
N
R
T
T R
N
T
N
RNT
RNT
Harm Harm_Meon
Rank-
Rank-
Rank-
Rank-1
Rank
Rank-1
Rank-1
HTT [50]
5.94
5.89
4.64
2.33
3.00
0.64
3.59
1.05
3.57
2.33
3.90
2.33
32.21
53.00
4,48
1.76
TOP-RelD [46]
15.47
20.06
12.24
12.01
3.50
2.86
3.17
1.40
3.96
4.31
3.80
1.92
71.47
89.15
5.48
3.57
PrompIMA [68]
41.73
53.64
42.64
51.78
8.54
7.46
9.96
8.28
7.87
4,90
10.37
8.92
71,07
86.53
13,93
11.28
EDITOR [67]
2.59
1.40
2.99
2.04
3.45
2.92
3.20
2.39
1.57
3.19
2.16
74.25
93.00
3.44
2.28
DeMo [48]
3.12
2.39
2.42
3.07
(Ours)
48.41
64.08
49.83
64.61
16,50
19.42
17.20
18.31
17.26
21.40
17.25
18.08
79.41
94.87
22.63
27.50
Comparison with State-of-the-art Methods
We present comprehensive comparison of UPCL with state-of- totie----thh-rrr art methods outlined in Table To thoroughly evaluate the model's holistic performance on six cross- moosll and one multi- modal testing scenarios, employ the harmonic mean of task-specific metrics as the aggregated performance measure, given that the harmonic mean places greater emphasis on smaller values compared to the arithmetic mean.
As demonstrated in Table 1, on the person-category dataset (RGBNT201). UPCL significantly outperforms all competing methods across all six cross-modal scenarios, which strongly validates its cross-modal matching capability. For multi-modal retrieval results, compared with PromptMA which contains module specifically designed for multi-modal fusion matching, our approach trails by merely 0.8 percentage points in mAP. Regarding the arithmetic mean of all seven metrics, UPCL achieves substantial superiority over other methods, surpassing the second-best approach by 5.43 and 9.02 percentage points on these two metrics, respectively. For the vehicle-category dataset (RGBNTI00) UPCL demonstrates consistent superiority across every scenario. Across both category- datasets (person and vehicle) under all seven evaluation protocols, our method achieves remarkable performance, which conclusively demonstrates its effectiveness for the M'T-RelD task.
4.3 Ablation Studies
To thoroughly validate the effectiveness of our proposed UPME and CPCR modules, conducted extensive ablation experiments on both RGBNT201 and RGBNTION 100 datasets. The experimental protocol involves incrementally integrating introducing any extra modifications, enabling direct performance comparison through controlled ablation studies. As illustrated in Table 2, all reported metrics presented the harmonic mean of performance across seven evaluation scenarios.
proposed modules into the baseline model without
Effect of UPME. By leveraging the Mlentity-cnnssteer information captured in modality-unbiased prototypes, the UPME module enhances feature representations across different modalities, thereby boosting cross-modal and multi-modal representation capabilities. The experimental results demon- strate that our UPME module achieves mAP improvements of 3.20% and 291 on RGBNT201 and RGBNTION 100 respectively, compared to the baseline. When aggregated across both datasets, the module delivers average gains 2.41% % mAP and 3.37% rank-1 accuracy.
Effect of CPCR. The CPCR module clusters pondality-unbiasee prototypes from UPME to derive category-consistent prototypes, then exploits their antegory-discriminee optimization. After incorporating CPCR, the model achieves a 4.10% mAP and 5.99% rank-1 improvement on RGBNT201, while attaining 0.73% mAP and 2.91% rank- gains on RGBNT100.
semantics to guide model

---

Table 2: Ablation study of each component RGBNT201 and RGBNT100.
Components
RGBNT201
RGBNT100
Average
UPME
CPCR
mAP
Rank-1
Rank-5
Rank-10
Runk-1
Rank
Rank-10
Runk-1
Rank-5
Rank-10
13.85
10.06
20.62
27.93
19.89
21.78
24.39
26.39
16.87
15.92
22.51
27.16
16.65
13.98
25.59
33.56
21.90
24.59
28.06
30.61
19.28
19.29
26.83
32.09
20.75
19.97
36.03
46.13
22.63
27.50
31.13
32,79
21.69
23.74
33.58
39.46
Multi-modal Feature Space
Cross-modal Feature Space
Baseline
UPME
UPME + CPCR
Baseline
UPME
UPME + CPCR
Figure 4: The I-SNE visualization of several randomly selected identities. Color indicate identities.
Visualization. To intuitively demonstrate the effectiveness of UPME and CPCR, visualize the [-SNE[44] feature distribution of several identities in Figure 4. Specifically, the three subplots on the left visualize the embedding space of multi-modal fused features. Under the effects of UPME and CPCR, the fused features the same ID progressively converge, while the discriminability between different identities becomes increasingly pronounced. The cross-modal features of identical IDs in the right figure exhibit consistent variation trends. Therefore, the t-SNE visualization clearly indicates that integrating UPME and CPCR enhances the intra-identity consistency and inter-identity discriminability of both fused and cross-modal features.
Effects of Diverse Categories As discussed carlier. different categories of retrieval tar- gets contain distinct discriminative semantics, making it challenging for a model to main- tain high performance across retrieval tasks in- volving diverse categories. To validate this claim, Table presents comparison between sategory-specifii model and a unified model. The category-specifie model is trained single-category dataset, either RGBNT201 or RGBNT100, while the unified model is trained on a multi-category dataset formed by combin- ing RGENT201 and RGBNTI00. It can be clearly observed that, for the same method, the unified model exhibits significant performance drop compared to corresponding category -peeefc model. This suggests that mixing multiple categories in training introduces substantial interference, hindering the model's 's ability to simultancously learn discriminative semantics for all categories. These findings strongly support our hypothesis that multi-category training can negatively impact model optimization.
Table 3: Comparison between specife and uni- fied model. Each model trained on a combined dataset consisting of RGBNT201 and RGBNT100. and evaluated separately. mAP(%) is reported.
RGBNT201
RGBNT100
Methods
Specific
Unified
Specific
Unified
HTT [50]
69.0
9.16
75.7
32.21
TOP-ReID [46]
72.3
63.74
81.2
71.47
PromptMA [68]
78.4
65.71
85.3
71.07
EDITOR [671
66.5
56.03
79.8
74.25
DeMo [48]
79.0
64.35
86.2
79.12
Cross-domain Retrieval Evaluation
In conventional Re-ID evaluations, both training and testing sets are typically collected under the same scene conditions. As result, the learned feature space may be constrained specific environment, making the evaluation results insufficient to reflect the generalization capability required in real- world applications. To assess the cross-domain generalization ability of model, train it combined dataset RGBNT201 and RGBNTI 100, and evaluate it on the unseen MSVR310 dataset.
The comparison results are presented in Table 4. All models show significantly lower detection accuracy on the MSVR310 dataset compared RGBNT201 and RGBNT100, indicating that cross- domain generalization remains highly challenging for existing Re-ID models. In particular, methods

---

Table 4: Performance analysis of cross-domain generalization MSVR310 dataset. All models trained on the combined dataset of RGBNT201 and RGBNT100, and evaluated on the MSVR310 dataset. Rank-k accuracy (%) and mAP(%) are reported.
MSVR310
Methods
+
N N R
R
+
T
T
N
T
T
RNT + RNT
Harm Meon
mAP
Ronk-1
MAP
Rank-/
mAP Rank- /
mnA Rank-
mAP
Ronk-1
mAP
Rank-
mAP Rank- /
mAP Ronk-1
HTT [50]
1.99
1.18
2.33
1.34
1.88
2.20
5.90
8.97
TOP-RelD [46]
4.31
6,43
3.87
3.21
2.23
0.85
1.84
1.52
2.30
1.02
2.54
2.71
14.91
25.55
2.94
1.89
PrompeMA [68]
11.92
14.72
10.71
13.37
2.43
1.02
2.64
1.52
2.82
2.53
3.70
3.21
23.14
31.98
4.28
2.78
EDITOR [67]
2.16
1.18
1.53
0.52
1.59
0.51
1.49
0.63
1.45
0.85
1.35
1.02
17.05
30.96
1.70
0.82
DeMo [48]
0.34
1.35
1.77
1.02
1.76
15.89
26.90
UPCL (Ourx)
13.89
20.30
14.09
20.14
9.43
10.15
9.03
9.14
9.57
11.68
9.35
11.17
24.60
38.07
11.44
13.77
such as HTT, TOP-ReID, EDITOR, and DeMo exhibit severe performance degradation under both cross-modal and multi-mode modal testing scenarios on the unseen MSVR310 dataset. In contrast, the UPCL. and PromptMA methods achieve considerably better results, with UPCL consistently outper- forming PromptMA across various detection settings. These experimental results provide strong evidence that our model possesses excellent cross-domain generalization capability. The superior cross-domain retrieval performance of UPCL. further validates our method from another perspec- tive only does it enhance the robustness of the fused feature space across modalities, but it also strengthens the model's 's ability to discriminate semantic features across different categories.
Limitations
Although our method has demonstrated promising performance on existing multi-modal ReID datasets, it still exhibits several notable limitations. The current multi-modal ReID datasets are relatively scarce, mainly restricted to the pedestrian and vehicle domains. Considering the practical demands of real-world applications, it is crucial to explore how incorporating a broader range of categories affects retrieval performance. In addition, our unified model may slightly underperform task-specific models in a very limited number of test scenarios. Although this is reasonable, this also indicates that our approach still has potential for further enhancement. Therefore, our future work will focus extending UPCL to more generalized multi-modal and multi-task retrieval scenarios.
7 Conclusion
In this paper, we introduce novel M'T-ReID task to address the practical demands of retrieval across diverse modalities and categories. To tackle the challenges in M'T-ReID, we propose the Unbiased Prototype Consistency Learning framework (UPCL) which consists of two main modules UPME and CPCR. Specifically, UPME mitigates the divergence between cross-modal shared spaces and multi- modal fusion distributions by leveraging ddentity-consisterd information from poddality-unbissed prototypes, thereby enhancing both cross-modal and multi-modal representations. Meanwhile, CPCR reduces semantic discrepancies across categories by clustering modality unbiase prototypes to obtain category-consistet prototypes with discriminative semantics. Extensive experiments on multiple datascts validate the superiority and effectiveness of our method, demonstrating its robustness and generalization ability in diverse retrieval scenarios.
Acknowledgements
This work is partially supported by National Natural Science Foundation of China under Grants (62501428, 62176188), the Innovative Research Group Project of Hubei Province under Grants (2024AFA017) the Major Project of Science and Technology Innovation of Hubei Province (2024BCA003, 2025BEA002), Postdoctoral Fellowship Program of China Postdoctoral Science Foun- dation (GZC20241268. 2024M762479), Hubei Postdoctoral Talent Introduction Program (2024HBB- HJD070), Hubei Provincial Natural Science Foundation of China (2025AFB219) and WHU-Kingsoff Joint Lab. The numerical calculations this paper had been supported by the super-computing system in the Supercomputing Center of Wuhan University.
10

---

References
[1] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio González. Gated multimodal units for information fusion. In ICLR Workshop, 2017.
[2] Cuiqun Chen, Mang Ye, and Ding Jiang. Towards modality person re-identification with descriptive query. In CVPR, pages 15128-15133 2023.
[3] Seokeon Choi, Tackyung Kim, Minki Jeong, Hyoungseob Park, and Changick Kim. Meta batch- instance normalization for generalizable person re-identification. In CVPR, pages 3425-3435. 2021.
[4] Zhenyu Cui, Jiahuan Zhou, Xun Wang, Manyu Zhu, and Yuxin Peng. Learning continual compatible representation for re-indexing free lifelong person re-identification. CVPR, pages 16614 16623, 2024.
[5] Pingyang Dai, Rongrong Ji, Haibin Wang. Qiong Wu, and Yuyu Huang. Cross-modality person re-identification with generative adversarial training. In IJCAL, volume 1, page 6, 2018.
[6] Yunpeng Gong, Liqing Huang, and Lifei Chen. Person re-identification method based on color attack and joint defence. In CVPR, pages 4313-4322, 2022.
[7] Yuanfan Guo, Minghao Xu, Jiawen Li, Binghing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hese: Hierarchical contrastive selective coding. In CVPR, pages 9706-9715, 2022.
[8] Xin Hao, Sanyuan Zhao, Mang Ye, and Jianbing Shen. Cross-modality person re-identification via modality confusion and center aggregation. In ICCV, pages 16403-16412, 2021.
[9] Shuting He, Hao Luo, Pichao Wang. Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In ICCV, pages 15012-1502,, 2021.
[10] Alexander Hermans, Lucas Beyer, and Bastian Leibe. defense of the triplet loss for person 2017.
re-identification. arXiv preprint
[11] Zhangyi Hu, Bin Yang, and Mang Ye. Empowering visible-infrared person re-identification with large foundation models. In NeurlPS, pages 117363-11738 2024.
[12] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, pages 16312-16322. IARE, 2023.
[13] Saumya Jetley, Bernardino Romera- Paredes, Sadeep Jayasumana, and Philip Tom. Prototypical priors: From improving classification to zero-shot learning. arXiv preprint 15 2015.
[14) Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. In CVPR, pages 2787-2797, 2023.
[15] Hongchao Li, Chenglong Li, Xianpeng Zhu, Aihua Zheng, and Bin Luo. Multi-spectral vehicle re-identification: A challenge. In AAAI, volume 34, pages 11345-11353, 2020.
[16] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 2020.
[17] Qiwei Li, Kunlun Xu, Yuxin Peng, and Jiahuan Zhou. Exemplar-free lifelong person re- identification via prompt-guidee adaptive knowledge consolidation. IJCV, 132(11):4500-485 2024.
[18] Wen Li, Cheng Zou, Meng Wang, Furong Xu, Jianan Zhao, Ruobing Zheng, Yuan Cheng, and Wei Chu, De-former: Diverse and compact transformer for person re-identification. In AAAI, volume 37, pages 1415-1423, 2023.
[19] Yongxiang Li, Yuan Sun, Yang Qin, Dezhong Peng, Xi Peng, and Peng Hu. Robust duality learning for unsupervised risible-infrared person re-identification. IEEE TIFS, 2025.

---

[20] Wenqi Liang, Guangcong Wang, Jianhuang Lai, and Xiaohua Xie. Homogeneous-to- heterogeneous: Unsupervised learning for rgb-infrared person re-identification. IEEE TIP. 30: 6392-6607,, 2021.
[21] Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Huabin Zhu, Yanchao Tan, Jun Wang. and Yue Qi. Hyperfed: hyperbolic prototypes exploration with consistent aggregation for non-iid data in federated learning. arXiv preprint arXiv:2307. /4384, 2023.
[22] Jiale Liu, Yu-Wei Zhan, Xin Luo, Zhen-Duo Chen, Yongxin Wang, and Xin-Shun Xu. Prototype- based layered federated cross-modal hashing. ICASSP, pages 1-2, 2023.
[23] Jialun Liu, Yifan Sun, Feng Zhu, Hongbin Pei, Yi Yang, and Wenhui Li. Learning memory- augmented unidirectional metrics for cross-modality person re-identification. In CVPR, pages 19366- 2022.
[24] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. In CVPR, pages 0-0, 2019.
[25] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang. Jian Liang, and Jiashi Feng. No fear of het- erogeneity: Classifier calibration for federated learning with non-iid data. In NeurlPS, pages 5972-5984, 2021.
[26] Feng Lyu, Cheng Tang, Yongheng Deng, Tong Liu, Yongmin Zhang, and Yaoxue Zhang. A prototype-based knowledge distillation framework for heterogeneous federated learning. In ICDCSW, pages 1-11, 2023.
[27] Pascal Mettes, Elise Van der Pol, and Cees Snoek. Hyperspherical prototype networks. In NeurlPS, 2019.
[28] Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-iid data. FGCS, 143:93-104, 2023.
[29] Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, and Kang Ryoung Park. Person recognition system based on combination of body images from visible light and thermal cameras. Sensors, 17(3):605, 2017.
[30] Zhiqi Pang, Junjie Wang, Lingling Zhao, and Chunyu Wang. Identity-clothing similarity modeling for unsupervised clothing change person re-identification. In CVPR, 2025.
[31] Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. In CVPR, pages 27197- 27206. 2024.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML., pages 8748-8763. PMLR,, 2021.
[33] Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In CVPR, pages 8934-8943, 2019.
[34] Wei Shen, Wenke Huang, Guancheng Wan, and Mang Ye. Label-free backdoor attacks in vertical federated learning. In AAAL, volume 39, pages 20389-20397, 2025.
[35] Wei Shen, Mang Ye, Wei Yu, and Pong C Yuen. Build yourself before collaboration: Vertical federated learning with limited aligned samples. IEEE TMC, 2025.
[36] Jiangming Shi, Xiangbo Yin, Yeyun Chen, Yachao Zhang, Zhizhong Zhang, Yuan Xie, and Yanyun Qu. Melti-memory matching for unsupervised visible-infrared person re-identification. In ECCV, pages 456-474. Springer, 2024.
[37] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremennal learning. In ICCV, pages 1772-1781, 2023.
[38] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NewrIPS, 2017.
12

---

[39] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, pages 28182886, 2016.
[40] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI, volume 36, pages 8432-8440 8440, 2022.
[41] Xiao Teng, Long Lan, Dingyao Chen, Kele Xu, and Nan Yin. Relieving universal label noise for unsupervised visible-infrared person re-identification by inferring from neighbors. In AAAI, volume 39, pages 7356-7364, 2025.
[42] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classífication: a good embedding is all you need? In ECCV, pages 266-282, 2020.
[43] Marco Toldo and Mete Ozay. Bring evanescent representations life in lifelong class inere- mental learning. In CVPR, pages 16732-16741, 2022.
[44] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMIR, 9(11) 2008.
[45] Tao Wang, Hong Liu, Pinhao Song, Tianyu Guo, and Wei Shi. Pose- Pose-guided feature disentangling for occluded person re-identification based on transformer. In AAAI, volume 36, pages 2540- 2549, 2022.
[46] Yuhao Wang, Xuehu Liu, Pingping Zhang, Hu Lu, Zhengzheng Tu, and Huchuan Lu. Top-reid: Multi-spectral object re-identification with token permutation. In AAAI, volume 38, pages 2024.
[47] Yuhao Wang, Xuehu Liu, Tianyu Yan, Yang Liu, Aihua Zheng, Pingping Zhang, and Huchuan Lu. Mambapro: Multi-modal object re-identification with mamba aggregation and synergistic prompt. In AAAI, volume 39, pages 8150-8158, 2025.
[48] Yuhao Wang, Yang Liu, Aihua Zheng, and Pingping Zhang. Decoupled feature- mixture of experts for multi-modal object re-identification. In AAAI, volume 39, pages 8141-8149, 2025.
[49] Zi Wang, Chenglong Li, Aihua Zheng, Ran He, and Jin Tang. Interact, embed, and enlarge: Boosting poodllity-speciffi representations for multi-modal person re-identification. In AAAI, volume 36, pages 2633-2641, 2022.
[50] Zi Wang, Huaibo Huang, Aihua Zheng, and Ran He. Heterogeneous test-time training for multi-modal person re- re-identification. In AAAI, volume 38, pages 3850-5858, 2024.
[51] Joe H Ward Jr. Hicrarchical grouping to optimize an objective function. JASA, 58(301):236-244,, 1963.
[52] Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, and Jianhuang Lai. Rgb-infrared cross-modality person re-identification. In ICCV, pages 5380-5389, 2017.
[53] Zesen Wu and Mang Ye. Unsupervised visible-infrared person re-identification via progressive graph matching and alternate learning. In CVPR, pages 9548-9558, 2023.
[54] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-maramttric instance discrimination. In CVPR, pages 9733-3742 2018
[55] Kunlun Xu, Chenghao Jiang, Peixi Xiong, Yuxin Peng, and Jiahuan Zhou. Dask: Distribution rehearsing via adaptive style kernel learning for exemplar-free lifelong person re-identification. In AAAI, volume 39, pages 8923, 2025.
[56] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. In NeurlPS, pages 21969-21980, 2020.
[57] Bin Yang, Mang Ye, Jun Chen, and Zesen Wu. Augmented dual-contrastive aggregation learning for unsupervised visible-infrared person re-identification. In ACM MM, page 2843-2851, 2022.
13

---

[58] Bin Yang, Jun Chen, Xianzheng Ma, and Mang Ye. Translation, association and augmentation: Learning cross pross-modality re-identification from single-modality annotation. IEEE TIP, 2023.
[59] Fan Yang, Yang Wu, Zheng Wang, Xiang Li, Sakriani Sakti, and Satoshi Nakamura. Instance- level heterogeneous domain adaptation for limited-labeled sketch-to-photo retrieval. IEEE TMM, 01:2347-2360, 2020.
[60] Yue Yao, Tom Gedeon, and Liang Zheng. Large-scale training data search for object re- identification. In CVPR, pages 155688-5577, 2023.
[61] Mang Ye, Weijian Ruan, Bo Du, and Mike Zheng Shou. Channel augmented joint learning for visible-infrared recognition. In ICCV, pages 13567-13576, October 2021.
[62] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEEE TPAMI, 44(6))2872,2893, 2021.
[63] Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng. David Crandall, and Bo Du. Transformer for object re- re-identification: A survey. IJC,, 133(5):2410-244 24400 2025.
[64] Mang Ye, Wei Shen, Bo Du, Eduard Snezhko, Vassili Kovalev, and Pong C Yuen. Vertical federated learning for effectiveness, security, applicability: A survey. ACM Computing Surveys, 17(9):1-32, 2025.
[65] Hao Yu, Xu Cheng, Wei Peng, Weihao Liu, and Guoying Zhao. Modality unifying network for visible-infrared person re-identification. In ICCV, pages 11185-11195, 2023.
[66] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In CVPR, pages 6982-6991, 2020.
[67] Pingping Zhang, Yuhao Wang, Yang Liu, Zhengzheng Tu, and Huchuan Lu. Magic tokens: Select diverse tokens for multi-modal object re-identification. In CVPR, pages 17117-17126, 2024.
[68] Shizhou Zhang, Wenlong Luo, De Cheng, Yinghui Xing, Guoqiang Liang, Peng Wang, and Yan- ning Zhang. Prompt-based modality alignment for effective multi-modal object re-identification. IEEE TIP, 2025.
[69] Xiao Zhang, Yixiao Ge, Yu Qiao, and Hongsheng Li. Refining pseudo labels with clustering consensus over generations for unsupervised object re- re-identification. In CVPR, pages 3436- 3445, 2021.
[70] Yukang Zhang, Yang Lu, Yan Yan, Hanzi Wang, and Xuelong Li. Frequency domain nuances mining for visible-infrared person re-identification. IEEE TIFS, 20:541 1-5424, 2025.
[71] Yukang Zhang, Yan Yan, Yang Lu, and Hanzi Wang. Adaptive middle modality alignment learning for visible-infrared person re-identification. IJCV, 133(4))216--2196, 2025.
[72] Aihua Zheng, Zi Wang. Zihan Chen, Chenglong Li, and Jin Tang. Robust multi-modality person re-identification. In AAAI, volume 35, pages 3529-3537, 2021.
[73] Xiao Zhou, Yujie Zhong, Zhen Cheng, Fan Liang, and Lin Ma. Adaptive sparse pairwise loss for object re-identification. In CVPR, pages 19691-19701, 2023.
[74] Fei Zhu, Xu-Yao Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. CVPR. pages 5871-5880, 2021.
[75] Haowei Zhu, Wenjing Ke, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Dual cross-attention learning for fine-grained visual categorization and object re-identification. In CVPR, pages 4692-4702 2022.
[76] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. CVPR, pages 9296-9305, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are clearly stated in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We included the discussion of the limitations our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our work focuses on computer vision applications, not including theoretical
results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all the details of our proposed methods to reproduce the results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [No]
Justification: The datasets used in this paper are fused directy by two public datasets which be downloaed from https://gite htttps///giibb...mm////9/2999999 comm99249739992/Deo We will release the relevant algorithm code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wttpccccpppcccccccccccccccccccct gww.ides////eeeeee/oooece heto//oodiimiaaaeeeeeeeee ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The training and testing details are available in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the limited resources, our paper does not provide error bars. Also, previous relative methods do not provide error bars either.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information of computer resources is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the delines?
NeurIPS Code of Ethics https: / http///eurrpp.ee/ee cc
Answer: [Yes]
Justification:
The research conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification:
Our research has no negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of assets used the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https htt////eerpppp.pe http///aaap..////oeeeeeeeeeeeeee 22/ps///ee.oeeeeeeee/22///////// for what should or should not be described.
21


Reviews:
[{"review_id": "g0RRT8GgLQ", "reviewer": "Reviewer_wjAM", "summary": "This method proposes to address both multi-modal and cross-modal re-id tasks, and person and vehicle re-id tasks, within a unified model. It introduces the Unbiased Prototype Consistency Learning framework, which designs an Unbiased Prototypes-guided Modality Enhancement module to enhance cross-modal shared features and multi-modal fused features simultaneously, and proposes a Cluster Prototypes Consistent Regularization module to regulate the learning of discriminative semantics across diverse object categories through prototype clustering.", "questions": "1. What is the reason for the large gap between fused features in the Cross-Modal Space under Distribution Divergence? It is easy to understand that in the Multi-Modal Space, the multi-modal fused features tend to be compact within the same identity due to the identity loss objective, i.e., they exhibit a small gap. A large gap in this space implies insufficient alignment across modalities. However, in the Cross-Modal Space, since the three modality-specific features are already pulled closer due to cross-modal loss and tend toward a small gap, why do the fused features still exhibit a large gap? Does simple concatenation inherently lead to a large gap?\n\n2. Is it really necessary to unify person and vehicle re-identification tasks into a single model? Methods such as MambaPro and ICPL-ReID adopt parameter-efficient fine-tuning (PEFT) strategies, achieving strong performance with fewer trainable parameters and more flexible model designs. From this perspective, these methods can also address the problem efficiently.\n\n3. The motivation behind cross-modal shared feature space and multi-modal fusion feature space lacks in-depth discussion.\n\n4. Why is U_c not biased toward a dominant modality? Eq. (9) seems to be equivalent to taking the mean in Eq. (8), which is in turn based on Eq. (7). What is the difference among Eq. (7), Eq. (8), and Eq. (9)?\n\n5. What is the performance upper bound when a single model handles different tasks? Are there more multi-modal combination experiments, such as R+N → N+T? Is there any comparison with more dataset combinations and scenarios?\n\n6. The cross-modal retrieval performance of existing MM-ReID methods on RGBNT201, and RGBNT100 is lacking. Why does the baseline method in Table 2 outperform existing multi-modal methods that also use CLIP as the encoder? Why is there such a significant performance gap between PromptMA and DeMo?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n1.This method provides a reasonable analysis of the challenges in joint training for multi-modal and cross-modal tasks, specifically focusing on learning a robust cross-modal shared feature space and an effective multi-modal fusion feature space.\n\n2.The proposed method introduces two different types of prototypes to address modality-level and category-level multi-task problems, respectively.\n\n3.The figures in the paper are well-drawn, and the overall presentation is relatively clear.\n\n###  Weaknesses:\n1.The motivation for integrating person and vehicle re-id into a unified model is not sufficiently justified.\n\n2.The method shows limited innovation. The proposed Modality-Unbiased Prototypes and Category-Consistent Prototypes are essentially simple prototype aggregations, without a deep exploration of the critical role prototypes play in the M^3T-ReID task.\n\n3.The designed contrastive loss lacks effective explanation, making it difficult to understand how it effectively addresses the practical challenges of the M^3T-ReID task.", "was_revised": true, "final_justification": "The author's responses have solved my concerns.", "rebuttal": "## **Response to Reviewer wjAM**\nWe sincerely thank you for dedicating your time to review our work and for your constructive feedback. We hope this rebuttal will help clarify our contributions and encourage a favorable reevaluation. We will also\nimprove writing , clarity, correct typos, and release our code to better show the details.\n\n---\n### Weaknesses & Questions \n\n**W 1) & Q 2) :  Motivation and Necessity of M³T**\n\n**A1** :We thank the reviewer for the insightful comment. Real-world surveillance often involves diverse object types and multiple sensing modalities. While PEFT-based approaches like ICPL-ReID achieve competitive results with lower training costs, they are still limited to single-objective tasks and require separate models for different scenarios, resulting in redundant and expensive deployment. We propose the M³T task, which aims to address multiple retrieval modalities and identity categories within a unified model UPCL. This approach not only reduces deployment overhead but also maintains strong retrieval performance across diverse scenarios. In future work, we will explore how to leverage PEFT strategies to enhance the efficiency and adaptability of the M³T task.\n\n\n**Q 1) & Q 3):  Divergence between Cross-modal and Multi-modal Features**\n\n**A2** :Thank you for the insightful question. In the cross-modal embedding space, most existing cross-modal loss functions primarily aim to enforce modality-invariant identity representations across different modalities. However, they often neglect the preservation of modality-specific identity cues. As a result, the substantial gap observed in the fused multi-modal space does not originate from the fusion strategy itself (e.g., concatenation), but rather from the loss of modality-specific identity information during cross-modal learning, which in turn causes divergence within each modality.\n\nTo further illustrate this characteristic, we conduct a feature comparison experiment between DeMO and our proposed UPCL framework. Specifically, for two image pairs of the same identity from the RGBNT201 test set, we extracted features using both DeMo and UPCL, and computed pairwise feature distances across all modalities.  The sample feature distances are respectively shown in the two tables below. Values enclosed in parentheses in the second table represent the changes relative to the corresponding figures in the first table. The results show that for the same-identity samples A and B, while the cross-modal feature distance decreases, both the intra-modal and fused multi-modal feature distances increase. This phenomenon suggests that the variation in the fused feature space stems from the loss of modality-specific identity information, rather than from the fusion strategy itself. Our method addresses identity-level and category-level consistency by mitigating the loss of modality-specific information during cross-modal learning, while simultaneously enhancing modality-specific and modality-relevant discriminative cues, thereby improving the model’s performance across diverse retrieval scenarios.\n\n| DeMO|  |    |   |  | \n| ------  | ------  | --- | --- | --- | \n|         |  RGB_B | NI_B | TI_B | RNT_B | \n|   RGB_A |   0.62  |0.88|  1.01| -| \n|   NI_A  |   0.92  |0.71|  1.16| -|\n|   TI_A  |   1.04  |1.18|  0.81|-|\n|   RNT_A  |   -  |-|  -| 0.65|\n\n| UPCL|  |    |   |  | \n| ------  | ------  | --- | --- | --- | \n|         |  RGB_B | NI_B | TI_B | RNT_B | \n|   RGB_A |   0.65(+0.03)  |0.69(-0.19)|  0.91(-0.10)|  -| \n|   NI_A  |   0.71(-0.21)  |0.73(+0.02)|  1.15(-0.01)|  -| \n|   TI_A  |   0.85(-0.19)  |1.13(-0.05)|  0.87(+0.06)| -| \n|   RNT_A  |   -  |-|  -| 0.74(+0.09)|\n\n**W 2) :  Prototype-based Innovations for the M³T Task**.\n\n**A3** :\n1）**Conventional VI-ReID Methods in M³T**. Conventional VI-ReID cross-modal methods are not readily extendable to multi-modal retrieval scenarios and often suffer significant performance drops under more complex modality combinations. For example, we evaluated the VI-ReID-specific method DEEN on the RGBNT201 dataset. While DEEN achieves mAP/R1 scores of 22.27/22.41 and 16.51/12.77 under R-N and R-T settings respectively, its performance drastically declines to only 10.05/8.78 under the N-T setting.\n\n2）**Multimodal ReID Methods in M³T**. Similarly, existing multi-modal methods struggle when faced with diverse retrieval categories. For instance, although the DeMO algorithm achieves strong performance on the standalone RGBNT201 dataset (79.0/82.3 in mAP/R1), its accuracy drops markedly to 64.4/63.8 under the M³T task setting, where multiple modalities and categories are jointly considered.\n\n3）**Innovations of Our Method**.  Existing cross-modal and multi-modal methods are typically designed with fixed modality characteristics, limiting their scalability to another modalities.  Our prototype-based approach possesses greater flexibility, enabling seamless extension to a wider range of modalities and identity categories. Moreover, our prototype-based method achieves mAP/R1 scores of 22.33/23.21 (R-N), 16.77/14.59 (R-T), 17.93/14.47 (N-T), and 64.91/67.12 (multi-modal) under the M³T task setting. These results clearly surpass those of existing cross-modal and multi-modal methods. In summary, our method achieves superior experimental results within a simple framework, while demonstrating strong portability and scalability, collectively validating the innovation of our approach.\n\n\n**W 3）& Q 4) :  The Innovation and Effectiveness of Components in UPCL**\n\n**A4** :\n1）**Discrepancies across Different Prototype Types**\nEq. (7) aggregates features from all modalities of the same identity, but tends to be dominated by one modality. Equation (8) computes a modality-specific prototype for each individual modality, but these prototypes tend to be biased toward the characteristics of their respective modalities. To mitigate this, Equation (9) performs an aggregation over modality-specific prototypes, balancing the identity-related information from each modality. This generates a robust modality-unbiased prototype that better captures the consistent semantic representation of the corresponding identity. In addition, within CPCR, we perform clustering over modality-unbiased prototypes belonging to the same category, thereby producing in a category-consistent prototype that encapsulates class-level semantic consistency.\n\n2）**Clarification on the Role of Contrastive Loss**.\nBased on the identity-level and the category-level consistenc semantics in the above two types of prototypes, UPCL utilizes a contrastive loss to promote more compact feature distributions for the same identity across different modalities.  Compared with the baseline, UPCL achieves average mAP/R1 improvements of 6.90/9.91 on RGBNT201 and 2.70/5.72 on RGBNT100 across all retrieval modality combinations（six cross-modal and one multimodal testing scenarios. These results clearly demonstrate the effectiveness of the proposed method. \n\n\n\n\n\n\n**Q 5) : Single-Model Performance and Extended Retrieval Scenarioss**\n\n**A5** :\n1） **The Upper Bound of Single-model Performance**. We evaluated DEEN and DeMO on the single RGBNT201 dataset for cross-modal and multi-modal retrieval tasks, respectively. DEEN achieves mAP/R1 scores of 22.17/22.41, 16.51/12.77, and 10.05/8.78 on R-N, R-T, and N-T retrieval tasks. DeMO achieves 79.0/82.3 in the multi-modal retrieval setting. However, DeMO suffers a significant performance drop under the M³T setting, yielding 64.35/63.76（-14.65/-18.54） under the multi-modal retrieval setting. In comparison, our method achieves superior performance of 64.91/67.12, demonstrating better generalization and robustness in complex multi-modal, multi-object scenarios.\n\n2）**Extended Retrieval Scenarioss**. Theoretically, there are 7 modality types (3 uni-modal, 3 bi-modal, 1 tri-modal), resulting in 49 total retrieval scenarios. Due to the limited number of samples in existing multi-modal ReID datasets and the lack of a standardized cross-modal evaluation metric, we focus on commonly-used retrieval setups for fair comparison. Your suggestion is highly valuable, and we plan to develop multimodal ReID datasets of new categories and explore the full spectrum of modality combinations in future work.\n\n\n**Q 6) : Analysis of Performance Differences Among Multimodal Methods**\n\n**A6** : 1) **Performance Improvement of Baseline**. Our baseline includes additional cross-modal alignment losses (all shown in Eq. 5), which directly enhance consistency across modalities. This is not present in most CLIP-based methods, hence the performance improvement.  \n2) **Comparison between DeMo and PromptMA**. DeMo designs modality-specific experts for fusion, which enhances multi-modal integration but does not explicitly optimize for cross-modal consistency. In contrast, PromptMA introduces cross-modal interactions at the prompt level, which significantly boosts cross-modal retrieval performance. Thus, the performance gap between PromptMA and DeMo reflects the difference in design focus—fusion versus alignment."}, {"review_id": "JvnNSrqB0s", "reviewer": "Reviewer_aGKs", "summary": "The paper introduces a new Multi-Modal and Multi-Task Re-ID (M3T-ReID) task and proposes the UPCL framework, which couples Unbiased Prototypes-guided Modality Enhancement (UPME) and Cluster Prototypes Consistent Regularization (CPCR). UPME leverages identity-consistent information from modality-unbiased prototypes to simultaneously enhance both cross-modal and multi-modal feature representations. CPCR performs clustering on modality-unbiased prototypes to derive category-consistent prototypes with discriminative semantics. Experiments demonstrate the challenge of the M3T-ReID task and the effectiveness of the UPCL framework.", "questions": "(1). The prototype memory bank is rebuilt at the start of every epoch; what is the additional computational and time overhead introduced by this step in practical training?\n(2). Have you performed a sensitivity study on the loss-weight hyperparameters to show that the method is robust across a reasonable range of settings?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This study introduces a well-motivated M3T-ReID task that explicitly addresses the multimodal and cross-category challenges encountered in real-world Re-ID scenarios. Building on this, the proposed UPCL framework has been exhaustively validated—its constituent modules supported by extensive experiments and compelling empirical evidence demonstrating the method’s overall robustness and high quality.\nThe task definition and the high-level framework are easy to follow, yet the description of each prototype-learning step could be more explicit. In addition, a more detailed introduction of the dataset can make it easier for readers to understand, and L229 should be sampling eight pairs or instances rather than images.\nBy widening Re-ID toward a universal paradigm that spans both task types and object categories, M3T-ReID opens a fertile avenue for future unified retrieval research.\nWhile using prototypes is not entirely new, applying modality-unbiased and category-consistent prototype learning in such a broad multi-task setting is a reasonable contribution, though the paper should clarify differences from the most recent prototype-based methods.", "was_revised": true, "final_justification": "I have no more concerns. Good luck!", "rebuttal": "### **Response to Reviewer aGKs**\n\nWe greatly appreciate the reviewer’s time and effort in evaluating our paper. We are grateful for your thoughtful feedback, which has guided us in refining our manuscript. We will release our code to better illustrate the details.\n\n---\n### Weaknesses & Questions \n**Q 1) :  Construction of the Prototype Memory Bank**\n\n**A1** : \n We recalculate the prototype for each identity at the start of every epoch. Although this introduces additional computational overhead, the prototypes encapsulate both class-level and identity-discriminative information, which significantly enhances retrieval accuracy across various modalities.\n\n**Q 2) : Ablation Study of Loss Weight Hyperparameters**\n\n**A2** :   We conduct ablation studies to evaluate the impact of different loss weight configurations for α and β. The detailed results are presented in the table below. As shown, our proposed method consistently achieves strong and stable performance across a wide range of loss weight settings, demonstrating its robustness to hyperparameter variations.\n\n*Table. **Ablation study of loss-weight hyperparameters $\\alpha$ and $\\beta$***\n\n| $\\alpha$| $\\beta$ |  RGBNT201  | RGBNT100  | Average |\n| ------  | ------  | --- | --- | --- |\n|         |         | mAp ｜Rank-1| mAp ｜Rank-1|mAp ｜Rank-1|\n|   0.5   |   0.5   |17.93｜17.16|  21.97｜25.47| 19.95｜21.32\n|   1.0   |   0.5   |19.37｜18.64|  22.11｜26.86| 20.74｜21.85\n|   **2.0**   |   **0.5**  |**20.75**｜**19.97**|  **22.63**｜**27.50**| **21.69**｜**23.74**\n|   2.0   |   1.0   |20.39｜19.75|  22.52｜27.17| 21.46｜23.46\n|   2.0   |   2.0   |20.27｜19.18|  22.23｜27.09| 21.25｜23.14"}, {"review_id": "dYkBYoVOMZ", "reviewer": "Reviewer_U2xk", "summary": "This paper introduces a novel Multi-Modal and Multi-Task object Re-Identification (M3T-ReID) problem, aiming to deploy a unified model that simultaneously handles person and vehicle retrieval under both multi-modal and cross-modal settings. To solve it, the authors propose UPCL framework with two modules: UPME builds modality-agnostic prototypes to align the cross-modal shared space as well as the multi-modal fusion space. CPCR clusters these prototypes to impose category-consistent semantics across heterogeneous classes. Extensive experiments on RGBNT201 (person) and RGBNT100 (vehicle) show that UPCL surpasses SOTA multi-modal ReID methods and delivers consistent gains in both multi-modal and cross-modal tasks.", "questions": "1. The experimental comparison focuses on multi-modal methods. Have the authors evaluated UPCL against strong uni-modal or traditional cross-modal ReID approaches?\n2. The number of identities and the number of images per identity in the RGBNT201 and RGBNT100 datasets are quite different. Did the authors apply any balancing strategy during training or study how this imbalance affects performance?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The manuscript is clearly written, logically structured, and easy to follow.\n2. Training a unified Re-ID model that can simultaneously handle heterogeneous modalities and object categories is a highly meaningful goal. M3T-ReID task proposed in this paper carries strong practical significance. Moreover, the framework and ideas presented here can further spur the development of more general-purpose Re-ID.\n3. The UPME and CPCR proposed in the paper promote M3T-ReID learning through modality-unbiased prototypes and category prototypes. The framework remains conceptually simple while directly addressing both modality divergence and semantic heterogeneity.\n4. The prototype-based representation strategy introduced in this work is highly inspiring and readily transferable to other domains—a commendable innovation.\n5. The model is benchmarked on two types of datasets under seven settings (six cross-modal + one multi-modal), demonstrating strong and consistent performance improvements.\n\nWeaknesses\n1. This paper only uses one person dataset RGBNT201 and one vehicle dataset RGBNT100 to implement of different types of objects ReID. Incorporating additional categories or environments would better validate the robustness claimed for M3T-ReID. \n2. I think pseudocode can be added to show the process of the proposed UPCL framework more clearly. This can lower the barrier for researchers in adjacent domains to leverage and build upon your methodology.\n3. In Figure 4 and L269-L276, the UPME module is incorrectly labeled as UMPE or UMPR.\n4. All reported experiments are conducted exclusively on the hybrid RGBNT201 + RGBNT100 dataset, with no attempt to evaluate the proposed approach on a mixed RGBNT201 + MSVR310 setting.", "was_revised": true, "final_justification": "The response has addressed all my concerns, and I will raise my rating score.", "rebuttal": "### **Response to Reviewer U2xk**\n\nThank you for your time and thoughtful review. We appreciate your positive feedback on our contributions, writing clarity, and experimental design. We will release our code to provide greater clarity and transparency regarding the implementation details. We hope our responses below address your concerns and lead to a favorable reassessment.\n\n---\n\n### Weaknesses & Questions \n\n**Q 1) :  Comparison with Traditional Cross-modal Methods**\n\n**A1** :  The cross-modal method DEEN achieves Rank-1 accuracies of 22.41, 12.77, and 8.78 on R-N, R-T, and N-T on the RGBNT201 dataset, respectively. Under more challenging settings involving multiple retrieval modalities and increased category interference, our method achieves corresponding accuracies of 23.21, 14.59 and 14.47. It can be observed that our method consistently outperforms DEEN all three scenarios.\n\n\n**Q 2): Imbalance in the Number of Identities and Images**\n\n**A2** : We acknowledge that there exists a significant discrepancy in the number of identities and the number of images per identity between the RGBNT201 and RGBNT100 datasets. However, the primary focus of this work is to address the challenges arising from heterogeneous retrieval modalities and category variations, rather than applying any strategy to balance the distribution of identities or image counts. That said, we agree that a unified ReID model should be capable of handling such imbalanced scenarios in real-world applications. Therefore, we plan to explore strategies for mitigating these sample imbalance issues in our future work.\n\n**W 1) & W 4) :  Dataset Selection and Configuration**\n\n**A3** :  1） **Scarcity of Multimodal ReID Datasets**. Your suggestion aligns well with the limitations we discussed in Section 5 of our paper. Indeed, incorporating additional object categories or environmental contexts would further demonstrate the effectiveness and generalizability of our proposed method. However, existing multi-modal ReID datasets are still relatively limited, mostly constrained to pedestrian and vehicle categories. In future work, we will actively monitor the release of new multi-modal datasets covering other categories, and we also plan to construct our own datasets for these categories. These will be integrated into our framework to support more comprehensive and in-depth studies.\n\n2） **Selection of Datasets for Cross-Domain Generalization**. Given the similarity in data collection environments between RGBNT201 and RGBNT100, we conduct the main experiments in our paper on the combined RGBNT201 + RGBNT100 dataset to validate the effectiveness of our method in both cross-modal and cross-category scenarios. In contrast, MSVR310 is collected under a different setting, and thus we use it separately for cross-domain generalization experiments to evaluate the generalization ability of the model trained on RGBNT201 + RGBNT100.\n\n\n**W 2) & W 3) : Textual Errors and Pseudocode**  \n\n**A4** :  Thank you for your suggestion. We will revise  the textual errors and include the corresponding pseudocode in the supplementary materials of the final version to facilitate reproduction and further development by other researchers."}, {"review_id": "9UmoT3nCim", "reviewer": "Reviewer_NDQB", "summary": "This paper introduces a new and practically important Re-Identification (Re-ID) scenario with both heterogeneous modalities and different object categories. To address the challenges posed by this setting, the authors propose UPCL to hierarchically leverage the modality-agnostic and category-specific cues embedded in prototypes. Extensive experiments on two public multi-modal ReID datasets verify the effectiveness of UPCL.", "questions": "See the weaknesses above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths：\n\n1. To the best of my knowledge, this paper is the first to analyze how heterogeneous modalities and diverse object categories impact Re-ID models, and it presents a concise yet effective approach that boosts performance across a range of retrieval scenarios.\n2.The manuscript is is well-structured with a coherent narrative flow, and the accompanying figures are clear and informative.\n3. To tackle retrieval challenges stemming from heterogeneous modalities and different categories, the motivation and formulation of the core modules are articulated concisely：（1）Unbiased Prototypes-guided Modality Enhancement (UPME) simultaneously enhances cross-modal representations and multimodal fusion features. （2）Cluster Prototypes Consistent Regularization (CPCR) regularizes the learning of discriminative semantics over diverse object categories.\n4. Extensive experiments show that the approach achieves superior performance across multiple evaluation settings on the RGBNT201 and RGBNT100 benchmarks. \n5. In cross-domain comparison experiments, the proposed method consistently achieves the best generalization performance.\n\nWeakness：\n\n1. Multimodal Re-ID benchmarks usually resize vehicle and pedestrian images to category-specific resolutions to suit their respective retrieval scenarios. In this study, however, images from all categories are uniformly normalized to 256 × 128 during experimentation. Might this protocol diverge from real-world deployment conditions or incur adverse effects?\n2. Are the temperature hyperparameters in Eqs. (10) and (16) instantiated with the same value? Moreover, the manuscript does not examine how varying the temperature affects model performance, so an ablation study on this hyperparameter is strongly recommended.\n3. In each training epoch, are the prototypes recomputed from scratch, kept static, or updated through a momentum-based mechanism? The paper does not explicitly specify the prototype update mechanism.\n4. The module label in Figure 4 contains a typographical error: “UMPE” should be revised to “UPME.”", "was_revised": true, "final_justification": "The proposed method is novel, to the best of my knowledge, this paper is the first to analyze how heterogeneous modalities and diverse object categories impact Re-ID models, and it presents a concise yet effective approach that boosts performance across a range of retrieval scenarios. During the rebuttle, the ahthous have well responsed my concerns. Besides, three reviewers consistently tends to accept it and one give the borderline reject. Based on above consideration, I tend to accept it.", "rebuttal": "### **Response to Reviewer NDQB**\n\nThank you for your detailed review and helpful suggestions. Your feedback is highly appreciated and has played an important role in enhancing the presentation and technical depth of our work. We will release our code to provide the implementation details.\n\n---\n### Weaknesses & Questions \n**W 1) :  Effect of Image Resizing on Performance**\n\n**A1** : We normalize images from different categories to a resolution of 256×128 and use a shared tokenizer for processing. In theory, employing separate tokenizers for pedestrians and vehicles while sharing a common encoder could potentially lead to better performance. However, the imbalance in data distribution across different categories may introduce optimization difficulties. Furthermore, our experimental results reveal that resizing vehicle images to the same resolution as pedestrian images has a relatively minor effect on retrieval performance. Specifically, we conduct experiments with DeMo on the RGBNT100 dataset with different input resolutions (256×128 and 128×256). The results of mAP/R1 under 256×128 and 128×256 are 86.0/97.2 and 86.7/97.7 . There exists only a negligible performance difference of around 0.7%/0.5% , which can be attributed to the fact that most original images have widths smaller than 128 pixels, resulting in minimal data compression or loss during resizing.\n\n\n**W 2) :  Analysis of Temperature Hyperparameter** \n\n**A2** : The temperature hyperparameters used in Equations (10) and (16) are set to the same value. We have supplemented ablation studies on the temperature hyperparameter. The detailed experimental results are shown in the table below. As observed, our method demonstrates its effectiveness across different temperature settings.\n\n*Table. **Ablation study of temperature hyperparameters $\\tau$  \n\n| $\\alpha$|  RGBNT201  | RGBNT100  | Average |\n| ------  |  --- | --- | --- |\n|         | mAp ｜Rank-1| mAp ｜Rank-1|mAp ｜Rank-1|\n|   0.01   | 19.45｜18.29|  20.44｜23.23| 19.95｜20.76\n|   **0.03**|   **20.75**｜**19.97**|  **22.63**｜**27.50**| **21.69**｜**23.74**\n|   0.05   |18.49｜17.96|  22.36｜25.41| 20.43｜21.54\n|   0.07   |  18.89｜17.68|  21.98｜24.72| 20.44｜21.20\n|   0.09   |  18.30｜17.58|  20.87｜23.55| 19.59｜20.57\n\n**W 3) :  Computation Details of Prototypes**  \n\n**A3** :  At the beginning of each training epoch, we recalculate the prototype corresponding to each identity, which is then used in the subsequent training process. Thank you for pointing out this omission, we will provide a detailed explanation of this computation procedure in the manuscript.\n\n\n**W 4) :  Textual Errors**  \n \n**A4** :Thank you for your suggestion. We will revise and correct the textual errors in the  the final version."}]

Author rebuttal:

Meta-review: This paper presents a new method for Multi-Modal and Multi-Task Object Re-Identification. After rebuttal and discussions, all reviewers acknowledged the novelty and strengths of the paper and agreed that the authors have addressed their concerns. Reviewers wjAM and U2xk further raised their ratings in confirming the acceptance. AC agrees with this recommendation and therefore is happy to accept the paper. Authors are required to update the rebuttal and discussion contents to the camera-ready version of the paper to improve it so as to address the raised concerns in the final paper.
Decision: Accept (spotlight)