Title: MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-modal Person Re-identification; Modality Decoupled Learning
Abstract: The challenge of inconsistent modalities in real-world applications presents significant obstacles to effective object re-identification (ReID). However, most existing approaches assume modality-matched conditions, significantly limiting their effectiveness in modality-mismatched scenarios. To overcome this limitation and achieve a more flexible ReID, we introduce MDReID to allow any-to-any image-level ReID systems. MDReID is inspired by the widely recognized perspective that modality information comprises both modality-shared features, predictable across modalities, and unpredictable modality-specific features, which are inherently modality-dependent and consist of two key components: the Modality Decoupling Module (MDM) and Modality-aware Metric Learning (MML). Specifically, MDM explicitly decomposes modality features into modality-shared and modality-specific representations, enabling effective retrieval in both modality-aligned and mismatched scenarios. MML, a tailored metric learning strategy, further enhances feature discrimination and decoupling by exploiting distributional relationships between shared and specific modality features. Extensive experiments conducted on three challenging multi-modality ReID benchmarks (RGBNT201, RGBNT100, MSVR310) consistently demonstrate the superiority of MDL. MDReID achieves significant mAP improvements of 9.8\%, 3.0\%, and 11.5\% in modality-matched scenarios, and average gains of 3.4\%, 11.8\%, and 10.9\% in modality-mismatched scenarios, respectively.

MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification

Abstract

Real-world object re-identification (ReID) systems often face modality inconsistencies, where query and gallery images come from different sensors (e.g., RGB, NIR, TIR). However, most existing methods assume modality-matched conditions, which limits their robustness and scalability in practical applications. To address this challenge, we propose MDReID, a flexible any-to-any image-level ReID framework designed to operate under both modality-matched and modality-mismatched scenarios. MDReID builds on the insight that modality information can be decomposed into two components: modality-shared features that are predictable and transferable, and modality-specific features that capture unique, modalitydependent characteristics. To effectively leverage this, MDReID introduces two key components: the Modality Decoupling Learning (MDL) and Modality-aware Metric Learning (MML). Specifically, MDL explicitly decomposes modality features into modality-shared and modality-specific representations, enabling effective retrieval in both modality-aligned and mismatched scenarios. MML, a tailored metric learning strategy, further enforces orthogonality and complementarity between the two components to enhance discriminative power across modalities. Extensive experiments conducted on three challenging multi-modality ReID benchmarks (RGBNT201, RGBNT100, MSVR310) consistently demonstrate the superiority of MDReID. Notably, MDReID achieves significant mAP improvements of 9.8%, 3.0%, and 11.5% in general modality-matched scenarios, and average gains of 3.4%, 11.8%, and 10.9% in modality-mismatched scenarios, respectively. The code is available at: https://github.com/stone96123/MDReID.

1 Introduction

Object Re-Identification (ReID) focuses on identifying and retrieving specific objects across nonoverlapping camera views. Conventional object re-identification (ReID) methods [1, 2, 3, 4, 5, 6, 7] primarily rely on RGB images. However, RGB-based approaches often face significant limitations under challenging environmental conditions, such as poor illumination, shadows, and low image resolution. These adverse conditions frequently lead to the extraction of misleading features, thereby diminishing discriminative capability. To overcome these limitations, multi-modal ReID [8, 9, 10, 11, 12, 13], which integrates complementary information from multiple spectrums, has emerged as a promising approach. By leveraging the inherent strengths and complementary features of different modalities, multi-modal ReID significantly enhances feature representations, enabling more robust and accurate identification in complex real-world scenarios.

[RGB]

[RGB, NIR, TIR]

[RGB, TIR]

Modality-matched

Modality-mismatched

[RGB, TIR]

[NIR, TIR]

[RGB, TIR]

[NIR]

[RGB]

[NIR]

Uncertain

Query

(a) Examples of modality-(mis)matched ReID scenarios.

Modality-shared Feature

(b) The motivation of proposed MDReID.

Modality-specific Feature

MDReID

Uncertain

Gallery

[RGB]

[RGB, NIR, TIR]

[RGB, TIR]

2

1

II

II

II

11

11

I

I1

II

二

11

II

The introduction of multi-modal data has significantly enhanced the performance of ReID in challenging scenarios, thereby stimulating extensive research efforts in this field [8, 9, 10, 11, 12]. EDITOR [11] selects diverse tokens from Vision Transformer (ViT) to mitigate the impact of irrelevant backgrounds and narrow the gap between modalities, achieving strong detection performance. TOP-ReID [12] incorporates a cyclic token permutation module to aggregate multi-spectral features and a complementary reconstruction module to minimize the distribution gap across different image spectra. These innovations enable TOP-ReID to achieve robust performance under both modality-complete and modality-missing scenarios. Although these methods effectively leverage the characteristics of multi-modal data, they are developed based on a fundamental assumption that all modalities within the dataset are strictly aligned (as illustrated in Figure 1 (a) Modality-matched). Ideally, every camera would simultaneously provide RGB, NIR, and TIR modalities. However, due to practical constraints such as device differences and deployment diversity, achieving fully matched retrieval conditions is challenging in real-world scenarios. Therefore, it is crucial to develop a flexible image-level any-to-any ReID framework capable of effectively operating under both modality-matched and modality-mismatched scenarios (as illustrated in Figure 1 (a)).

To address this limitation, we propose MDReID, a flexible any-to-any ReID framework designed to support retrieval tasks involving arbitrary combinations of query and gallery modalities. The core challenge of this problem lies in learning effective representations that remain robust under both modality-matched and modality-mismatched conditions. A straightforward solution, as adopted by TOP-ReID [12], attempts to predict the missing modality representation from the available one, thereby enabling modality-aligned retrieval. However, as highlighted by RLE [14], predicting crossspectrum features solely from visual inputs constitutes an ill-posed problem. The modality-specific characteristics that are inherently unpredictable often lead to suboptimal learning. To this end, MDReID introduces a Modality-Decoupled Learning (MDL), which introduces a modality-shared and modality-specific token into the ViT architecture, leveraging multi-layer attention within the transformer to extract shared and specific features from multi-modal inputs. MDL explicitly disentangles the modality representations into two complementary components: a modality-shared component that captures the predictable cross-modal representation and a modality-specific component that preserves the unpredictable modality characteristics. This decoupling enables the model to better generalize across diverse modality combinations while maintaining the modality-specific cues.

Furthermore, to enhance the disentanglement between the two components, we propose a Modalityaware Metric Learning (MML) strategy. MML comprises two complementary objectives: a representation orthogonality loss (ROL) and a knowledge discrepancy loss (KDL). ROL is applied at the channel level to both promote the aggregation of modality-shared features across modalities and enforce orthogonality between shared and specific components, ensuring they capture distinct and non-overlapping information. KDL, on the other hand, encourages representational complementarity between shared and specific components by ensuring that the combined representation is more discriminative than either component alone.

To sum up, the main contributions of this paper are as follows:

We propose MDReID, a flexible any-to-any object re-identification framework that supports retrieval across arbitrary query-gallery modality combinations, addressing the practical limitations of strictly aligned multi-modal datasets.

We introduce a Modality-Decoupled Learning (MDL) strategy, coupled with a Modalityaware Metric Learning (MML) strategy. MDL explicitly disentangles modality-shared and modality-specific representations, while MML further strengthens this disentanglement by introducing a representation orthogonality loss and a knowledge discrepancy loss, encouraging the two components to encode distinct and complementary information.

Extensive experiments on multi-spectral object ReID datasets (RGBNT201, RGBNT100, MSVR310) demonstrate the superior adaptability and performance of the MDReID across diverse scenarios. MDReID achieves significant mAP improvements of 9.8%, 3.0%, and 11.5% in general modality-matched scenarios, and average gains of 3.4%, 11.8%, and 10.9% in modality-mismatched scenarios, respectively.

2 Related Work

Compared to the general RGB-to-RGB object re-identification [15, 16, 17, 18, 19, 20, 21], multimodal object re-identification (ReID) has demonstrated significant practical value, achieving widespread applications in cross-scenario systems [22, 23] (e.g., security surveillance, intelligent transportation, etc.). Current methods [24, 9, 25] primarily enhance re-identification accuracy by effectively leveraging the complementarity and integration of multi-modal features. For example, Zheng et al. [26] proposed the cross-directional consistency network (CCNet), which overcomes discrepancies in both modality and sample aspects, thereby achieving more effective multi-modal data collaboration. To address the modal-missing problem, Zheng et al. [27] proposed the DENet model, which incorporates a feature transformation module to recover information from missing modalities and a dynamic enhancement module to improve multi-modality representation. Li et al. [8] proposed HAMNet, which automatically fuses different spectral features using a specially designed heterogeneous score coherence loss. He et al. [28] proposed the GPFNet model, which utilizes graph learning to fuse multi-modal features.

Notably, models based on Vision Transformer (ViT) [29, 30, 10, 31, 32, 33] for object re-identification have achieved excellent results in recent years and gained widespread attention. Pan et al. [34] proposed the H-ViT model, which alleviates challenges caused by heterogeneous multi-modalities and reduces feature deviations arising from modal variations. Additionally, they introduced a progressively hybrid transformer (PHT [35]) that effectively fuses multi-modal complementary information through random hybrid augmentation and a feature hybrid mechanism. Crawford et al. [36] addressed the issue of modality laziness in multi-modal fusion by proposing the UniCat model based on a ViT architecture. By selecting diverse tokens from ViT to mitigate the impact of irrelevant backgrounds and narrow the gap between modalities, Zhang et al. [11] developed the EDITOR model. Wang et al. [12] designed the Top-ReID model, which achieves state-of-the-art accuracy by incorporating a cyclic token permutation module to aggregate multi-spectral features and a complementary reconstruction module to minimize the distribution gap across different image spectra.

Despite advancements in multi-modal object ReID, existing methods rely on modality-aligned conditions, limiting their adaptability to real-world scenarios. To address this, we introduce MDReID, which disentangles modality-shared and modality-specific features for effective retrieval across both matched and mismatched conditions. It incorporates Modality-Decoupled Learning (MDL) to refine feature separation and Modality-aware Metric Learning (MML) to enhance discrimination through orthogonality and knowledge discrepancy losses. As a result, MDReID enables robust object re-identification across diverse modality configurations.

3 Methodology

3.1 MDReID: Any-to-any Object ReID

In ideal multispectral person re-identification (ReID) settings, complete RGB, NIR, and TIR modalities are available for both query and gallery (RNT-to-RNT). However, real-world deployments often suffer from different modalities due to heterogeneous sensor availability and deployment

RGB

NIR

TIR

Linear Projection

[𝐼

𝑠𝑝

𝑅

, 𝐼 𝑖𝑚𝑔

𝑅

]

[𝐼

𝑠𝑝

𝑁

,

𝐼 𝑖𝑚𝑔

𝑁

]

[𝐼

𝑠𝑝

𝑇

, 𝐼 𝑖𝑚𝑔

𝑇

]

[𝐼

𝑠𝑝

𝑅

, 𝐼

𝑠ℎ

𝑅

, 𝐼 𝑖𝑚𝑔

𝑅

]

Shared

token

[𝐼

𝑠𝑝

𝑁 , 𝐼

𝑠ℎ

𝑁 , 𝐼 𝑖𝑚𝑔

𝑁

]

[𝐼

𝑠𝑝

𝑇

, 𝐼

𝑠ℎ

𝑇

, 𝐼 𝑖𝑚𝑔

𝑇

]

Vision Transformer Block

Vision Transformer Block

…

[𝐼

𝑠𝑝

𝑅

, 𝐼

𝑠ℎ

𝑅

]

[𝐼

𝑠𝑝

𝑁 , 𝐼

𝑠ℎ

𝑁 ]

[𝐼

𝑠𝑝

𝑇

, 𝐼

𝑠ℎ

𝑇

]

𝐼

𝑠𝑝

𝑇

𝐼

𝑠𝑝

𝑅

𝐼

𝑠𝑝

𝑁

[𝐼

𝑠ℎ

𝑅

,

𝐼

𝑠ℎ

𝑁 , 𝐼

𝑠ℎ

𝑇

]

Target cross-correlation

𝒅 𝒔𝒉

𝒑

𝒅𝒔𝒉

𝒏

[𝐼

𝑠𝑝

𝑅

, 𝐼

𝑠𝑝

𝑁 , 𝐼

𝑠𝑝

𝑇

, 𝐼

𝑠ℎ

𝑅

, 𝐼

𝑠ℎ

𝑁

, 𝐼

𝑠ℎ

𝑇

]

Representation Orthogonality Loss (ROL)

Knowledge Discrepancy Loss (KDL)

Anchor

Positive

Negative

Target:

𝒅𝒔𝒑+𝒔𝒉

𝒏

> 𝒅𝒔𝒑

𝒏

,

𝒅𝒔𝒑

𝒏

𝒅 𝒔𝒑+𝒔𝒉

𝒑

< 𝒅 𝒔𝒑

𝒑

,

𝒅 𝒔𝒉

𝒑

(a) Shared Feature (

𝐼

𝑠ℎ

)

𝒅𝒔𝒑

𝒏

𝒅𝒔𝒑

𝒑

(b) Specific Feature (

𝐼

𝑠𝑝

)

𝒅 𝒔𝒑+𝒔𝒉

𝒑

𝒅𝒔𝒑+𝒔𝒉

𝒏

(c) Combined Feature (

[𝐼

𝑠𝑝

, 𝐼

𝑠ℎ

]

)

constraints. To address this fundamental challenge, we propose MDReID, a flexible, any-to-any ReID framework that supports arbitrary combinations of input and output modalities. As illustrated in Fig. 2, MDReID adopts a Vision Transformer (ViT) backbone and integrates two core components: Modality-Decoupled Learning (MDL) and Modality-Aware Metric Learning (MML). MDL explicitly disentangles each modality's representation into modality-shared features (crucial for cross-modality retrieval) and modality-specific features (essential for preserving discriminative cues in matched scenarios). MML further enhances this decoupling by enforcing metric-level consistency across modalities. This design enables robust and flexible object re-identification in diverse and modality-incomplete environments, significantly advancing the applicability of ReID in practical settings.

3.2 Modality Decoupled Learning

Generally, with a ViT as the backbone, we process input images from different modalities ( M ∈ { R,N,T } , corresponding to RGB, NIR, and TIR). Standard ViT procedure involves segmenting the image into non-overlapping patches, which are then linearly projected into patch embeddings as:

To explicitly model and decouple modality characteristics crucial for cross-modal ReID, we deviate from the standard use of a single [CLS] token. Instead, we prepend two distinct learnable tokens to the patch embedding sequence for each modality: a modality-specific token and a modality-shared token. This augmented sequence is then processed by the ViT encoder to get:

where I M sp denotes the modality-specific features, I M sh denotes the modality-shared features, and I e M represents the encoded features.

Leveraging these decoupled features from different modalities ( I R sp , I N sp , I T sp , I R sh , I N sh , I T sh ), we construct a unified representation for each sample using a fixed-dimensional structure and an associated availability mask. This allows consistent handling across different modality presence scenarios. Specifically, we define a potential full feature vector, v full , by concatenating all possible specific and shared components, assuming all three modalities (RGB, NIR, TIR) are present:

Correspondingly, a binary availability mask, mask full , indicates the presence of each component:

where '1' signifies that the feature component at that position is present and valid, and '0' (as introduced below) signifies absence or invalidity.

The actual feature vector v and mask mask used for any given input sample are derived from this full structure based on the modalities available for that specific sample. If a modality M ∈ { R,N,T } is missing for a sample, the corresponding modality-specific feature I M sp and the modality-shared feature I M sh in the vector v are replaced with a zero vector 0 . Also, the entries in the mask mask corresponding to these zeroed-out features ( I M sp and I M sh ) are set to 0.

For example, if the TIR modality (T) is missing for a sample, its feature vector v and mask mask would be:

This structured representation ( v and mask ) is used consistently across all scenarios, including both modality-matched and modality-mismatched retrieval. In modality-matched scenarios ( e.g. , RN-to-RN), both the query and gallery samples will have the same modalities present (or absent), resulting in identical structures for their masks ( e.g. , [1, 1, 0, 1, 1, 0] for both if T is missing). In modality-mismatched scenarios ( e.g. , R-to-N), the query and gallery samples will inherently have different available modalities. Their respective v and mask representations are generated according to the rules above based on their own available data. For instance, the RGB query sample would have v q = [ I R sp , 0 , 0 , I R sh , 0 , 0 ] and mask q = [1 , 0 , 0 , 1 , 0 , 0] . And the NIR gallery sample would have v g = [ 0 , I N sp , 0 , 0 , I N sh , 0 ] and mask g = [0 , 1 , 0 , 0 , 1 , 0] .

The subsequent retrieval process utilizes both the feature vectors ( v q , v g ) and their corresponding availability masks ( mask q , mask g ) to compute a similarity score that respects the decoupled nature of the features and handles missing modalities robustly. We define the total similarity, Sim ( v q , v g ) , as the combine of two components, i.e. , the modality-specific similarity Sim sp and the modalityshared similarity Sim sh . The modality-specific similarity compares modality-specific features between identical modalities only. The similarity is calculated as the sum of dot products between corresponding specific features, masked for joint availability:

(6)

The modality-shared similarity computes the similarity between all pairs of available shared features across modalities. Define V sh,q be a matrix with shared features I R sh,q , I N sh,q , I T sh,q (or zero vectors if unavailable) as columns, and similarly for V sh,g , mask sh,q and mask sh,g . We first compute the matrix of all pairwise shared feature similarities:

This results in a 3 × 3 matrix where ( S sh ) ij is the dot product ( I i sh,q ) T ( I j sh,g ) for i, j ∈ { R,N,T } . To account for missing modalities, we define a pairwise availability mask matrix with the outer product of the shared masks:

where ( M sh _ pair ) ij is 1 if and only if both the i -th shared feature of the query and the j -th shared feature of the gallery are present. The total shared similarity is the sum of all valid pairwise similarities, obtained by Hadamard product of S sh and M sh _ pair :

Finally, the total similarity score is:

This approach, based on a fixed-size zero-padded feature vector and an explicit availability mask, provides a flexible and robust mechanism for handling arbitrary combinations of modality presence and absence, effectively adapting to all modality-matched and mismatched object ReID scenarios addressed by our method.

3.3 Modality-aware Metric Learning

After MDL, the features are separated into modality-specific and modality-shared components. To achieve more complete and thorough feature decoupling, we propose a modality-aware metric learning approach. Note that, during the training phase, we assume all the modalities for the samples are available. The core objective of this metric learning is twofold: 1) Enhance Shared Feature Consistency : Promote high similarity between modality-shared features irrespective of their originating modality, which is crucial to bridge the modality gap in mismatched retrieval scenarios. 2) Reinforce Specific Feature Purity : Ensure modality-specific features are distinct from each other and also orthogonal to all modality-shared features. This preserves unique modality characteristics and prevents leakage between specific and shared representations.

To achieve these goals simultaneously, we define a representation orthogonality loss, L ROL , that minimizes the discrepancy between the computed pairwise similarities of the decoupled features and a predefined target similarity structure. Within our framework, we operate on the feature vector v and its availability mask mask generated for each sample, as defined previously. We compute the 6 × 6 pairwise similarity matrix V sim based on the L 2 -normalized vector v :

We then define the ideal target similarity matrix as:

Finally, the loss L ROL is then formulated as the sum of squared errors between the computed similarities V sim and the target similarities A , masked by M pair :

Additionally, the concept of triplet loss is incorporated to further enhance the model's feature decoupling capability. In other words, the goal of feature decoupling is to achieve a synergistic effect and prevent information from collapsing into modality-specific representations. To this end, as shown in Fig. 2, we introduce a knowledge discrepancy loss (KDL) to enforce the complementarity between shared and modality-specific features, ensuring that their combination yields better retrieval performance than using either alone. Specifically, within a batch of samples, given an anchor a , the positive samples P a denotes the samples with the same label as a while the negative samples N a denotes the opposite samples. The distances using specific feature from the anchor a to the batch of positive or negative samples are computed as follows:

where S indicates the features used (modality-specific one, modality-shared one, or the combined one). Note that we omit the superscript representing the modality for simplicity, considering the modalities are completed, and the features of RGB, NIR, TIR modalities are concatenated to calculate the distance. Then, for the combination of shared and modality-specific features, the maximum positive sample distance should be as small as possible compared to using a single type of feature, and the minimum negative sample distance should be as large as possible compared to using a single type of feature. As shown in Fig. 2, this formulation can be described as: max( d sp + sh ( a, p )) > max( d sp ( a, p )) , max( d sh ( a, p )) and min( d sp + sh ( a, n )) < min( d sp ( a, n )) , min( d sh ( a, n )) . Therefore, we define the knowledge discrepancy loss L KDL as:

It is worth noting that detach operations are applied to d sp ( a, n ) and d RNT sh ( a, n ) to avoid gradient computation. Thus, minimizing D p reduces the farthest positive sample distance for the combined shared and modality-specific features, while minimizing ∥ D n -1 ∥ 1 increases the nearest negative sample distance. Consequently, improving the retrieval capability of the combined features enhances the performance of model. In summary, the loss function for modality-aware metric learning is defined as follows:

3.4 Objective Function

As illustrated in Fig. 2, our model incorporates two primary loss functions: one for the ViT backbone and another for modality-aware metric learning. For the ViT backbone, we adopt label smoothing cross-entropy loss L ce and triplet loss L tri following previous research to optimize the representation. In summary, the total loss function is as follows:

4 Experiments

4.1 Datasets and Evaluation Protocols

We evaluate the proposed MDReID framework on three publicly available datasets spanning both person (RGBNT201 [24]) and vehicle re-identification (RGBNT100 [8] and MSVR310 [26]) tasks. For evaluating modality mismatch scenarios, we focus on representative cross-modal configurations, including RT-to-NT, RT-to-N, R-to-N, and R-to-NT. Here, NIR is the second most widely used modality in surveillance systems after RGB, and RGB-to-NIR research continues to attract significant interest. Therefore, we evaluate our method in four R-to-N-based application scenarios. Performance is measured using mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at ranks 1, 5, and 10 (R-1, R-5, R-10). We also report key complexity indicators, including the number of trainable parameters and floating point operations (FLOPs), to quantify the efficiency of the model.

RGBNT201 [24] is a multi-modality person re-identification dataset designed to overcome the limitations of single-modal imaging in challenging surveillance scenarios. It was captured on a university campus using four non-overlapping camera views with a synchronized triple-camera system that simultaneously records RGB, near-infrared (NIR), and thermal-infrared (TIR) images. The dataset contains 201 identities, each represented by at least 20 non-adjacent image triplets, resulting in 4,787 images per modality. The collection covers a range of real-world conditions, including severe illumination changes, occlusion, viewpoint variations, and background clutter. The dataset is split into 141 identities for training, 30 for validation, and 30 for testing, with the entire test set serving as the gallery and 10 records per identity sampled as probes. RGBNT201 thus provides a robust benchmark for evaluating multi-modal fusion strategies and addressing missing-modality issues in person re-identification.

RGBNT100 [8] contains 17,250 spatially aligned image triples from 100 vehicles captured under uniform conditions. Each triple includes RGB, NIR, and TIR images. Fifty vehicles (8,675 triples) are used for training, and the remaining 50 vehicles (8,575 triples) form the testing/gallery set, with 1,715 triples randomly selected as queries. The RGB and NIR images are recorded at a resolution of 1920 × 1080, while the TIR images are captured at 640 × 480, all at a consistent frame rate of 25 fps. TIR images provide thermal information that is robust to illumination changes, further enhancing the multi-spectral data for challenging vehicle re-identification scenarios.

MSVR310 [26] is a high-quality multi-spectral vehicle re-identification benchmark captured under diverse, challenging conditions. It contains 2,087 triplet samples (6,261 images in total) from 310 vehicles. Each sample includes spatially aligned images from three modalities: RGB, NIR, and TIR. RGB images are captured by a 360 D866 camera during the day and by a Mi8 mobile phone at night, NIR images are obtained with the 360 D866 in near-infrared mode, and TIR images are recorded with a FLIR SC620 camera at 640×480 resolution. The number of samples per vehicle ranges from 2 to 20, and time labels are provided for cross-time matching. The dataset is divided into 1,032 samples from 155 vehicles for training and 1,055 samples from 155 vehicles for testing/gallery, with 591 samples

(from 52 vehicles) used as queries. MSVR310 provides a comprehensive platform for addressing intra-class appearance variations and modality differences to support robust vehicle re-identification.

4.2 Implementation Details

All experiments are implemented in PyTorch and conducted on a single NVIDIA RTX 4090 GPU with CUDA 12.5 and Python 3.8. We adopt the CLIP-Base [37] visual encoder as the backbone. Input images are resized to 256 × 128 for RGBNT201 and 128 × 256 for RGBNT100 and MSVR310. We employ random horizontal flipping, cropping, and erasing [38] for data augmentation. The model is optimized using Adam with a batch size of 64. The base learning rate is initialized at 3 . 5 × 10 -4 , while the visual encoder is fine-tuned with a reduced rate of 5 × 10 -6 . Training is performed for 50 epochs.

4.3 Comparison with State-of-the-Art Methods

Multi-spectral Object ReID (RNT-to-RNT) We compare our MDReID against a range of state-ofthe-art single-spectral and multi-spectral approaches on RGBNT201, RGBNT100, and MSVR310. For person ReID on RGBNT201, as shown in Table 1, the TOP-ReID [12] achieves 72.3% mAP, 76.6% R-1, 84.7% R-5, and 89.4% R-10. In comparison, our MDReID outperforms these results by 9.8%, 8.6%, 5.6%, and 3.2%, respectively, highlighting the effectiveness of our proposed MDReID. For vehicle re-ID, MDReID achieves the highest mAP on RGBNT100 (85.3%) and surpasses the nextbest approach on MSVR310 by a significant margin of 11.5% in mAP and 13.7% in R-1. These results clearly demonstrate the superior discriminative power and robustness of our modality-decoupled design in multi-spectral re-ID scenarios where all modalities are available.

Evaluation on Missing-spectral Scenarios. In practical applications, sensor limitations or environmental factors often lead to incomplete modality inputs. Following the setting of TOP-ReID [12], we evaluate the robustness of our method under all six possible missing-modality configurations across the RGB, NIR, and TIR channels using the RGBNT201 dataset. As shown in Table 2, our MDReID consistently outperforms the TOP-ReID, achieving an average improvement of 10.0% in mAP and 9.4% in Rank-1 accuracy across all missing-modality scenarios. These results demonstrate that our modality-decoupled framework remains highly effective even when partial modality information is absent, highlighting its strong generalization ability in incomplete multispectral settings.

Evaluation on Modality-mismatched Scenarios. To evaluate the generalizability of our approach under realistic deployment conditions, we benchmark MDReID across four representative modalitymismatched scenarios on three public datasets. We compare against two strong baselines, TOPReID [12] and EDITOR [11], by reproducing their results using official open-source implementations. As shown in Table 3, EDITOR is designed exclusively for modality-matched settings, which exhibits limited effectiveness when modalities differ across query and gallery. Although TOP-ReID attempts

to address missing modalities via reconstruction, its performance remains constrained due to the ill-posed nature of the reconstruction problem. In contrast, our MDReID framework consistently improves the average mAP by 3.4%, 11.8%, and 10.9% on the RGBNT201, RGBNT100, and MSVR310 datasets, respectively. In addition, we train four expert models on the RGBNT201 dataset for four specific scenarios. Results show that our method outperforms these experts by 9.2% on average in mAP and by 8.9% in R-1. Importantly, our single-model approach adapts to all four scenarios and offers greater flexibility. In summary, these results highlight the robustness and adaptability of MDReID in handling modality-mismatched ReID, affirming its effectiveness capable of supporting arbitrary modality combinations in both query and gallery inputs.

4.4 Ablation Study

To evaluate the effectiveness of each component in our proposed MDReID framework, we conduct comprehensive ablation studies on the RGBNT201 dataset. To ensure a thorough evaluation across both modality-matched and modality-mismatched scenarios, we report the average performance over 8 evaluation settings, including RNT-to-RNT, RT-to-RT, RT-to-NT, RT-to-N, R-to-N, and R-to-NT. The detailed results are presented in Table 4 (a). Note that configuration '1' corresponds to using a single classifier for all modalities, while configuration '3' corresponds to assigning a separate classifier to each modality. Ablation results show that, compared with one classifier, modality-specific classifiers effectively improve performance, increasing mean mAP by 13.4% and mean R-1 by 13.7%.

Modality Decoupled Learning Our base framework integrates a shared vision transformer, utilizing label smoothing cross-entropy along with triplet loss for optimization. To better address modality-mismatched object re-identification, we design a modality decoupling learning (MDL) that separates features into shared and specific representations. Experimental results demonstrate that this module boosts performance, with an average increase of 11.5% in mAP and 11.1% in Rank-1. This demonstrates that modality decoupling not only benefits re-identification in mismatched settings but also enhances retrieval performance in matched scenarios.

Modality-aware Metric Learning To further decouple shared and specific features, we design a modality-aware metric learning (MML) loss function. This loss consists of two components: the first aligns shared features across multiple modalities, while the second enhances the decoupling of features based on the aligned representation. Experimental results show that introducing ROL improves average mAP and Rank-1 by 1.8% and 2.7%, respectively. Similarly, introducing KDL increases mAP by 0.5% and Rank-1 by 1.7%. When both components are combined, the average mAP and Rank-1 improve by 3.8% and 4.1%, respectively. These results indicate that ROL significantly

(a) Ablation study. MDL, L ROL , and L KDL indicate the Modality Decoupled Learning, Representation Orthogonality Loss (ROL), and Knowledge Discrepancy Loss (KDL), respectively.

(b) Performance under different w 1 . The optimal performance achieves when w 1 is set to 1 . 5 .

(c) Performance under different w 2 . The optimal performance achieves when w 2 is set to 5 . 25 .

enhances re-identification accuracy in mismatch scenarios. Moreover, while KDL alone provides limited improvement, its combination with ROL effectively boosts overall model performance.

Modality Decoupled Learning We design the KDL objective to enforce a knowledge gap between specific and shared features. Without KDL, the model may degrade into relying only on shared features. Therefore, we ensure that combining specific and shared features yields higher retrieval performance than using either feature alone. To achieve this, we apply formulation 15 to impose distance constraints in the feature space based on the described relations. As shown in Table 4 (a), index 3 and 5, adding KDL improves mAP by 2.0% and Rank-1 by 1.5% across multiple scenarios. In summary, the MDReID framework, comprising the modality decoupled learning and modality-aware metric learning, achieves an average performance of 43.2% mAP, 42.3% Rank-1, 50.2% Rank-5, and 54.9% Rank-10 across various scenarios, including both modality-matched and modality-mismatched cases. Its modular design effectively decouples shared and specific features, enhancing object reidentification accuracy under multi-spectral conditions while maintaining robust adaptability across diverse modalities.

4.5 Discussions 4

Hyper-parameter settings of MDReID In Eq. (16), we introduce two hyperparameters w 1 and w 2 to balance the importance of ROL and KDL. Therefore, this part evaluates the performance under different hyperparameter settings and shows the results in Table 4. We first vary w 1 without KDL, observing that the model achieves optimal performance at w 1 = 1 . 5 , highlighting the importance of enforcing orthogonality between shared and specific components. Subsequently, with w 1 = 1 . 5 , we tune w 2 and find that the best results are obtained at w 2 = 5 . 25 , confirming the benefit of encouraging representational synergy and complementarity. These observations validate that a contribution from both ROL and KDL is crucial for maximizing retrieval accuracy.

5 Conclusion

In this paper, we propose MDReID, an image-level any-to-any object re-identification framework that supports retrieval across arbitrary query-gallery modality combinations. Our approach leverages Modality-Decoupled Learning (MDL) to explicitly disentangle modality-shared and modality-specific features, while Modality-aware Metric Learning (MML) refines the feature space to enhance discrimination. Extensive evaluations on the RGBNT201, RGBNT100, and MSVR310 datasets validate MDReID's superior performance in both modality-matched and modality-mismatched scenarios, demonstrating its practical potential for any-to-any ReID tasks.

Acknowledgements. This research was supported in part by the China Postdoctoral Science Foundation under Grant Number 2025M771584 and 2025T180439, and the National Natural Science Foundation of China under Grant 6250071985

References

Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformerbased object re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15013-15022, October 2021.

Wen Li, Cheng Zou, Meng Wang, Furong Xu, Jianan Zhao, Ruobing Zheng, Yuan Cheng, and Wei Chu. Dc-former: Diverse and compact transformer for person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1415-1423, 2023.

Guiwei Zhang, Yongfei Zhang, Tianyu Zhang, Bo Li, and Shiliang Pu. Pha: Patch-wise highfrequency augmentation for transformer-based person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14133-14142, 2023.

Siyuan Li, Li Sun, and Qingli Li. Clip-reid: exploiting vision-language model for image reidentification without concrete text labels. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 1405-1413, 2023.

Guan'an Wang, Xiaowen Huang, Jitao Sang, et al. Denoiserep: Denoising model for representation learning. Advances in Neural Information Processing Systems , 37:40032-40056, 2024.

Lei Tan, Pingyang Dai, Rongrong Ji, and Yongjian Wu. Dynamic prototype mask for occluded person re-identification. In Proceedings of the 30th ACM international conference on multimedia , pages 531-540, 2022.

Lei Tan, Pingyang Dai, Jie Chen, Liujuan Cao, Yongjian Wu, and Rongrong Ji. Partformer: Awakening latent diverse representation from vision transformer for object re-identification. arXiv preprint arXiv:2408.16684 , 2024.

Hongchao Li, Chenglong Li, Xianpeng Zhu, Aihua Zheng, and Bin Luo. Multi-spectral vehicle re-identification: A challenge. Proceedings of the AAAI Conference on Artificial Intelligence , 34(07):11345-11353, Apr. 2020.

Zi Wang, Chenglong Li, Aihua Zheng, Ran He, and Jin Tang. Interact, embed, and enlarge: Boosting modality-specific representations for multi-modal person re-identification. Proceedings of the AAAI Conference on Artificial Intelligence , 36(3):2633-2641, Jun. 2022.

Zi Wang, Huaibo Huang, Aihua Zheng, and Ran He. Heterogeneous test-time training for multimodal person re-identification. Proceedings of the AAAI Conference on Artificial Intelligence , 38(6):5850-5858, Mar. 2024.

Pingping Zhang, Yuhao Wang, Yang Liu, Zhengzheng Tu, and Huchuan Lu. Magic tokens: Select diverse tokens for multi-modal object re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17117-17126, June 2024.

Yuhao Wang, Xuehu Liu, Pingping Zhang, Hu Lu, Zhengzheng Tu, and Huchuan Lu. Topreid: Multi-spectral object re-identification with token permutation. Proceedings of the AAAI Conference on Artificial Intelligence , 38(6):5758-5766, Mar. 2024.

Yingying Feng, Jie Li, Chi Xie, Lei Tan, and Jiayi Ji. Multi-modal object re-identification via sparse mixture-of-experts. In Forty-second International Conference on Machine Learning .

Lei Tan, Yukang Zhang, Keke Han, Pingyang Dai, Yan Zhang, Yongjian Wu, and Rongrong Ji. Rle: A unified perspective of data augmentation for cross-spectral re-identification. Advances in Neural Information Processing Systems , 37:126977-126996, 2024.

Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu. Diverse part discovery: Occluded person re-identification with part-aware transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2898-2907, 2021.

Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. Deep learning for person re-identification: A survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(6):2872-2893, 2022.

Chuanming Wang, Yuxin Yang, Mengshi Qi, Huanhuan Zhang, and Huadong Ma. Towards efficient object re-identification with a novel cloud-edge collaborative framework. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 7600-7608, 2025.

Lei Tan, Jiaer Xia, Wenfeng Liu, Pingyang Dai, Yongjian Wu, and Liujuan Cao. Occluded person re-identification via saliency-guided patch transfer. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 5070-5078, 2024.

Yizhen Jia, Rong Quan, Yue Feng, Haiyan Chen, and Jie Qin. Doubly contrastive learning for source-free domain adaptive person search. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 3949-3957, 2025.

Ziniu Yin, Yanglin Feng, Ming Yan, Xiaomin Song, Dezhong Peng, and Xu Wang. Roda: Robust domain alignment for cross-domain retrieval against label noise. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 9535-9543, 2025.

Haoran Liu, Ying Ma, Ming Yan, Yingke Chen, Dezhong Peng, and Xu Wang. Dida: Disambiguated domain alignment for cross-domain retrieval with partial labels. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 3612-3620, 2024.

Chao Su, Huiming Zheng, Dezhong Peng, and Xu Wang. Dica: Disambiguated contrastive alignment for cross-modal retrieval with partial labels. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 20610-20618, 2025.

Yukang Zhang and Hanzi Wang. Diverse embedding expansion network and low-light crossmodality benchmark for visible-infrared person re-identification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2153-2162, 2023.

Aihua Zheng, Zi Wang, Zihan Chen, Chenglong Li, and Jin Tang. Robust multi-modality person re-identification. Proceedings of the AAAI Conference on Artificial Intelligence , 35(4):35293537, May 2021.

Jinbo Guo, Xiaojing Zhang, Zhengyi Liu, and Yuan Wang. Generative and attentive fusion for multi-spectral vehicle re-identification. In 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP) , pages 1565-1572, 2022.

Aihua Zheng, Xianpeng Zhu, Zhiqi Ma, Chenglong Li, Jin Tang, and Jixin Ma. Multi-spectral vehicle re-identification with cross-directional consistency network and a high-quality benchmark, 2023.

Aihua Zheng, Ziling He, Zi Wang, Chenglong Li, and Jin Tang. Dynamic enhancement network for partial multi-modality person re-identification, 2023.

Qiaolin He, Zefeng Lu, Zihan Wang, and Haifeng Hu. Graph-based progressive fusion network for multi-modality vehicle re-identification. IEEE Transactions on Intelligent Transportation Systems , 24(11):12431-12447, 2023.

Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng, David Crandall, and Bo Du. Transformer for object re-identification: A survey. International Journal of Computer Vision , pages 1-31, 2024.

Yingquan Wang, Pingping Zhang, Dong Wang, and Huchuan Lu. Other tokens matter: Exploring global and local features of vision transformers for object re-identification. Computer Vision and Image Understanding , 244:104030, 2024.

Zhi Yu, Zhiyong Huang, Mingyang Hou, Jiaming Pei, Yan Yan, Yushi Liu, and Daming Sun. Representation selective coupling via token sparsification for multi-spectral object reidentification. IEEE Transactions on Circuits and Systems for Video Technology , 2024.

Yanping Li, Yizhang Liu, Hongyun Zhang, Cairong Zhao, Zhihua Wei, and Duoqian Miao. Occlusion-aware transformer with second-order attention for person re-identification. IEEE Transactions on Image Processing , 2024.

Kuan Zhu, Haiyun Guo, Shiliang Zhang, Yaowei Wang, Jing Liu, Jinqiao Wang, and Ming Tang. Aaformer: Auto-aligned transformer for person re-identification. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Wenjie Pan, Hanxiao Wu, Jianqing Zhu, Huanqiang Zeng, and Xiaobin Zhu. H-vit: Hybrid vision transformer for multi-modal vehicle re-identification. In Lu Fang, Daniel Povey, Guangtao Zhai, Tao Mei, and Ruiping Wang, editors, Artificial Intelligence , pages 255-267, Cham, 2022. Springer Nature Switzerland.

Wenjie Pan, Linhan Huang, Jianbao Liang, Lan Hong, and Jianqing Zhu. Progressively hybrid transformer for multi-modal vehicle re-identification. Sensors , 23(9), 2023.

Jennifer Crawford, Haoli Yin, Luke McDermott, and Daniel Cummings. Unicat: Crafting a stronger fusion baseline for multimodal re-identification, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. Proceedings of the AAAI Conference on Artificial Intelligence , 34(07):1300113008, Apr. 2020.

Xuelin Qian, Yanwei Fu, Yu-Gang Jiang, Tao Xiang, and Xiangyang Xue. Multi-scale deep learning architectures for person re-identification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , Oct 2017.

Guangyi Chen, Tianren Zhang, Jiwen Lu, and Jie Zhou. Deep meta metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Wei Li, Xiatian Zhu, and Shaogang Gong. Harmonious attention network for person reidentification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2019.

Xiaobin Chang, Timothy M. Hospedales, and Tao Xiang. Multi-level factorisation net for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

Jiajian Zhao, Yifan Zhao, Jia Li, Ke Yan, and Yonghong Tian. Heterogeneous relational complement for vehicle re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 205-214, October 2021.

Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. Omni-scale feature learning for person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Yongming Rao, Guangyi Chen, Jiwen Lu, and Jie Zhou. Counterfactual attention learning for fine-grained visual categorization and re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1025-1034, October 2021.

Haoli Yin, Jiayao Li, Eva Schiller, Luke McDermott, and Daniel Cummings. Graft: Gradual fusion transformer for multimodal re-identification, 2023.

Yukang Zhang and Hanzi Wang. Diverse embedding expansion network and low-light crossmodality benchmark for visible-infrared person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21532162, June 2023.

A Supplemental Material

A.1 Analysis of model computational complexity

Table 5 compares the computational cost of our method with recent state-of-the-art approaches. MDReID achieves the most efficient design, requiring only 57.2M parameters and 22.3G FLOPs, which is significantly lower than HTT (85.6M / 33.1G), EDITOR (117.5M / 38.6G), and TOP-ReID (278.2M / 34.5G). Despite its efficient architecture, MDReID consistently outperforms these heavier models in accuracy, demonstrating a superior trade-off between performance and efficiency. This highlights the effectiveness of our MDL and MML designs, which provide robust alignment in all kinds of query-gallery combinations without incurring large computational overhead. Furthermore, we evaluate the total feature generation and retrieval times of different models on the RGBNT201 test set under the RNT-to-RNT, RT-to-NT, and R-to-N scenarios. Theoretically, our method reduces retrieval cost compared to Top-ReID. For example, in the RT-to-NT scenario, we only perform T-to-T (specific features) and R-to-T (shared features) matching, whereas Top-ReID first generates missing-modality features before conducting a full RNT-to-RNT matching. Experimental results confirm this, showing that our method incurs no additional time and generates features faster.

A.2 Analysis of specific and shared features

In MDReID, both shared and modality-specific features are essential. For the RT-to-NT task, using only shared features overlooks the unique characteristics of the T modality, while using only modalityspecific features prevents valid R-to-N matching. To quantify this, we evaluate RT-to-NT performance under three settings: shared features only, modality-specific features only, and both combined. As shown in Table 6, combining shared and modality-specific features yields the best performance.

A.3 Comparison with cross-modality method

We compare our method with the cross-modality object re-identification model DEEN [50] on the RGBNT201 dataset under the R-to-N scenario. Table 7 shows that even when DEEN is specially trained for R-to-N, its mean precision is 3.5% lower than ours. Notably, our model demonstrates stronger generalization across diverse application scenarios.

A.4 Visualization

To better illustrate the impact of different components on feature disentanglement, we visualize the learned modality-specific and modality-shared features across RGB, NIR, and TIR using t-SNE, as shown in Figure 3. Notably, the baseline method does not use MDL, so it does not separate specific and shared features and only outputs fused features, hence no corresponding visualization.

(a) MDReID (MDL)

(b) MDReID (MDL + ROL)

(c) MDReID (MDL + KDL)

(d) MDReID (MDL + ROL + KDL)

Specifically, in (a), with only MDL applied, the features exhibit noticeable overlap, and the separation between modality-specific and shared components remains unclear. In (c), the addition of KDL slightly improves the clustering and separation of shared and specific features. In contrast, (b) shows that introducing ROL significantly enhances feature orthogonality, forming clearer boundaries between shared and modality-specific clusters. Finally, (d) demonstrates that the combination of both ROL and KDL yields the most structured and disentangled feature space, with shared features tightly clustered and well-separated from their modality-specific counterparts. These results confirm the complementary roles of ROL and KDL in refining the representation space.

A.5 Limitations and Broader Impact

Although our work is the first to explore object re-identification under the uncertain query-gallery modality setting and achieves significant performance improvements, the current accuracy is still insufficient for deployment in real-world applications. Furthermore, existing datasets are limited in both scale and modality diversity, making it unclear how well MDReID would generalize to larger and more complex datasets involving a broader range of modality types. Nevertheless, MDReID represents the first effort to address the any-to-any object re-identification task, marking a significant step toward this emerging direction. We believe our work will inspire new perspectives and challenges in the re-identification community.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction of this paper accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of this work in Section A.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have introduced all the details in our theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper has provided detailed information for reproducing the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at: https://github.com/stone96123/MDReID.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper has specified all the training and test details, such as dataset and hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This paper does not provide error bars due to the limited resources. Also, previous methods do not provide error bars either.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This paper has provided sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper meets the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper has no societal impact to the best of our knowledge.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks since it does not release data or models that have risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper has properly cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper only uses LLMs for writing and editing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MDReID, a framework for object re-identification across that allows any query and gallery combinations from RGB, NIR, and TIR images. Unlike prior work that focuses on fixed modality pairs, MDReID handles arbitrary modality combinations through two key components: Modality-Decoupled Learning (MDL), which separates features into modality-shared and modality-specific parts, and Modality-aware Metric Learning (MML), which ensures these parts are complementary via orthogonality and discrepancy constraints. This design enables robust matching under varying or missing modalities. Extensive experiments on RGBNT201, RGBNT100, and MSVR310 show that MDReID achieves state-of-the-art performance with significantly fewer parameters and lower computation cost.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this paper are quite clear. The proposed any-to-any object re-identification task is both novel and compelling. Existing works typically split modality-aligned and cross-modal retrieval into two separate problems. I do not believe this is a desirable direction, as it fragments the solution space and limits generalization. In contrast, this paper is the first to support arbitrary query-gallery modality combinations within a unified framework, which I consider a significant contribution.\n\nOn the downside, the overall methodology appears relatively simple. While this simplicity may be an advantage from an application standpoint, the idea of feature decoupling itself is not particularly novel within the ReID literature. \n\nNevertheless, I believe the paper’s contribution outweighs this concern. Its exploration of the any-to-any setting is timely and has the potential to make a notable impact on the re-identification community.\n\nQUESTIONS:\n1. This paper appears to be the only one among all compared methods that adopts CLIP as the backbone. It would be helpful to clarify whether this choice contributes additional performance gains.\n\n2. The discussion on KDL may be somewhat insufficient. While the authors have dedicated adequate space to describing how KDL is implemented, they have not clearly articulated how it contributes to the model’s performance or why it is designed in its current form. I encourage the authors to elaborate on the underlying design rationale of KDL in their rebuttal. \n\n3. I believe that using a classifier to explicitly distinguish between different modalities is a simple yet effective strategy. However, it appears that the authors did not adopt this approach in the current work. I would suggest including a comparison to assess whether incorporating such a strategy could lead to further performance improvements.\n\n4. In Table 2, the performance of TOP-ReID appears to be mistakenly highlighted in bold.\n\nLIMITATIONS:\nYes, this work has addressed their limitations in ‘Supplemental Material’.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MDReID, a general-purpose object Re-ID framework designed for modality-mismatched and modality-matched scenarios. It addresses the practical limitation of existing methods that assume consistent modalities. MDReID consists of two key modules: MDL decomposes features into modality-shared and modality-specific components. MML enhances feature discrimination via orthogonality and complementarity. The method is validated on three multi-modality Re-ID benchmarks and shows significant improvements over baselines in both modality-matched and mismatched settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper clearly identifies the modality inconsistency issue in real-world ReID systems and proposes a targeted solution.\nThis paper Introduces a dual mechanism of modality decoupling (MDL) and metric learning (MML), with a well-structured design.\nThe performance improvements are validated across multiple benchmark datasets, supported by strong empirical evidence.\nWeaknesses:\n1. Lack of a Clear Classification Framework in Related Work: The related work lacks a clear technical taxonomy. The current version lists existing methods but does not organize them by core challenges (e.g., Conventional multi-modal methods,ViT-based, missing-modality methods). It also lacks analysis of each method's key ideas and limitations.\n2. Redundant or Delayed Definition of Technical Terms (e.g., ViT): Several technical terms (e.g., ViT) are either repeated or introduced only after they have already been used multiple times.\n3. Binary Availability Mask Explanation is Overly Verbose: The explanation of the binary availability mask spans over 0.5 pages. The logic of zero-padding can be simplified or moved to the supplementary material. The current version distracts from the main technical innovation.\n4. Knowledge Discrepancy Loss Definition (Line 224) is Unclear: The structure of the knowledge discrepancy loss is hard to interpret.\nAnyway, this is a good paper with a new task and reasonable solution, although there are some minor flaws in the details.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMDReID is a novel framework for multi-modal object re-identification that supports any-to-any retrieval across both modality-matched and modality-mismatched scenarios. To tackle the challenges of modality discrepancy, MDReID introduces a Modality-Decoupled Learning (MDL) strategy to explicitly separate modality-shared and modality-specific representations. In addition, a Modality-aware Metric Learning (MML) strategy is proposed, which enhances this disentanglement through a representation orthogonality loss and a knowledge discrepancy loss, encouraging the two components to capture distinct yet complementary information. The framework is comprehensively evaluated under both modality-aligned and modality-misaligned settings to demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n-\tThis paper introduces the novel task of any-to-any multi-modal object re-identification, which supports flexible retrieval under arbitrary query-gallery combinations across three different spectral modalities.\n-\tCompared to the previous works, the proposed method achieves a significant improvement in all the testing protocols.\nCons:\n-\tThe any-to-any setting is also highly related to the cross-modality re-identification task. However, this paper may ignore this part.  \n-\tSome details should be further explained. Some important details require further clarification. For example, what is the computational cost of the baseline method? Additionally, how the 4 testing scenarios under modality-mismatched conditions were selected?\n\nQUESTIONS:\n1.\tThe any-to-any setting is highly related to both modality-matched object re-identification and cross-modality object re-identification. However, this work only considers the modality-matched side. Therefore, comparison and discussion with cross-modality methods will also be useful.\n2.\tDuring the retrieval phase. From the paper, the author considers the shared representations equally, whether the shared feature from the same modality. From the visualization results, the shared feature.\n3.\tFigure 3 lacks the visualization result of the baseline method. It will make audience hard to get the improvement of MDL. The same issue applies to Table 5.\n4.\tIt would be helpful to clarify how the 4 testing scenarios under modality-mismatched conditions were selected.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MDReID framework that addresses any-to-any multi-modal object ReID task, enabling robust re-identification even when query and gallery images come from different modalities. The proposed framework is built on the core idea of decoupling image features into two components: modality-shared features that are consistent across different modalities, and modality-specific features that preserve unique characteristics. This is achieved through two main contributions: Modality-Decoupled Learning (MDL), which uses dedicated tokens in a Vision Transformer to disentangle the features, and Modality-aware Metric Learning (MML), which applies specialized losses to refine the separation. Specifically, MDL extracts modality-shared and -specific features and then compute the robust similarity based on extracted features and availability masks. MML promotes further disentangled & discriminative features through a representation orthogonality loss (ROL) and a knowledge discrepancy loss (KDL). Experiments on three benchmarks show the proposed method approach outperforms state-of-the-art methods, particularly in modality-mismatched scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The tackled problem, any-to-any multi-modal object ReID, is interesting considering the practicality for real-world applications.  \n2. Overall, the key idea of the paper is reasonable and the proposed method to achieve this is logical and well-designed (especially ROL & KDL).\n\n**Weaknesses**\n\n1. The comparison to state-of-the-art methods seems potentially unfair due to the use of a \"modality availability mask\" during inference. This mask provides explicit information about the modality scenario for both the query and gallery, an advantage not available to the baseline methods. This calls into question whether the reported performance gain stems from the novelty of the modality-decoupling approach itself, or simply from leveraging this additional information. To provide a fairer assessment, the paper should compare MDReID against an alternative strategy that also utilizes this mask. For instance, one could train an ensemble of separate \"expert\" models, each specialized for a specific modality combination (e.g., an R-to-N model, an RT-to-NT model). At inference time, the availability mask could then be used to select and deploy the appropriate expert model for the given scenario. A comparison against such a baseline would better isolate the true contribution of the unified MDReID framework over a simpler, scenario-specific approach.\n2. While the paper compares computational costs in terms of parameters and FLOPs, it omits the retrieval cost (i.e., search time), which is a critical metric for ReID systems. The proposed similarity calculation is more complex than a standard feature dot product. Therefore, a comparison of the actual search time against baselines is necessary to fully evaluate the practical efficiency of the proposed method in large-scale gallery settings.\n3. To provide stronger intuition for the proposed feature decoupling, a more detailed ablation study is needed. The paper should analyze the individual contributions of the modality-shared and modality-specific features. For instance, what is the performance when using only the shared features or only the specific features during inference? This analysis should be conducted for both modality-matched and mismatched scenarios to clearly demonstrate the respective roles of the decoupled components. Also, the ablation studies of these individual features with different loss functions would be also helpful.\n4. (minor) Notations throughout the method is confusing and lacks rigor.\n\nQUESTIONS:\nPlease address my concerns, especially weaknesses 1, 2, and 3.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}