Multi-Modal Object Re-identification via Sparse Mixture-of-Experts
Authors: Yingying Feng
Jie Li
Chi Xie
Lei Tan
Jiayi Ji
Venue: ICML 2025 poster (2025)
Keywords: Object Re-identification
Multi-Modal Learning
Mixture-of-Experts
Primary area: applications->computer_vision

Abstract: We present MFRNet, a novel network for multi-modal object re-identification that integrates multi-modal data features to effectively retrieve specific objects across different modalities. Current methods suffer from two principal limitations: (1) insufficient interaction between pixel-level semantic features across modalities, and (2) difficulty in balancing modality-shared and modality-specific features within a unified architecture. To address these challenges, our network introduces two core components. First, the Feature Fusion Module (FFM) enables fine-grained pixel-level feature generation and flexible cross-modal interaction. Second, the Feature Representation Module (FRM) efficiently extracts and combines modality-specific and modality-shared features, achieving strong discriminative ability with minimal parameter overhead. Extensive experiments on three challenging public datasets (RGBNT201, RGBNT100, and MSVR310) demonstrate the superiority of our approach in terms of both accuracy and efficiency, with 8.4% mAP and 6.9% accuracy improved in RGBNT201 with negligible additional parameters.

Multi-Modal Object Re-Identification via Sparse Mixture-of-Experts
Yingying Feng" 1 Jie Li Chi Xie
Lei Tan Jiayi Ji
Abstract
(a) one-sfreau network
(b) Carrent dual-stram network
We present MFRNet, a novel network for multi- modal object re-identification that integrates multi-modal data features to effectively retrieve specific objects across different modalities. Cur- rent methods suffer from two principal limita- tions: (I) insufficient interaction between pixel- level semantic features across modalities, and (2) difficulty in balancing modality-shared and modality-specitfe features within a unified archi- tecture. To address these challenges, our net- work introduces two core components. First, the Feature Fusion Module (FFM) enables fine- grained pixel-level feature generation and flex- ible modal interaction. Second, the Fea- ture Representation Module (FRM) efficiently extracts and combines modality-specifte and modality-shared features, achieving strong dis- criminative ability with minimal parameter over- head. Extensive experiments on three challeng- ing public datasets (RGBNT201, RGBNTI00. and MSVR310) demonstrate the superiority of our approach in terms of both accuracy and effi- ciency, with 8.4% mAP and 6.9% accuracy im- proved in RGBNT201 with negligible additional parameters. The code is available https: //github. stonmstoon96613 /MFRNet.
Viston Transfemet
Intoractton
TT ITTISS nater
Intornaction
Truns Transforme formee
(c) Our proposed MERNet
Intvraction
FFM
FRM
BE RE
Vilua Tranformnc
Interaction
RB
Figure I. Comparison between our proposed MFRNet and cur- rent mainstream structure. (a) and (b) show current main- stream one-stream and two-stream networks, respectively. Besides the limitation neglecting the pixel-level alignment characteris- tics of multi-modal images, current methods also face the challenge of balancing the modality-specciff and modality-shared represen- tation in unified network. (c) shows proposed MFRNet and 'RE' refers to the representation expert. MFRNet inherits the idea of MoE and extends it with multi-modal fusion (FFM) and repre- sentation (FRM), allowing it to achieve fine-grained interaction and efficient representation for multi-modal data.
2021; Liu et al., 2024; Tan et al., 2024b). These limitations frequently result in the extraction of misleading features and a subsequent loss discriminative information (Zheng et al., 2021; Wang et al., 2022). To address these challenges, multi-modal ReID (Shi et al., 2023; Lu et al., 2023; Shi al., 2024; Yang et al., 2024) has emerged a promis- ing solution, leveraging complementary information from various modalities enhance feature representation and achieve more acceptable identification ability in complex scenarios.
1. Introduction
Object Re-Identification (ReID) focuses on retrieving spe- eific object across non-overlapping camera views using known information. Traditionally, single-modal RelD relies heavily on RGB images, which face significant challenges under adverse conditions such as poor lighting, shadows, and low image resolutions (Li et al., 2020; Zhang et al.,
With the widespread application of Vision Transformer (AiT))Alexey. 2020; Ji et al., 2025), numerous ViT-based ReID methods have emerged from general-purpose (He et al., 2021; Tan et al., 2022; 2024a) to multi-modal see- narios (Zhang et al., 2024a; Wang et al., 2024c). However, as illustrated in Figure 1, current mainstream methods face two fundamental limitations: insufficient interaction and feature imbalance. The insufficient interaction manifests in existing approaches that mainly emphasize interaction
'Equal contribution 'School of Computer Science and Engi- neering. Northeastern University. Shenyang, China. "School of In- formatics, Xiamen University, Xiamen, China. 'Tongii University, Shanghai, China. 'National University Singapore, Singapore. Correspondence to: Lei Tan <lei.tan nus.edu.sg>>
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Expert
at high-level semantic feature spaces (Wang al., 2024a; Zhang et al., 2024a). These methods overlook the inher- ent pixel-to-pixel alignment characteristics in multi-modal images, resulting in inadequate fine-grained interaction be- tween multi-modal information. The feature imbalance stems from the difficulty of balancing mosality-shared and modality-specific features (Wang et al., 2022; Zhang et al., 2024a). A single parameter-sharing ViT backbone, though parameter-effiiient, struggles to preserve modality-specifie discriminative features. Alternatively, employing separate ViT backbones for different modalities can better preserve modal characteristics but introduce significant parameter redundancy and computational overhead.
aims achieve fine-grained interaction between multi- modal inputs, while the latter aims to achieve effi- cient and balanced feature extraction between modality- shared and modality-speciff representations.
Extensive experiments on three public multi-modal ob- ject ReID datasets including RGBNT201, RGBNT100, and MSVR310, verifying the superior performance of MFRNet.
2. Related Work
2.1. Multi-modal Person RelD
To address the insufficient interaction and feature imbalance challenges, we propose a Modality Fusion and Representa- tion Network (MFRNet). Our design incorporates mixture- of-experts (Jacobs et al., 1991) paradigm into both gen- erators and feature extractors, achieving pixel-level cross- modal interaction and balanced feature learning. MFRNet consists of two key components: Feature Fusion Module (FFM) with mixture-of- -ggeerrttss for fine-grained inter- action, and Feature Representation Module (FRM) with mixture of representation experts for balanced feature ex- traction. The FFM tackles insufficient interaction through its mixture-of- mixture--f-geenraato generators design. Inspired by RLE (Tan et al., 2024c) that cross-spectral transformation exhibits locally lin- car characteristics with surface-dependen variations, FFM employs multiple simple generators rather than complex unified structures. Through the mixture-of generators anism, diverse tokens spontaneously select the most suitable generator for different modalities, locations, and attribute to- kens. Additionally, FFM achieves fine-grained information exchange through weighted feature fusion, where fusion weights are learned via a generative network. The FRM addresses feature imbalance through a mixture of represen- tation experts structure. It dynamically routes tokens from different modalities to various experts, enabling efficient parameter utilization while preserving both modality-shared and modality-speciff features. Through dynamic expert activation, different experts in FRM adaptively focus on various shared visual attributes while maintaining necessary modality-spedee speciifc representations. Notably, this design en- ables our framework to handle missing modalities without specific architectural modifications or training procedures.
Human-centrr computer vision tasks (Shen & Tang, 2024; Shen et al., 2025) have long been a central focus within the research community. Unlike single-modal object re- identification (ReID), multi-modal person ReID includes more information and is suitable for a wider range of appli- cations. Existing methods (Park et al., 2021; Zheng et al., 2023; 2022; He et al., 2023; Tan et al., 2023; Zhang & Wang, 2023) for multi-modal object ReID mainly focus on how to effectively integrate information from multiple modali- ties and alleviate the cross-modal heterogeneous issue. For example, Zheng et al. (Zheng et al., 2021) utilize the com- plementary advantages of multiple modalities to propose PFNet for learning effective multi-modal features. To boost coddllty-speeciff representations, Wang al. (Wang et al., 2022) proposed the IEEE method, which includes important information exchange, feature enrichment, and intra-class discrepancy maximization mechanisms. To effectively uti- lize the relationship of modalities to reduce the gap between different modalities, Guo et al. (Guo et al., 2022) proposed the GAFNet model fuse multiple data sources. Addition- ally, the vision transformers (ViT) (Alexey, 2020; Radford et al., 2021; Pan et al., 2022; 2023; Crawford et al., 2023; Wang et al., 2024a) model has achieved good results in many fields, promoting the development of ViT-based multi- modal person ReID algorithms. Wang et al. (Wang et al., 2024b) explored the influence of global and local features of ViT and proposed the GLTraan model. Wang al. (Wang et al., 2024c) explored the domain traits of unlabeled test data and proposed heterogeneous test-time training frame- work based on ViT to improve generalization performance. Yu et al. (Yu et al., 2024) proposed the RSCNet model based on the idea of token sparsification in ViT to alleviate the multi-modal heterogeneity problem. Zhang et al. (Zhang et al., 2024a) proposed the EDITOR model, which selects diverse tokens from ViT to mitigate the effect of irrelevant backgrounds and reduce the gap between modalities.
mixture-of-generaatoss mech-
The contributions of this paper are summarized follows:
We propose a Modality Fusion and Representa- tion Network (MFRNet) for multi-modal object re- identification, which inherits the idea of a sparse mix- ture of experts and extends it with multi-modal fusion and representation.
Despite the good performance of these methods, they tend to employ high-level semantic multi-modal feature fusion, which overlooks the fine- mme-grained spatial alignment charac- teristics of multi-modal data. Additionally, they struggle
We introduce a Feature Fusion Module (FFM) and a Feature Representation Module (FRM). The former
2

---

Multi-Modal Object Re-Identiifcation via Sparse Mixture-of-Experts
to efficiently extract modality-shared and -ppelic multi-modal repnesentation within a unified network. There- fore, we propose MFRNet, which adopts a mixture of gen- alignment of
Mixture-of-Experts (MoE) adapt to diverse modal input at extremely low cost and performs joint optimization of modality-peecific and modality-shared features.
erator experts to leverage the pixel-to-pint multi-modal data and introduces a mixture of representation experts to adaptively extract modality-shared features while retaining modality-specifie features.
3.1. Feature Fusion Module
In general, a given image can be divided into patches of the same size and mapped to fixed feature dimension through patchify operations. Then, we have the image feature as:
2.2. Mixtures of Experts
The Mixture of Experts (MoE) (Jacobs et al., 1991) orig- inates from Ensemble Learning (Ganaie et al., 2022) and can effectively enhance model performance. Notably, af- ter Lepikhin et al. (Lepikhin et al., 2020) combined MoE with the transformer framework and proposed the Gshard model, it garnered significant attention across various fields, leading to the development of more models incorporating MoE technology (Chen & Wang, 2025; Dai et al., 2024; Gui et al., 2024; Lin et al., 2024; Liu et al., 2025; Mustafa et al., 2022; Li et al., 2025). Subsequently, to facilitate the implementation and flexible design of MoE, Hwang et al. (Hwang al., 2023) open-sourced the Tutel codebase. Additionally, to reduce training and inference costs, Chowd- hury et al. (Chowdhury et al., 2023) proposed pMoE, which can filter label-irrelevant patches and route similar class- discriminative patches to the same expert. Recently, many works have attempted to introduce MoE into ReID. Among them, Li et al. (Li al., 2023) proposed the MPC model, which leverages MoE to learn multiple pseudo-label spaces, thereby flexibly handling different variations. Xu et al. (Xu et al., 2022) proposed the META model based on MoE to better exploit the domain-invariant characteristics of multi- modal data. Kuang et al. (Kuang et al., 2024) proposed the MiKeCoCo model, which incorporates multiple experts with unique perspectives into CLIP and fully leverages high-level semantic knowledge for comprehensive feature representa- tion. Additionally, Li et al. (Li et al., 2022) explored an orthogonal direction for domain generalization research and proposed the GMoE model by combining ViT and MoE technologies. Based on the results of the aforementioned studies, we leveraged the advantages of MoE technology and utilized the Tutel framework to design and implement MoE techniques enhance model performance.
Is = (1171)) 13, ... , where M E R,,,T,
(1) where R. N, denote RGB, NIR, and TIR modalities respec- tively. Given that multi-modal images maintain pixel-level alignment, our feature fusion module (FFM) is designed to enable effective interaction and mutual enhancement be- tween different modality features. As illustrated by the RLE (Tan et al., 2024c) and AMML (Zhang et al., 2024b), the cross-spectral transformation can be considered as a local linear transformation, while the linear factor will be varied across different surfaces. It shows that the transfor- mation between different spectra is quite simple but variant. Therefore, inspired by the idea of the Mixture-of-exxprts strategy and the above observation, FFM attempts to com- bine multiple simple generators to adapt to diverse input tokens. In addition, it is interesting to find that this kind of fusion strategy of FFM also works well for modality- missing scenarios in modality-complementng way.
Specifically, as shown in Figure 2, taking the RGB feature In for example, it can be re-generated with NIR and TIR features as:
1 - \ In + W (IN) X IN (IN) + ((TT x (Ir),
(2)
where À is of information retained by the modality itself, w (.) and (-) generate the weighting coefficients for the features generated from NIR and TIR, respectively. And of (-) and T-{ represent the reconstructed RGB features from the NIR and TIR modality, respectively. The re-generated fea- tures of NIR and TIR are similar.
to determine the proportion
For the weight coefficient function w(-)', we define it as follows:
3. Methodology
As shown in Figure 2, the proposed MFRNet employs a CLIP pre-trained vanilla ViT-base as its basic backbone network. On this basis, our network comprises two princi- pal components, i.e., the feature fusion module (FFM) and the feature representation module (FRM). The FFM intro- duces the pixel-level feature generation for flexible cross- modal interaction, which is also suitable for modal-missing scenarios, while the FRM inherits the idea of the sparse
wc( = fc(AugPool(TT) Raxd - Rixd= RIXI (3)
where 1 E Raxd represents image feature from spe- cific modality, AugPool represents global average pool- ing, and fc represents a linear layer. To normalize the weight coefficients, we apply the Softmax operation on the weights of each reconstructed modality feature like
We omit the subscript for modality for simplification.
3

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Expert
MMN n0n
Malti Multi-modal modd-omdaall Feature Fusion
Vision Transformer Blocks
RE-0
TIR
Multi Mulfi-modal miii-modall Feature
Liaeur Projection
Laux
Representation
lice
B
+ Ltri
RGB
Figure 2. Overall architecture of MFRNet. MrRNet is built from the basic Transformer blocks inherited from vanilla VIT, with two new modules (feature fusion module and feature representation Module) added to adapt to multi-modal object re-identification tasks. In the blue box, the Feature Fusion Module (FFM) employs multiple simple generators adaptively provide lineggraiedd interaction information, while in the green box, the Feature Representation Module (FRM) employs diverse representation experts to extract and combine modality-speeifff and modality-shared features. Here, GE refers the generation experts in FFM, while RE represents the representation experts in FRM. Notably, the figure illustrates the NIR+RGB + TIR interaction, with the same interaction applying to NIR and RGB.
Softmar y (Tx), (Ix), tion is modality- and there are total 6 functions for 3 modalities.
(Tr)) The weight generation func-
scenarios in a nodality-complementin way. Specifically, for the modality missing scenarios during testing, we de- velop conditional reconstructed rules for modality M as:
For the feature generation function g()), as mentioned be- fore, the transformation between different modality images simple but variant. Compared to complex structures, sim- ple generators work well (Zhang et al., 2024b). Therefore, apply the mixture-of-expere (MoE) architecture (Li et al., 2022) to combine multiple simple generators. We define the simplest generation expert ( as:
AM EsEA w (Ts) x g(Ts) EsEA ESe (I, g(Is)
if M present if M missing,
(6)
where A denotes available modalities. For example, if RGB is missing, its features are generated based on NIR and TIR as follows: 1R = w (IN) x g(Ix) + (x()) x g(T).. Here, g(IN) denotes the features for RGB generated from NIR, g
(T( = conv2 (drop (cone1 (T))).
(4)
(T) g(Ir) represents the features for RGB generated from TIR, w (Tx) and (Trr are the respective weights.
Then we enhance it using sparse MoE technology. Specifi- cally, we categorize feature generation into two types based on the different modalities: {IN-RR II- IR-N) and {TT-N: /R-T, J- Since each of the three modalities can generate features for the other two modalities, we define two moe layers. Finally, the feature generation function is as follows:
3.2. Feature Representation Module
Existing approaches face fundamental challenges in bal- ancing modality-shared and modality-specifie information. While employing separate networks for each modality pre- serves modality-specific characteristics, this introduces sig- nificant structural and computational redundancy. Con- versely, using a fully shared single network for all modalities exclusively captures shared representations across modali- ties. Hence, we propose the Feature Representation Mod- ule (FRM), which integrates Mixture-of- architec- ture to preserve valuable modality-specifie information with parameter-efficiee implementation.
lemperts = {g/ (fe: (I)) for i E [1,
WF IWF
Syouter Softmarl
|W [WFI
(5)
Gate(X) =
g(I)= = Gate(T). Tempertss
where fc; refers to the linear layer for expert, Srowter represents cosine router which indicates routing scores for selecting experts. ||-|| indicate the L2 normalization, and t indicates the learnable temperature scaler. W/F is the weight of linear projection which aims to project the input token to low-rank subspace, while W.F is the expert embedding that transforms the feature to final scoring distribution. Note that k = represents only the highest-scoring expert is selected for feature generation each time.
With the reconstructed multi-modal image features T (1g 12,13) 17)), we use a vision transformer (ViT) to extract fundamental features. Within this module, we aim to capture feature representations within unified architectural framework.
S Specifically, we modify the residual attention layer based on RepAdapter as follows: To extract unique features for different modality data, we employ RepAdapter (Luo et al., 2023), which is designed to rapidly adapts to vision tasks by inserting adapter layers between key layers. Specifically,
As stated before, although do not have any specific training, we find that FFM can also work well for modality-missing

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Expere
the residual attention layer can be represented as:
defined the normalized routing weight corresponding expert summed over images
= X MH-Attn (LayerNorm(XX)) Y RAda (LayerNorm(Y));
Imp)(1b)
Σ IEL.
Srouter
(11)
where X and Z are the input and the output of the current layer, MH-Attt(-)) represents the multi-head attention layer, R RAda() (-) represents the RepAdapter layer. The formula of
The important loss Limp is given by the squared coefficient
of variation of the importance distribution over experts as:
(TD((mpp(T)) (T))
the RepAdapter layer is as:
Cimp = IMEANNImpll
(12)
RAda (Yinorm)= = Ymorm + conve (drop (conv A (iioom))) (8)
Meanwhile, the load loss Lload can be given as:
STD((oad(()) Liond (1n) (E (T))*
where Yinorm refers to the Y after the LayerNorm layer,
A and cone represent ID convolutions with kernel size of 1 and groups set I. Similarly, to better adapt to multi-modal data, we use Ado as the expert and enhance it by the MoE (Li et al., 2022) layer.
with loade = Σ1E1, Pe()), and Pe(T) = 1 $ 1h (WIIWe)e
(13)
Herein, the T/s refers the Krn largest entry after softmax while o indicates the cumulative distribution function of Gaussian distribution. Therefore, the whole auxiliary losses Lour is formulated as:
WRY Yioom WR || RynRYmnomm Yiorm Gale (Ynorm)) Topp (Stoaaer)
SAda
Stouter = Softmax(
(9)
Limp + Lioad
where 7 = 0.01 and k = 1. is important to note that since the number of modalities 3, the number of experts should be greater than 3. W/R is the weight of the linear projection, which aims project the input token low- rank subspace, while W is the expert embedding that transforms the feature to the final scoring distribution.
Lava
(14)
2
Finally, the total loss function of MFRNet given as fol- lows:
-=
LVVT + CViT + (C + Lmmo2 + LAda (15)
Based on the above derivation, using m to represent the MoE-enhanced RepAdapter, the modality-specific features are follows:
4. Experiments
4.1. Implementation
Z = Y Ada (tnorm).
(10)
Datasets. We evaluate our model performance three public multi-modal object RelD datasets. Specifically, RGBNT201 (Zheng et al., 2021) is a multi-modal person RelD dataset, which includes 4,787 aligned RGB, NIR, and TIR images from 201 identities. RGBNTION (Li et al., 2020) is large-scale
Here, the auxiliary loss for MoE is denoted as Ade.
3.3. Objective Function
vehicle ReID dataset compris- ing 17,250 image triples. It covers wide range of challeng- ing visual conditions, making it suitable for assessing the ro- bustness of vehicle RelD methods. MSVR310 (Zheng et al., 2022) is small-scale multi-modal vehicle ReID dataset that includes 2,087 high-quality image triples captured across diverse environments and time spans, providing complex visual challenges for vehicle RelD evaluation.
As shown in Figure 2, our model consists of three main loss functions: losses for the ViT backbone, auxiliary losses for the multi-modal feature completion module, and auxil- iary losses for the modality-specifie feature representation module. For the losses for the ViT backbone, we follow previous research by using label smoothing cross-entropy loss (Szegedy et al., 2016) and triplet loss (Hermans al., 2017) for optimizing the representation. Besides the basic losses, following the (Riquelme al., 2021; Li et al., 2022), define two auxiliary losses: important loss Cimp and load loss Liood to balance the frequency being activated for the expert. The important loss Cimp to encourage balanced usage of different experts across tokens and load loss Ltond to encourage balanced assignment across experts. Specifi- cally, the importance of expert e in batch of images Ja is
Evaluation protocols. To assess the performance of our method, we utilize the mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at Rank- K (K = 1,5, 10). These metrics are standard in this field and provide a comprehensive evaluation of our model's 's effec- tiveness. Additionally, we present the number of parameters and FLOPss to analyze the complexity of our model.
5

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Eppers
Table Performancc comparison on RGBNT201. The best and second results are in bold and underlined, respectively.
Table 2. Performance comparison on RGBNTT00 and MSVR310. The best and second results are in bold and underlined, respectively.
Methods
mAP
R-1
R-5
R-10
MUDeep (Qian et al., 2017)
23.8
19.7
33.1
44.3
RGBNT100
MSVR310
Methods
HACNN (Li et al., 2018)
21.3
19.0
34.1
42.8
mAP
R-1
MAP
R-1
Single
MLFN (Chang et al., 2018)
26.1
24.2
35.9
44.1
PCB (Sun al., 2018)
57.2
83.5
23.2
42.9
PCB (Sun et al., 2018)
32.8
28.1
37.4
46.9
MGN (Wang et al., 2018)
58.1
83.1
26.2
44.3
OSNet (Zhou et al., 2019)
25.4
22.3
35.1
44.7
DMML (Chen et al., 2019)
58.5
82.0
19.1
31.1
CAL (Rao et al., 2021)
27.6
24.3
36.5
45.7
Single
BoT (I.uo et al., 2019)
78.0
95.1
23.5
38.4
HAMNet (Li al., 2020)
27.7
26.3
41.5
51.7
OSNet (Zhou et al., 2019)
75.0
95.6
28.7
44.8
PFNet (Zheng et al., 2021)
38.5
38.9
52.0
58.4
Circle Loss (Sun et al., 2020)
59.4
81.7
22.7
34.2
IEEE (Wang et al., 2022)
47.5
44.4
57.1
63.6
HRCN (Zhao al., 2021)
67.1
91.8
23.4
44.2
DENet (Zheng et al., 2023)
42.4
42.2
55.3
64.5
Multi
UniCat (Crawford et al., 2023)
57.0
55.7
TransRelD (He al., 2021)
75.6
92.9
18.4
29.6
AGW (Ye al., 2022)
73.1
92.7
28.9
46.9
HTT (Wang et al., 2024c)
71.1
73.4
83.1
87.3
HAMNet (Ii et al., 2020)
74.5
93.3
27.1
42.3
EDITOR (Zhang et al., 2024a)
66.5
68.3
81.1
88.2
PFNet (Zheng et al., 2021)
68.1
94.1
23.5
37.4
RSCNet (Yu et al., 2024)
68.2
72.5
GAFNet (Guo et al., 2022)
74.4
93.4
TOP-ReID (Wang et al., 2024a)
72.3
76.6
84.7
89.4
GraFT (Yin et ul., 2023)
76.6
94.3
Ours
80.7
83.6
91.9
94.1
GPENet (He et al., 2023)
75.0
94.5
PHT (Pan et al., 2023)
79.9
92.7
Muld
UniCat (Crawford et al., 2023)
79.4
96.2
Implementation details. Our model is implemented using the PyTorch toolbox, and experiments are conducted on an NVIDIA V100 GPU. We utilize pre-trained CLIP (Rad- ford et al., 2021) as the visual encoder. Images are re- sized to 256 x 128 for RGBNT201 and 128 x 256 for RGBNT100 and MSVR310. For data augmentation, we apply random horizontal flipping, cropping, and erasing fol- lowing (Zhong et al., 2020). The mini-batch size is set to 128 for RGBNTI00, and 64 for RGBNT201 and MSVR310, with corresponding sampling strategies for each dataset. We employ the Adam optimizer with an initial learning rate of 3.56 - 4 and the learning rate of the visual encoder is 5e 6. The total number of training epochs is set 45 for RGBNT201 and RGBNTIOO, and 50 for MSVR310.
CCNet (Zheng et al., 2022)
77.2
96.3
36.4
55.2
HTT (Wang et al., 2024c)
75.7
92.6
TOP-ReID (Wang et al., 2024a)
81.2
96.4
35.9
44.6
EDITOR (Zhang et al., 2024a)
82.1
96.4
39.0
49.3
RSCNet (Yu et al., 2024)
82.3
96.6
39.5
49.6
Ours
88.2
97.4
50.6
64.8
results of our proposed MFRNet with other methods on the RGENTI00 and MSVR310 datasets. For the RGBNT 100 dataset, TransReID achieves the best performance among single-spectral methods, while RSCNet significantly out- performs it among multi-modal methods. However, for the MSVR310 10 dataset, AGW achieves the best performance among single-spectral methods, while RSCNet has the high- est mAp and CCNet has the highest R-1 among multi-modal methods. Notably, our proposed method achieves significant improvements on both datasets, increasing mAP by 5.9% and R-1 by 0.8% RGBNT100, and increasing mAP by 11.1% and R-1 by 9.6% on MSVR310. This demonstrates the superior performance of MFRNet for multi-mode vehi- cle re-identification.
4.2. Comparison with State-of-the-Art Methods
multi-modal Person ReID. We compare the proposed MFRNet with several single-spectral and multi-modal meth- ods on the RGBNT201 dataset, as shown in Table 1. It is evident that multi-modal methods generally outperform single-spectral methods by leveraging complementary multi- modal information. Among current multi-modal methods, TOP-ReID achieves the best performance, with 72.3% mAP, 76.6% R-1, 84.7% R-5, and 89.4% R-10. Our proposed method improves these metrics by 8.4%, 6.9%, 7.2%, and 4.7%, which demonstrates the superior performance of MFRNet for multi-modal person RelD. Notably, we use the CL.I--based VIT to reproduce TOP-ReID, and the re- sults show that our method surpasses TOP-ReID by approx- imately 9.7% mAP and 9.6% R-1.
Evaluation on Missing-spectra Scenarios. In real-world application scenarios, persistent modality availability can- not be guaranteed. Therefore, exploring the performance of models under the condition of missing modalities helps to explore the application boundaries of the model. Therefore, we also evaluate the performance of MFRNet under differ- ent modality missing scenarios on the RGBNT201 dataset. As shown in Table PCB achieves the highest average mAP and R- among single-spectral methods. TOP-RelI)
multi-modal Vehicle ReID. Table 2 shows the comparison
6

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Experrs
Table Performance of missing-modallit settings on RGENT201. "M (X)" means missing the image modality. The best and second results are in bold and underlined, respectively.
M (RGB)
M (NTR)
M (TIR)
(RGB+NIR)
TRGB+TTRR
M (NIR+TIR)
Average
Methods
mAP
R-1
mAP
R-1
mAP
R-1
mAP
R-1
mAP
R-1
mAP
R-1
MAP
R-1
MUDecp (Qian ct al., 2017)
19.2
16.4
20.0
17.2
18.4
14.2
13.7
11.8
11.5
6.5
12.7
8,5
15.9
12.9
HACNN (Li ct al., 2018)
12.5
11.1
20.5
19.4
16.7
13.3
9.2
6.2
6.3
2.2
14.8
12.0
13.3
10.7
Single
MLFN (Chang et al., 2018)
20.2
18.9
21.1
19.7
17.6
11.1
13.2
12.1
8.3
3,5
13.1
9.1
15.6
12.4
PCB (Sun et al., 2018)
23.6
24.2
24,4
25.1
19.9
14.7
20.6
23.6
11.0
6.8
18.6
14.4
19.7
18.1
OSNet (Zhou al., 2019)
19.8
17.3
21.0
19.0
18.7
14.6
12.3
10.9
9.4
5.4
13.0
10.2
15.7
12.9
PFNet (Zheng et al., 2021)
31.9
29.8
25.5
25.8
26.4
23.4
Molti
DENct (Zheng al., 2023)
35.4
36.8
33.0
35,4
32.4
29.2
TOP-ReID (Wang al., 2024a)
54.4
57.5
64.3
51.9
54.5
35.3
35,4
26.2
26.0
34.1
31.7
44.4
45.4
Ours
64.7
65.2
72.3
76.1
51.6
49.5
41.4
43.4
27.3
27.9
37.2
35.6
49.1
49.6
Table 4. Comparison of computational cost with recent meth- ods. We show the best result bold.
Table 6. Performance analysis under different numbers of ex- perts for FRM. We show the best score in bold.
Methods
Params(M)
Flops(Gi
Number
mAP
R-1
R-5
R- 10
Average
HTT (Wang al., 2024c)
85.6
33.1
3
77.6
78.6
87.9
92.3
84.1
EDITOR (Zhang et al., 2024a)
117.5
38.6
6
77.8
80.9
88.6
92.2
84.9
TOP-ReID (Wang et al.. 2024a)
278.2
34.5
9
75.1
76.7
87.8
92.3
83.0
Ours
57.1
22.1
Table Comparison with different modules. We show the best
score in bold.
Table 7. Performance analysis of FRM at different locations within VIT. We show the best score in bold.
Modules
Metrics
Params
FLOPs
Index
FFM
FRM
MAP
R-1
R-5
R-10
M
Layer
mAP
R-1
R-5
R-10
Average
X
X
69.2
76.3
57.2
10,11,12
71.6
75.5
87.1
90.8
81.2
X
75.1
78.1
X
77.8
80.9
92.2
10,12
75.7
77.6
87.6
91.7
83.1
4
80.7
83,5
91.9
94.1
57.1
22.1
12
77.8
80.9
88.6
92.2
84.9
outperforms it with average performance of 44.4% mAP and 45.4% R-1 among multi-modal methods. Our proposed method improves these metrics by 4.7% mAP and 4.2% R-1 over TOP-ReID, validating the adaptability of our model to modality missing scenarios. It is worth noting that com- pared the TOP-ReID which has a specific training phase for the modality missing condition, our MFRNet framework notably trains without explicit task-specific architectural modifications or dedicated training strategies, yet shows its intrinsic capability to compensate for the missing modality.
integrating the FFM module increases the model's parame- ters and FLOPs by 3.6M and 1.4G, respectively, therefore further efforts needed reduce them.
Feature Representation Module. Building the afore- mentioned base framework, we introduce the Feature Rep- resentation Module (FRM). Experimental results show that integrating the FRM module into the haseline model im- proves mAp, R-1, R-5, and R-10 by 8.6%, 4.6%, 3.8%, and 2.7%, respectively. Notably, the model's parameters and FLOPS decrease by 3.7M and 1.4G as the FRM module RepAdapter to build MoE representations, reducing computational cost with two convolutional layers. This demonstrates that the FRM module enhances model accu- racy by effectively representing shared and specific features of multi-modal data while using fewer parameters.
4.3. A Ablation Study
To explore the effectiveness of different components in our model, we conduct comprehensive ablation studies on the RGBNT201 dataset, as shown in 5.
Feature Fusion Module. We use a shared vision trans- former combined with label smoothing cross-entropy and triplet loss as our base framework. Building on this, we integrate the Feature Fusion Module (FFM), which boosts the performance the mAP, R-1, R-5, and R-10 metrics by 5.9%, 1.8%, 2.6%, and 2.5% respectively. This shows that FFM effectively enhances overall model performance by generating and interacting multi-modal features using the fine-grained alignment multi-modal data. However,
MFRNet. MFRNet, which integrates the FFM and FRM modules, achieves 80.7% mAP, 83.5% R-1, 91.9% R-5, and 94.11 94.1% R-10. In addition, MFRNet reduces the parameters by O. IM compared to the baseline framework, while being on par with the bascline model on FLOPS. Consequently, the design of effective multi-modal feature fusion and rep- resentation mechanisms enables more precise multi-modal object re-identification.

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Expee Expertss
Table 8. Performance analysis under different experts number for FFM. We show the best score bold.
RGB
Number
mAP
R-1
R-5
R-10
Average
1
78.7
81.0
90.8
93.5
86.0
2
74.9
78.6
86.8
90.8
82.8
3
80.7
83.5
91.9
94.1
87.5
6
76.9
80.4
88.2
90.6
84.0
9
79.2
82.3
90.7
93.5
86.4
10
76.4
80.1
88.0
92.0
84.1
NIR
11
74.1
78.9
87.2
92.2
83.1
12
74.1
77.8
86.8
91.1
82.4
4.4. More Analysis
Analysis of model computational complexity. We com- pared computational costs across three major recent works. As shown in Table 4, our method achieves the lowest Params and FLOPs, measuring 57.1M and 22.1G, respectively.
TIR
Experts Number in FRM. We explore the model's 's perfor- mance with different numbers of experts in the FRM, where experts are responsible for extracting modality-specific fea- tures. As shown in Table 6, the performance is optimal when the number of experts 6. Although more experts may make sense in maintaining modality-specific repre- sentations, this may also limit the network's learning of modality-shared knowledge.
Figure 3. Visualization of expert in FRM. As expected earlier, similar content can share their knowledge by selecting similar experts reduce redundancy, while for modality-specifie repre- sentations that are difficult to fuse, their knowledge can also be retained by new experts.
Table 9. Performance analysis under different À for FFM. We show the best score in bold.
The Location of FRM. We further explore the model's per- formance when the FRM module is placed in different layers of the VIT. As shown in Table 7, using the FRM only in the final layer is sufficient to capture modality-specifie features for multi-modal data, achieving optimal performance.
Scale of A
mAP
R-1
R-5
R-10
Average
0.0
4.7
2.4
9.8
18.4
8.8
0.3
79.9
82.1
89.2
92.8
86.0
0.5
80.7
83.5
91.9
94.1
87.5
0.7
74.8
77.8
86.1
90.4
82.3
Expert visualization of FRM. To visually observe the allocation of multiple experts in the FRM when extract- ing multi-modal features, we conduct visualizations on the RGBNT201 test set. The numbers in Figure 3 represent the index of the six experts. We also utilize similar colors by the same expert.
1.0
77.5
81.5
88.8
92.5
85.1
modal features. As shown in Table 8, the model achieves the highest average performance when the number is 3.
mark patches that are focused It can be observed that Expert 0 concentrates on extract- ing pedestrian features in the NIR modality, while Experts 1, 2, and 3 focus on pedestrian regions in the RGB and TIR modalities. Additionally, Experts and 5 tend to ex- tract background features across all modalities. As our initial hypothesis predicted, semantically similar content achieves knowledge sharing through selecting analogous experts, while podality-specifff representations that resist fusion maintain their distinct characteristics via dedicated expert allocation.
Parameter À of FFM. For the parameter À in the FFM, we analyze the model's 's performance at different values. As shown in Table 9, the model performs the worst when À = 0, while achieving the highest average accuracy at À = 0.5.
The Location of FFM. We evaluate the approach by insert- ing FFM into the 3rd, 6th, and 9th layers of the network. Additionally, we evaluate the post-fusion method by posi- tioning FFM after the VIT feature encoding. As shown in Table 10, the pre-fusion method proves to be the most effec- tive. We deem that high-level semantic representation may already lose most of the fine-grained perception and align- ment characteristics, demonstrating the limitation current methods that use the high-level semantic fusion strategy.
Experts Number in FFM. Similarly, we analyze the model's ' performance with different numbers of experts in the FFM, where experts are responsible for generating cross-
8

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-Expeee
Table 10. Performance analysis under different locations for FFM.
We show the best score in bold.
Crawford, J., Yin, H., McDermott, L.,, and Cummings, D. Unicat: Crafting stronger fusion baseline for multi- modal re-idntiffict,,. arXiv preprint arXtv:2310.18812 2023.
Location
mAP
R-1
R-5
R-10
Average
O(Before VIT)
80.7
83,5
91.9
94.1
87.5
50.6
50.5
62.9
70.7
58.7
74.0
78.7
86.8
91.1
82.6
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of -xxeeettss-eees lan- guage models. arXiv preprint wrXiv:240I.0000 1.6066, 2024.
75.3
78.3
86.8
90.1
82.6
12(Aferr VIT)
76.7
79.7
86.8
92.0
83.8
5. Conclusion
Ganaie, M. A., Hu, M., Malik, K., Tanveer, M., and Suganthan, P. N. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 1 115: 105151, 2022.
In this paper, we propose MFRNet, novel framework for multi-modal object RelD that enhances feature fusion and representation through the mixture of experts (MoE). Specifically, we introduce a multi-modal feature fusion mod- ule (FFM) to enable fine-grained cross-modal interaction. Following this, the designed feature representation mod- ule (FRM) extracts modality-speciinn and modality shared features through parameter- trffmittr-efficient expert architecture. Extensive experiments on the RGBNT201, RGBNT100, and MSVR310 datasets demonstrate the effectiveness and efficiency of our model.
Gui, Y., Chen, M., Su, Y. Luo, G., and Yang, Y. Eeg- mamba: Bidirectional state space model with mixture of experts for eeg multi-task classification. arXiv preprint arXiv:2407.20254, 2024.
Guo, J., Zhang. X., Liu, Z., and Wang. Y. Generative and at- tentive fusion for multi suntti-ppeetral vehicle re-identification. In 2022 7th International Conference on Intelligent Com- puting and Signal Processing (ICSP), pp- 1565-1572, 2022.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
He, Q. Lu, Z., Wang, Z.,, and Hu, H. Graph-based pro- gressive fusion network for multi-modality vehicle re- identification. IEEE Transactions Intelligent Trans- portation Systems, 24(11):12431-2241) 2023.
References
He, S., Luo, H., Wang, P., Wang, F., Li, H., and Jiang, W. Transreid: Transformer-based object re-identiffiatton.. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 15013-15095 2021.
Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv: 2010. / 1929, 2020.
Chang. X., Hospedales, T. M., and Xiang, T. Multi-level fac- torisation net for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Hermans, A., Beyer, L., and Leibe, B. In defense of the triplet loss for person re-identification. arXiv preprint 1PXYv:7003007331 2017.
Hwang, C., Cui, W.. Xiong, Y., Yang. Z., Liu, Z., Hu, H.,
Chen, G., Zhang. T., Lu, J., and Zhou, J. Deep meta metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Wang, Z., Salas, R., Jose, J., Ram, P., Chau, H., Cheng, P., Yang, F., Yang, M., and Xiong, mf scale. Proceedings of Machine Learning and Systems, volume 5, pp. 269-287, Curan, 2023.
Tutel: Adaptive
Chen, Y. and Wang. L. emoe-tracker: Environmental moe- based transformer for robust event-guided object tracking. IEEE Robotics and Automation Letters, 10(2):1393-1400, 2025.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural Computation, 3(1):79 87, 1991.
Chowdhury, M. N. R., Zhang, S., Wang, M., Liu, S., and Chen, P.-Y. Patch-level routing in mixture-of-expe -ofeeeers--eeetts--s--eee is provably safple-efficient for convolutional neural net- works. In Proceedings of the 40th International Confer- ence on Machine Learning, volume 202, pp. 6074-6114. PMLR, 2023.
Ji, J., Wang. H., Wu, C., Ma, Y., Sun, X., and Ji, R. Jm3d & jm3d-llm: Elevating 3d representation with joint multi- modal cues. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4):2475-244 24922, 2025.

---

Multi-Modal Object Re-Identification via Sparse Mixture-of-
Kuang, Z., Zhang, H., Ye, M., Yang, B., Liu, Y., Huang, Y., Ding, X., and Li, H. Unity in diversity: Multi- expert knowledge confrontation and collaboration for generalizable vehicle re- identification. arXiv preprint arXiv:2407.0735/ 2024.
Luo, G., Huang, M., Zhou, Y., Sun, X., Jiang, G., Wang. Z.,, and Ji, R. Towards efficient visual adaption via structural ce-parameterization. arXiv preprint S.Ki:23302088000 2023.
Luo, H.,, Gu, Y., Liao, X., Lai, S., and Jiang, W. Bag of tricks and strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang. Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding, arXiv preprint arXiv:2006.16666 2020.
Mustafa, B., Riquelme, C.,, Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal contrastive learning with limoe: the language-image mixture of experts. In Advances in Neural Information Processing Systems, volume 35, pp. Curran Associates, Inc., 2022.
Li, B., Shen, Y., Yang, J., Wang, Y., Ren, J., Che, T., Zhang, J., and Liu, Z. Sparse mixture-of-experts are domain generalizable learners. arXiv preprint arXiv:2206.0 04046, 2022.
Li, H., Li, C.,, Zhu, X., Zheng, A., and Luo, B. Multi-spectral vehicle re-identification: A challenge. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07): 11345-11353, 2020.
Pan, W., Wu, H., Zhu, J., Zeng, H., and Zhu, X. H-vit: Hybrid vision transformer for multi-modal vehicle re- identification. In Artificial Intelligence, pp. 255-267, Cham, 2022. Springer Nature Switzerland.
Pan, W., Huang, L., Liang, J., Hong, L.. and Zhu, J. Pro- gressively hybrid transformer for re-identification. Sensors, 23(9), 2023.
Li, W., Zhu, X., and Gong. S. Harmonious attention network for person re-identification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
vehicle
Park, H., Lee, S., Lee, J., and Ham, B. Learning by align- ing: Visible-infrared person re-identification using cross- modal correspondences. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV). pp. 12026- 12035, 2021.
Li, X., Li, Q.. Liang, F., and Wang, W. Multi colti-gramularity pseudo-label collaboration for unsupervised person re- identification. Computer Vision and Image Understand- ing, 227: 103616, 2023.
Qian, X., Fu, Y., Jiang, Y.-G., Xiang, T., and Xue, X. Multi-scale deep learning architectures for person re- identification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., and Zhang, M. Uni-moe: Scaling unified multimodal Iims with mixture of experts. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Lin, B., Tang, Z., Ye, Y., Cui, Zhu, B., Jin, P., Huang, J., Zhang, J., Pang, Y., Ning, M., et al. Moe-llava: Mixture of experts for large vision-languag models. arXiv preprint arXiv:2401. 15947 2024.
Liu, T., Liu, H., Shang, F., Yu, L., Han, T., and Wan, L. Completed feature disentanglement learning for multi- modal mris analysis. IEEE Journal of Biomedical and Health Informatics, 2025.
Rao, Y., Chen, G., Lu, J., and Zhou, Counterfactual attention learning for fine- fine-grained visual categorization and re-identification. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 1025-1034, 2021.
Liu, X., Zhang, P., Yu, C., Qian, X., Yang, X., and Lu, H. A video is worth three views: Trigeminal transformers for video-based person re-identification. IEEE Transac- tions on Intelligent Transportation Systems, 25(9):12818- 12828, 2024.
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. Ad- vances in Neural Information Processing Systems, 34: 8583-8595, 2021.
Lu, H., Zou, X., and Zhang. P. Learning progres- sive modality-shared transformers for effective visible- infrared person re-identification. Proceedings Conference on Artificial Intelligence, 37(2):1835-1843 2023.
Shen, F. and Tang. J. Imagpose: A unified conditional framework for pose-gue dasee-gided person generation. Advances in neural information processing systems, 37:6246 6266, 2024.
the the AAAI
10

---

Multi-Modal Object Re-Identification via Sparse -ixture-of-Experts
Shen, F., Jiang, X., He, X., Ye, H., Wang, C., Du, X., Li, Z.., and Tang, J. Imagdressing-vl: Customizable virtual dressing, In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 6804, 2025.
Wang, G., Yuan, Y., Chen, X., Li, and Zhou, X. Learning discriminative features with multiple granularities for person re-identification. In Proceedings of the 26th ACM International Conference on Multimedia, pp. 274 282, 2018.
Shi, J., Zhang, Y., Yin, X., Xie, Y., Zhang, Z., Fan, J., Shi, Z., and Qu, Y. Dual pseudo- labels interactive self- training for sumirsuiee visible-infrared person re- identification. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 11218- 11228, 2023.
Wang. Y., Liu, X., Zhang. P., Lu, H., Tu, Z., and Lu, H. Top- reid: Multi-spectral object re-identification with token permutation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6):5758-577 5766, 2024a.
Wang, Y., Zhang, P., Wang. D., and Lu, H. Other tokens matter: Exploring global and local features of vision transformers for object re-identification. Computer Vision and Image Understanding. 244:104030, 2024b.
Shi, J., Yin, X., Zhang, Y., Zhang, Z., Xie, Y., and Qu, Y. Learning commonality, divergence and variety for unsu- pervised disible-infrared person re-identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Wang. Z., Li, C., Zheng, A., He, R., and Tang, J. Interact, embed, and enlarge: Boosting modality-specifff represen- tations for multi-modal person re-identification. Proceed- ings of the AAAI Conference on Artificial Intelligence, 36 (3):2633 2022.
Sun, Y., Zheng, I.., Yang. Y., Tian, Q., and Wang. S. Beyond part models: Person retrieval with refined part pooling (and strong convolutional baseline). In Proceedings of the European Conference Computer Vision (ECCV), 2018.
Wang, Z., Huang, H., Zheng, A., and He, R. Heterogeneous test-time training for multi-modal person re-identification. Proceedings of the AAAI Conference on Artificial Intelli- gence, 38(6):5850-5888 5858, 2024c.
Sun, Y., Cheng, C., Zhang. Y., Zhang, C., Zheng, L., Wang, Z., and Wei, Y. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Xu, B., Liang, J., He, I.,, and Sun, 7. Mimic embedding via adaptive aggregation: Learning generalizable person re-identification. In European Conference on Computer Vision, pp. 372-388. Springer, 2022.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Com- puler Vision and Pattern Recognition (CVPR), 2016.
Yang, B., Chen, J., and Ye, M. Shallow-deep collabora- tive learning for unsupervised visible-infrared person re- identification. In Proceedings of the IEEE/CVE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 16870-16879, 2024.
Tan, L.., Dai, P., Ji, R., and Wu, Y. Dynamic prototype mask for occluded person re-identification. In Proceedings of the 30th ACM international conference on multimedia, pp. 531-540, 2022.
Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., and Hoi, S. C. H. Deep learning for person re-identification: A survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 446))22872-2893, 2022.
Tan, L., Dai, P., Ye, Q. Xu, M., Wu, Y., and Ji, R. Spectral aware softmax for visible-infrared person re- identification. arXiv preprint arXiv:2302.01 015.01 2023.
Tan, I.,, Dai, P., Chen, J., Cao, I.., Wu, Y., and Ji, R. Part- former: Awakening latent diverse representation from vision transformer for object re-identification. arXiv preprint arXiv:2408. 16684, 2024a.
Yin, H., Li, J., Schiller, E., McDermott, L., and Cummings, D. Graft: Gradual fusion transformer for multimodal re-identification.
arXiv preprint 10Xiv:2310.16856, 2023.
Tan, L., Xia, J., Liu, W., Dai, P., Wu, Y., and Cao, L. Oc- cluded person re-identification via saiieeey-guided patch transfer. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 38, pp. 5070-5078, 2024b.
Yu, 7.,, Huang, 7., Hou, M., Pei, J., Yan, Y., Liu, Y., and Sun, D. Representation selective coupling via token spar- sification for multi multi-speectal object re-identification. IEFE Transactions on Circuits and Systems for Video Technol- ogy, 2024.
Tan, L., Zhang, Y., Han, K., Dai, P., Zhang, Y., WU, Y., and Ji, R. Rle: A unified perspective of data augmentation for cross spectrl re- re-identification. In Advances in Neural Information Processing Systems, volume 37, pp. 126977- 126996, 2024c.
Zhang, G., Zhang, P., Qi, J., and Lu, H. Hat: Hierarchical aggregation transformers for person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 516-525, 2021.

---

Multi-Modal Object Re-Identification via Sparse coxture-of-Expertt
Zhang, P., Wang, Y., Liu, Y., Tu, Z., and Lu, H. Magic tokens: Select diverse tokens for multi-modal object identification. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), Pp. 17117-17 2024a.
Zhang, Y. and Wang, H. Diverse embedding expansion net- work and low-light cross mosslity benchmark for visible- infrared person re-identification. In 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2153-2162, 2023.
Zhang, Y., Yan, Y. Lu, Y., and Wang. H. Adaptive middle modality alignment learning for visible-infrared person re- identification. International Journal of Computer Vision, pp. 1-21, 2024b.
Zhao, J., Zhao, Y., Li, J., Yan, K., and Tian, Y. Heteroge- neous relational complement for vehicle re-identification. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 205 214, 2021.
Zheng, A., Wang. Z., Chen, Z., Li, C., and Tang, J. Robust multi-modalit person re-identification. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4): 3529-3537, 2021.
Zheng, A., Zhu, X., Ma, Z., Li, C., Tang, J., and Ma, J. Multi-spectral vehicle re-identification with cross- directional consistency network and high-quality bench- mark. arXiv preprint 220i 00632, 2022.
Zheng, A., He, Z.., Wang. Z., Li, C., and Tang. J. Dynamic enhancement network for partial multi-modality person re-identification. arXiv preprint arXiv:2305.1577 15762, 2023.
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang. Ran- dom erasing data augmentation. Proceedings the AAAI Conference Artificial Intelligence, 34(00):13001- 13008, 2020.
Zhou, K., Yang, Y., Cavallaro, A., and Xiang. T. Omni- scale feature learning for person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
12


Reviews:
[{"review_id": "aEl1O9BJRI", "reviewer": "Reviewer_PMp6", "summary": "This work presents the Modality Fusion and Representation Network (MFRNet) aiming to address the limitations in modality interaction and representation of recent works. Two modules named Feature Fusion Module (FFM) and Feature Representation Module (FRM) are proposed to tackle the interaction and representation limitations, respectively. Both two modules follow a MoE structure. The FFM employs multiple generator experts to adaptively provide fine-grained interaction information, while the FRM employs diverse representation experts to extract and combine modality-specific and modality-shared features. Experiments are conducted in multi-modal ReID datasets. Both qualitative and quantitative results are reported to show the effectiveness of each component in the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer PMp6:\n\n> **Q1:** From Table 4, we can observe that the FRM has decreased the 'Params' and 'FLOPs'. Generally, the MoE structure should keep similar or slightly higher 'Params' and 'FLOPs' with the baseline during the inference.\n> \n**A1:** Thank you for your suggestion. As demonstrated in Equation 8, the FRM module leverages RepAdapter to build MoE representations. Compared to a traditional MLP, RepAdapter reduces computational costs by incorporating two convolutional layers. If replaced with the original MLP structure, as indicated in Table A, the parameters and FLOPs would increase by 3.7M (from 53.5 to 57.2) and 1.4G (from 20.7 to 22.1), respectively.\n\nTable A: Comparison of FRM and original MLP.\n\n|  |  Params (M) |  FLOPs (G) |\n| --- | --- | --- |\n| Original MLP | 57.2 | 22.1 |\n| FRM | 53.5 | 20.7 |\n\n> **Q2:** The discussion of 'Params' and 'FLOPs' only contains this method itself. I understand that it can be similar efficiency to the baseline method. But it would be better if the author could show a comparison with recent methods.\n\n**A2:** Following your suggestion, we compared the computational costs with three recent works. As shown in Table B, our method not only demonstrates significant improvements in metrics such as mAP and R-1 but also achieves the lowest Params and FLOPs, with values of 57.1M and 22.1G, respectively.\n\nTable B: Comparison of computational cost with recent methods. \n\n|  | mAP | R-1 | R-5 | R-10 | Params(M) | Flops(G) |\n| --- | --- | --- | --- | --- | --- | --- |\n| HTT | 71.1 | 73.4 | 83.1 | 87.3 | 85.6 | 33.1 |\n| EDITOR | 66.5 | 68.3 | 81.1 | 88.2 | 117.5 | 38.6 |\n| TOP-ReID | 72.3 | 76.6 | 84.7 | 89.4 | 278.2 | 34.5 |\n| Ours | **80.7** | **83.5** | **91.9** | **94.1** | **57.1** | **22.1** |\n\n> **Q3:** Table 7 may not enough to show that 3 is the optimal selection.\n> \n**A3:** Thank you for your suggestion. We conducted additional experiments as shown in Table C. The results indicate that the optimal performance, with an average accuracy of 87.5%, is achieved when the number of experts in the FFM is set to 3.\n\nTable C: Performance analysis under different expert numbers for FFM.\n\n| Number | mAP | R-1 | R-5 | R-10 | Average |\n| --- | --- | --- | --- | --- | --- |\n| 1 | 78.7 | 81.0 | 90.8 | 93.5 | 86.0 |\n| 2 | 74.9 | 78.6 | 86.8 | 90.8 | 82.8 |\n| 3 | **80.7** | **83.5** | **91.9** | **94.1** | **87.5** |\n| 6 | 76.9 | 80.4 | 88.2 | 90.6 | 84.0 |\n| 9 | 79.2 | 82.3 | 90.7 | 93.5 | 86.4 |\n\n> **Q4:** FFM doesn't seem to conflict with VIT. It seems can be inserted in the network, but it is not discussed at all.\n> \n**A4:** Following your suggestion, we further validated the approach by inserting FFM into the 3rd, 6th, and 9th layers of the network. As shown in Table D, applying FFM before the ViT leverages the pixel-by-pixel alignment of multimodal image data more effectively, resulting in optimal model performance.\n\nTable D: Performance analysis under different locations for FFM.\n\n| Location | mAP | R-1 | R-5 | R-10 | Average |\n| --- | --- | --- | --- | --- | --- |\n| 0 (Before ViT) | **80.7** | **83.5** | **91.9** | **94.1** | **87.5** |\n| 3 | 50.6 | 50.5 | 62.9 | 70.7 | 58.7 |\n| 6 | 74.0 | 78.7 | 86.8 | 91.1 | 82.6 |\n| 9 | 75.3 | 78.3 | 86.8 | 90.1 | 82.6 |\n| 12 (After ViT) | 76.7 | 79.7 | 86.8 | 92.0 | 83.8 |\n\n> **Q5:** There are still several typos in the paper.\n> \n**A5:** Thank you for your suggestion. We will correct these typos in the updated version of the paper.", "claims_and_evidence": "This paper proposes two modules to solve two claimed problems. From its motivation and implementation, the modules present in this paper effectively support its claims. Experiments and visualization result also show the improvements in these two modules.", "methods_and_evaluation_criteria": "Yes, the problem of multi-modal person re-identification is important and valuable when considering real application scenarios. The proposed method makes sense in this topic due to its performance and efficiency.", "theoretical_claims": "This work proposes two modules that are somewhat reasonable from both its claims and design. The FFM uses the MoE structure for the image completion, while the FRM uses the MoE structure for the representation. Since the MoE structure has already shown its ability in LLM to adapt to multiple modality data, using such a structure to strengthen the ability of multi-modal re-identification reasonable.", "experimental_designs_or_analyses": "I have checked the experiments of this work. Most experiments are appropriate and complete. While there still exist several concerns:\n\n1. From Table 4, we can observe that the FRM has decreased the 'Params' and 'FLOPs'. Generally, the MoE structure should keep similar or slightly higher 'Params' and 'FLOPs' with the baseline during the inference.  \n\n2. The discussion of 'Params' and 'FLOPs' only contains this method itself. I understand that it can be similar efficiency to the baseline method. But it would be better if the author could show a comparison with recent methods.\n\n3. Table 7 may not enough to show that 3 is the optimal selection.\n\n4. FFM doesn't seem to conflict with VIT. It seems can be inserted in the network, but it is not discussed at all.", "relation_to_broader_scientific_literature": "This work uses the concept of MoE to solve the limitations of recent works in two ways. Compared to recent methods such as TOP-ReID [1] which uses multiple networks for different modalities and interacts with them in the final, this work largely simplified the network structure while maintaining the dynamical specific encoding processing and achieving fine-grained for each modality. Because this work aims to solve the specific limitations of this topic, I think it is moderately related to the general re-identification task. \n[1] Wang, Yuhao, et al. \"Top-reid: Multi-spectral object re-identification with token permutation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 6. 2024.", "essential_references_not_discussed": "I think essential references are adequate discussed in this paper.", "other_strengths_and_weaknesses": "Strengths：\n1. The idea of using MoE in interaction and representation is somewhat interesting in this topic.\n2. This method has achieved strong performance while maintaining good efficiency.\n\nWeaknesses:\n1. There still several typos:\n- The caption in Table 6, ‘…for FRM’ -> ‘…for FRM’\n- The caption of Table 3 and Table 4 should be bolded.", "other_comments_or_suggestions": "This is a paper with sufficient motivation and an interesting solution. However, as mentioned before, there are several issues that limit the value of this work. If the two weaknesses can be addressed, I would like to improve my score.", "questions_for_authors": "1. From Table 4, we can observe that the FRM has decreased the 'Params' and 'FLOPs'. However, as an MoE structure, even ignoring the router's parameter, the 'Params' and 'FLOPs' should remain the same with the baseline method. How it can even decrease the 'Params' and 'FLOPs'?\n2. The discussion of 'Params' and 'FLOPs' only contains this method itself. I understand that it can be similar efficiency to the baseline method. But, can you show the comparison with other methods?\n3. From Table 7, what is the performance when number of experts of FFM is less than 3? Table 7 is insufficient to indicate that 3 is optimal. \n4. FFM doesn't seem to conflict with VIT. Why is Table 9 only discussing before and after ViT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lDT0THbhYp", "reviewer": "Reviewer_hHKf", "summary": "This paper proposes a novel Multi-modal Fusion and Representation Network (MFRNet) approach for multi-modal object re-identification, inspired by the sparse Mixture-of-Experts (MoE) paradigm. The proposed framework enhances performance by introducing a Feature Fusion Module (FFM) for fine-grained pixel-level cross-modal interaction and a Feature Representation Module (FRM) to extract modality-shared and modality-specific features dynamically. Experimental evaluations on three benchmark datasets, RGBNT201, RGBNT100, and MSVR310, demonstrate that the proposed method achieves superior performance compared to existing state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** While the MFRNet presents a novel perspective within multi-modal ReID by introducing a sparse Mixture-of-Experts (MoE) framework, its overall novelty remains moderate. It is more like a representing of an effective combination of existing techniques.\n> \n**A1:** Our novelty primarily lies in two aspects: generation and representation. For FFM generation, unlike previous methods that rely on coarse-grained feature interactions, we creatively harness the pixel-level consistency of multimodal data (pixel-wise alignment) to achieve fine-grained multimodal interaction. For FRM representation, our approach avoids the independent encoding of each modality. Instead, we introduce multimodal feature representation experts based on the MoE architecture, achieving a balance between modality-specific and shared features while maintaining minimal computational cost.\n\n> **Q2:** Figure 2 is slightly cluttered, and the author seems to have forgotten to describe GE and RE.\n> \n**A2:**  We sincerely apologize for any inconvenience caused. GE refers to the generation experts in FFR, while RE represents the representation experts in FRM. Based on your suggestion, we will revise Figure 2 accordingly.\n\n> **Q3:** The visualization section appears to focus solely on the Feature Representation Module, while the visualization of the Feature Fusion Module is also essential for a comprehensive analysis.\n> \n**A3:**  Thank you for your suggestion. In response, we have also visualized FFM. However, due to the constraints of the rebuttal format, we are unable to include the image directly. Nonetheless, it will be incorporated in future versions of the paper.\n\n> **Q4:** Can the authors provide a more detailed comparison of computations with different methods?\n> \n**A4:**  Following your suggestion, we compared the computational costs with three recent works. As shown in Table B, our method not only demonstrates significant improvements in metrics such as mAP and R-1 but also achieves the lowest Params and FLOPs, with values of 57.1M and 22.1G, respectively.\n\nTable A: Comparison of computational cost with recent methods. The best results are shown in bold.\n\n|  | mAP | R-1 | R-5 | R-10 | Params(M) | Flops(G) |\n| --- | --- | --- | --- | --- | --- | --- |\n| HTT | 71.1 | 73.4 | 83.1 | 87.3 | 85.6 | 33.1 |\n| EDITOR | 66.5 | 68.3 | 81.1 | 88.2 | 117.5 | 38.6 |\n| TOP-ReID | 72.3 | 76.6 | 84.7 | 89.4 | 278.2 | 34.5 |\n| Ours | **80.7** | **83.5** | **91.9** | **94.1** | **57.1** | **22.1** |\n\n> **Q5:** The author mentioned that ‘TOP-ReID has a specific training phase for the modality missing condition’ while MFRNet does not. Will MFRNet be better than TOP-ReID in the 'M (TIR)' protocol with the same specific training phase?\n> \n**A5:**  Following your suggestion, we conducted additional experiments targeting the 'M (TIR)' protocol. As shown in Table B, training our model to address the missing of TIR resulted in a 7.7% improvement in mAP and a 7.0% increase in R-1 compared to TOP-ReID.\n\nTable B: Experimental results for missing TIR modality.\n\n| M (TIR) | mAP | R-1 | R-5 | R-10 |\n| --- | --- | --- | --- | --- |\n| TOP-ReID | 51.9 | 54.5 | - | - |\n| Ours | 51.6 | 49.5 | 67.7 | 76.7 |\n| Ours (RGB+NIR) | **59.6** | **61.5** | **72.7** | **80.6** |\n\n> **Q6:** Why can the FRM reduce computational complexity?\n> \n**A6:**  Thank you for your suggestion. As demonstrated in Equation 8, the FRM module leverages RepAdapter to build MoE representations. Compared to a traditional MLP, RepAdapter reduces computational costs by incorporating two convolutional layers. If replaced with the original MLP structure, as indicated in Table C, the parameters and FLOPs would increase by 3.7M (from 53.5 to 57.2) and 1.4G (from 20.7 to 22.1), respectively.\n\nTable C: Comparison of FRM and original MLP.\n\n|  | Params (M) | FLOPs (G) |\n| --- | --- | --- |\n| Original MLP | 57.2 | 22.1 |\n| FRM | 53.5 | 20.7 |\n\n> **Q7:** There are several typos in the paper.\n> \n**A7:** Thank you for your suggestion. We will correct the typos in the updated version of the paper.", "claims_and_evidence": "Yes, the claims in the submission are generally supported by clear and convincing evidence. The authors present two well-structured modules with empirical results that support the proposed modules.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for addressing the problem. The selected benchmark datasets are appropriate, and the evaluation approach is well-aligned with the research objectives.", "theoretical_claims": "Yes, I have checked the correctness of the proofs for the theoretical claims and found no issues. The formulations and proofs are well-founded and align with standard methodologies in the field.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and related analyses and found it to be sound and sufficient.", "relation_to_broader_scientific_literature": "The key contributions of this paper relate closely to the broader scientific literature in the following the Sparse Mixture-of-Experts (MoE) Integration. The paper builds upon the concept of Sparse MoE, which has been widely explored in deep learning to improve model expressiveness and parameter efficiency. Specifically, prior works such as GMoE [1] have leveraged MoE frameworks in domain generalization problem. The current paper extends this idea explicitly to multi-modal object re-identification, demonstrating its effectiveness in this context. Overall, the proposed contributions build upon and advance existing findings in sparse MoE modeling and achieve fine-grained pixel-level interactions, and multi-modal representation balancing, extending these concepts specifically into the multi-modal object re-identification task and demonstrating clear empirical advantages over prior state-of-the-art approaches.\n\nReference:\n\n[1] Li, Bo, et al. \"Sparse Mixture-of-Experts are Domain Generalizable Learners.\" In The Eleventh International Conference on Learning Representations, 2023.", "essential_references_not_discussed": "Yes, related works that are essential to understanding the key contributions of the paper are adequately cited and discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes an effective sparse Mixture-of-Experts (MoE) architecture for multi-modal ReID, achieving notable performance improvements over existing state-of-the-art models.  \n\n2. The paper conducts comprehensive ablation studies, clearly demonstrating the contribution and effectiveness of each module.  \n\n3. MFRNet achieves significant performance gains on three public datasets, validating the effectiveness of the proposed approach.  \n\n4. MFRNet exhibits a certain level of robustness in scenarios with missing modalities, even without explicit training for such cases.  \n\nWeaknesses:\n\n1. While the MFRNet presents a novel perspective within multi-modal ReID by introducing a sparse Mixture-of-Experts (MoE) framework, its overall novelty remains moderate. It is more like a representing of an effective combination of existing techniques.", "other_comments_or_suggestions": "1. Page 2, Line 127-128: ‘we propose MRFNet’ should be ‘we propose MFRNet’.\n\n2. Page 4, Line 217-218: ‘we aim to modality-specific’ should be ‘we aim to capture modality-specific’?\n\n3. Figure 2 is slightly cluttered, and the author seems to have forgotten to describe GE and RE.", "questions_for_authors": "1. The visualization section appears to focus solely on the Feature Representation Module, while the visualization of the Feature Fusion Module is also essential for a comprehensive analysis.\n\n2. Can the authors provide a more detailed comparison of computations with different methods?\n\n3. The author mentioned that ‘TOP-ReID has a specific training phase for the modality missing condition’ while MFRNet does not. Will MFRNet be better than TOP-ReID in the 'M (TIR)' protocol with the same specific training phase?\n\n4. Why can the Feature Fusion Module reduce computational complexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7XDCDwLlfK", "reviewer": "Reviewer_yVLW", "summary": "This paper introduces MFRNet for multi-modal object re-identification. This approach addresses two core issues: insufficient pixel-level feature interaction and difficulty balancing between shared and specific modality features. The proposed Feature Fusion Module (FFM) fosters fine-grained cross-modal interaction, while the Feature Representation Module (FRM) efficiently merges modality-shared and modality-specific representations in a unified network. Experiments on three public datasets show that MFRNet significantly improves both accuracy and efficiency, with minimal computational overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** Specifically, in Section 3.1, the description of the FFM states that transformations for all three modalities occur simultaneously. Yet, Figure 2 only illustrates interactions involving the RGB image, which may not fully represent the fusion process.\n> \n**A1:**  Thank you for your suggestion. This transformation process applies equally to all three modalities. While the figure specifically illustrates the NIR+RGB->TIR interaction, we have conducted the same interaction for NIR and RGB. We will revise the caption of Figure 2 in light of your feedback to ensure the process is described clearly.\n\n> **Q2:** The necessary references have been discussed, but the related work section is somewhat lengthy. It would be better to condense it.\n> \n**A2:**  Thank you for your suggestion. We will accordingly reduce the content of the related work section.\n\n> **Q3:** Given that prior works such as TOP-ReID [1], EDITOR [2], and RSCNet [3] both use an ImageNet-based ViT as the backbone, while MFRNet adopts a CLIP-based ViT, I am curious about MFRNet’s performance when using an ImageNet-based ViT.\n> \n**A3:**  Following your suggestions, we conducted experiments on our server to evaluate TOP-ReID and our method using ImageNet-based ViT and CLIP-based ViT. As shown in Table A, the results demonstrate that both methods perform better with CLIP-based ViT compared to ImageNet-based ViT. Specifically, with ImageNet-based ViT, our method outperforms TOP-ReID by approximately 2.2% in mAP and 3.7% in R1. With CLIP-based ViT, our method surpasses TOP-ReID by about 9.7% in mAP and 9.6% in R1. These findings underscore the superior performance of our method over TOP-ReID under identical settings and backbones.\n\nTable A: Performance comparison on ImageNet-based and CLIP-based ViT.\n\n| Methods | mAP | R-1 | R-5 | R-10 |\n| --- | --- | --- | --- | --- |\n| ViT: TOP-ReID | 67.4 | 69.1 | 80.9 | 86.0 |\n| ViT: Ours | **69.6** | **72.8** | **84.9** | **90.6** |\n| CLIP-ViT: TOP-ReID | 71.0 | 73.9 | 81.6 | 86.6 |\n| CLIP-ViT: Ours | **80.7** | **83.5** | **91.9** | **94.1** |\n\n> **Q4:** Some details of the testing phase are not entirely clear. In Table 3, I would like to know how the FFM work in the absence of certain modalities.\n> \n**A4:**  Following your suggestion, we will refine this section in future versions to enhance clarity. For the missing modality, as illustrated in Equation 6 (lines 190-199), the other two modalities are utilized for prediction. For example, if RGB is missing, its features are generated based on NIR and TIR as follows: $I_R=w_R^{N}(I_N) \\times g(I_N) + w_R^{T}(I_T) \\times g(I_T)$. Here, $g(I_N)$ denotes the features for RGB generated from NIR, $g(I_T)$ represents the features for RGB generated from TIR, $w_R^{N}(I_N)$ and $w_R^{T}(I_T)$ are the respective weights. \n\n> **Q5:** Some captions lack some details. What do ‘RE’ and ‘GE’ mean in Figure 2?\n> \n**A5:**  We sincerely apologize for any inconvenience caused. 'GE' refers to the generation experts in FFR, while 'RE' represents the representation experts in FRM. Based on your suggestion, we will revise Figure 2 accordingly.\n\n> **Q6:** Eq.12 and Eq.13 are not aligned.\n> \n**A6:**  Thank you for your suggestion. We have correctted it in these days.", "claims_and_evidence": "This work‘s claims are intuitive, convincing, and supported by its experiments.", "methods_and_evaluation_criteria": "It looks well. These two limitations addressed in this topic are reasonable, the introduction of FFM and FRM is suitable for this task.", "theoretical_claims": "I have reviewed the methodology and corresponding equations in this work, and they appear to be both reliable and reasonable. However, Figure 2 may be misleading. Specifically, in Section 3.1, the description of the FFM states that transformations for all three modalities occur simultaneously. Yet, Figure 2 only illustrates interactions involving the RGB image, which may not fully represent the fusion process.", "experimental_designs_or_analyses": "The experimental design of the paper is comprehensive. In addition to the ablation studies on FFM and FRM, the paper also provides a detailed discussion of relevant hyperparameters, making the overall experiments convincing.", "relation_to_broader_scientific_literature": "Adapting to multi-modal data using MoE is an appropriate approach and is widely employed in current Multimodal Large Language Models. This work integrates this technique into the task of multi-modal object re-identification. Therefore, the impact of this paper is moderate, but it provides some valuable insights into methods in this field.", "essential_references_not_discussed": "The necessary references have been discussed, but the related work section is somewhat lengthy. It would be better to condense it.", "other_strengths_and_weaknesses": "Strengths：\n- The proposed framework is well-structured, with clearly defined motivations and a logically designed modular architecture. The integration of Feature Fusion Module (FFM) and Feature Representation Module (FRM) effectively enhances feature interaction and representation learning, addressing key challenges in multi-modal object re-identification.\n- The performance of this work is quite strong, achieving a significant improvement over the previous state-of-the-art.\n\nWeaknesses:\n- The captions lack some details. What do ‘RE’ and ‘GE’ mean in Figure 2?  \n- Eq.12 and Eq.13 is not alignment.", "other_comments_or_suggestions": "I have no more comments, please check the aforementioned parts.", "questions_for_authors": "1. Given that prior works such as TOP-ReID [1], EDITOR [2], and RSCNet [3] both use an ImageNet-based ViT as the backbone, while MFRNet adopts a CLIP-based ViT, I am curious about MFRNet’s performance when using an ImageNet-based ViT.\n\n2. Some details of the testing phase are not entirely clear. In Table 3, I would like to know how the FFM work in the absence of certain modalities.\n\n3. Some captions lack some details. What do ‘RE’ and ‘GE’ mean in Figure 2?\n\n[1] \"Top-reid: Multi-spectral object re-identification with token permutation.\" AAAI 2024.\n\n[2] \"Magic tokens: Select diverse tokens for multi-modal object re-identification.\" CVPR 2024\n\n[3] \"Representation Selective Coupling via Token Sparsification for Multi-Spectral Object Re-Identification.\" IEEE Transactions on CSVT (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIYy5u1o71", "reviewer": "Reviewer_am8H", "summary": "This work introduces MFRNet, which mitigates insufficient interaction and feature imbalance via two modules. The Feature Fusion Module (FFM) uses a mixture-of-generators for pixel-level alignment, while the Feature Representation Module (FRM) employs a mixture-of-experts for balanced modality-shared and modality-specific features.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** While the experiments are comprehensive, the analysis lacks depth. For instance, Table 3 presents results for M(TIR) without providing a corresponding discussion in which the performance of MFRNet is lower than TOPReID.\n> \n**A1:**  Thank you for your suggestion. TOP-ReID benefits from specific training tailored for handling modality-missing scenarios, whereas our method does not include dedicated training for such cases. During training, we utilize all modalities. For testing, when a modality is missing, we supplement it using Equation 6 from the paper. For example, if RGB is missing, its features are generated based on NIR and TIR as follows: $I_R=w_R^{N}(I_N) \\times g(I_N) + w_R^{T}(I_T) \\times g(I_T)$. Here, $g(I_N)$ represents the features for RGB generated from NIR, $g(I_T)$ denotes the features for RGB generated from TIR, $w_R^{N}(I_N)$ and $w_R^{T}(I_T)$ are the respective weights.) To ensure model generalizability, we do not employ dedicated training methods for handling missing modalities. Notably, performing specific missing modality training could yield even better results. As shown in Table A, training our model to address the missing TIR led to a 7.7% improvement in mAP and a 7.0% increase in R-1 compared to TOP-ReID.\n\nTable A: Experimental results for missing TIR modality.\n\n|  | mAP | R-1 | R-5 | R-10 |\n| --- | --- | --- | --- | --- |\n| TOP-ReID | 51.9 | 54.5 | - | - |\n| Ours | 51.6 | 49.5 | 67.7 | 76.7 |\n| Ours (RGB+NIR) | **59.6** | **61.5** | **72.7** | **80.6** |\n\n> **Q2:** This paper could be better by considering published literature in the cross-modality ReID community such as [1, 2]. [1] Zhang, Yukang, and Hanzi Wang. \"Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. [2] Park, Hyunjong, et al. \"Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n> \n\n**A2:** Thanks for your suggestion. We have included these papers in Sec. 2 to make our work more comprehensive. If you have any other recommended works that are related to this paper, you can provide them in the discussion. We are willing to add them in.\n\n> **Q3:** Besides the above mentions, I have one more question: While the authors' claim is convincing, the results in Table 7 do not provide sufficient evidence that the Feature Fusion Module (FFM) requires more experts. It would be beneficial to include a comparison with the case where the number of experts in FFM is set to 1 to better validate this claim.\n> \n**A3:**  Thank you for your suggestion. We conducted additional experiments as shown in Table B. The results indicate that the optimal performance, with an average accuracy of 87.5%, is achieved when the number of experts in the FFM is set to 3.\n\nTable B: Performance analysis under different expert numbers for FFM.\n\n| Number | mAP | R-1 | R-5 | R-10 | Average |\n| --- | --- | --- | --- | --- | --- |\n| 1 | 78.7 | 81.0 | 90.8 | 93.5 | 86.0 |\n| 2 | 74.9 | 78.6 | 86.8 | 90.8 | 82.8 |\n| **3** | **80.7** | **83.5** | **91.9** | **94.1** | **87.5** |\n| 6 | 76.9 | 80.4 | 88.2 | 90.6 | 84.0 |\n| 9 | 79.2 | 82.3 | 90.7 | 93.5 | 86.4 |\n\n> **Q4:** In Equations (5) and (9),  $W_1$ and $W_9$ originate from different modules and should be explicitly distinguished to avoid confusion.\n> \n**A4:**  Thank you for your suggestion. We have added markers in the top-right corner to distinguish between different components. \n\n> **Q5:**  The notation for Mean and Std in Equations (12) and (13) is inconsistent in font style. Please ensure uniform formatting for clarity and consistency.\n> \n**A5:**  Thank you for your suggestion. We have corrected them following your comments.", "claims_and_evidence": "This work presents two key claims. The first claim highlights the limitations of recent approaches in modal interaction. To address this, the authors advocate for pixel-level interactions instead of feature-level interactions. Given the characteristics of multimodal images, this proposition is technically sound, as further validated by subsequent experiments. The second claim concerns the trade-off between feature representation quality and computational efficiency in existing methods. To tackle this issue, the authors propose leveraging a Mixture of Experts (MoE) to enable dynamic parameter allocation and streamline the model structure. This paradigm is widely recognized as effective, as MoE not only adapts to data variations but also enhances model efficiency by reducing redundancy. Therefore, the claims in this paper are well-founded.", "methods_and_evaluation_criteria": "The paper presented in this paper primarily consists of two key modules: the Feature Fusion Module (FFM) and the Feature Representation Module (FRM). The FFM replaces traditional feature-level interactions with fine-grained interactions, which is particularly suitable given the unique characteristics of multispectral images. Meanwhile, the FRM leverages a Mixture of Experts (MoE) framework to balance modality-specific feature representation and structural redundancy, which is also technically sound.", "theoretical_claims": "This paper does not include extensive theoretical proofs; however, there are some ambiguities in the use of mathematical notations. For instance, in Equations (5) and (9), $W_1$ and $W_9$ originate from different modules and should be explicitly distinguished to avoid confusion.", "experimental_designs_or_analyses": "The experiments in this paper utilize Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP) as evaluation metrics. The experimental setup includes comparative experiments (Table 1, 2, 3), ablation studies (Table 4), hyperparameter configurations (Table 5, 6, 7, 8, 9), and visualization (Figure 3). While the experiments are comprehensive, the analysis lacks depth. For instance, Table 3 presents results for M(TIR) without providing a corresponding discussion, in which the performance of MFRNet is lower than TOPReID.", "relation_to_broader_scientific_literature": "The MFRNet in this paper is clearly presented and of good quality. The proposed two modules are built upon the Mixture of Experts (MoE) framework, which is not a novel technique. While MoE has been adopted in several topics existing in the computer vision community, its application to multimodal person re-identification, particularly from both interaction and representation perspectives, still holds meaning.", "essential_references_not_discussed": "This paper could be better by considering published literature in the cross-modality ReID community such as [1, 2]. \n\n[1] Zhang, Yukang, and Hanzi Wang. \"Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[2] Park, Hyunjong, et al. \"Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "other_strengths_and_weaknesses": "Most of the strengths and limitations have been discussed in the previous sections. Overall, the paper is well-structured, with a clear research motivation and methodology. It effectively addresses the stated problem and achieves strong performance. However, a key limitation is that the implementation of the Mixture of Experts (MoE) strategy is simple, lacks significant technical innovation.", "other_comments_or_suggestions": "-The notation for Mean and Std in Equations (12) and (13) is inconsistent in font style. Please ensure uniform formatting for clarity and consistency. \n\n- In Equations (5) and (9), $W_1$ and $W_9$ originate from different modules and should be explicitly distinguished to avoid confusion", "questions_for_authors": "Besides the above mentions, I have one more question: While the authors' claim is convincing, the results in Table 7 do not provide sufficient evidence that the Feature Fusion Module (FFM) requires more experts. It would be beneficial to include a comparison with the case where the number of experts in FFM is set to 1 to better validate this claim.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received four positive ratings, with all reviewers generally inclined to accept it. It presents a Modality Fusion and Representation Network (MFRNet) for multi-modal object re-identification, which aims to address the two challenges of insufficient pixel-level feature interaction and the difficult balance between shared and modality-specific features. According to the reviews, the paper is well-organized with a clear motivation. It provides a comprehensive ablation study demonstrating the contribution and effectiveness of each module. In particular, the experiments show that MFRNet still exhibits a certain degree of robustness to the scenario where the missing mode is not explicitly trained.
The authors have addressed the concerns raised, resolving most of the doubts. Overall, it is a good work, and the Area Chair (AC) recommends accepting the paper.
Decision: Accept (poster)