Empowering Visible-Infrared Person Re-Identification with Large Foundation Models
Authors: Zhangyi Hu
Bin Yang
Mang Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal
large foundation model
Visible-Infrared Person Re-Identification
Primary area: machine_vision

Abstract: Visible-Infrared Person Re-identification (VI-ReID) is a challenging cross-modal retrieval task due to significant modality differences, primarily resulting from the absence of color information in the infrared modality. The development of large foundation models like Large Language Models (LLMs) and Vision Language Models (VLMs) motivates us to explore a feasible solution to empower VI-ReID with off-the-shelf large foundation models. To this end, we propose a novel Text-enhanced VI-ReID framework driven by Large Foundation Models (TVI-LFM). The core idea is to enrich the representation of the infrared modality with textual descriptions automatically generated by VLMs. Specifically, we incorporate a pre-trained VLM to extract textual features from texts generated by VLM and augmented by LLM, and incrementally fine-tune the text encoder to minimize the domain gap between generated texts and original visual modalities. Meanwhile, to enhance the infrared modality with extracted textual representations, we leverage modality alignment capabilities of VLMs and VLM-generated feature-level filters. This enables the text model to learn complementary features from the infrared modality, ensuring the semantic structural consistency between the fusion modality and the visible modality. Furthermore, we introduce modality joint learning to align features across all modalities, ensuring that textual features maintain stable semantic representation of overall pedestrian appearance during complementary information learning. Additionally, a modality ensemble retrieval strategy is proposed to leverage complementary strengths of each query modality to improve retrieval effectiveness and robustness. Extensive experiments on three expanded VI-ReID datasets demonstrate that our method significantly improves the retrieval performance, paving the way for the utilization of large foundation models in downstream multi-modal retrieval tasks.

Empowering Visible-Infrared Person Re-Identification with Large Foundation Models
Zhangyi Hul Bin Yang Mang Ye
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China. {zhangyi hu, www.yanngnnn..c yemang] @whu. edu. https://githe: cww/wH
Abstract
Visible-Infrared Person Re-identification (VI-ReID) is challenging cross-modal retrieval task due to significant modality differences, primarily resulting from the absence of color information in the infrared modality. The development of large foundation models like Large Language Models (LLMs) and Vision Language Models (VLMs) motivates to explore feasible solution to empower VI-ReID with off-the-shelf large foundation models. To this end, we propose a novel Text- enhanced VI-ReID framework driven by Large Foundation Models (TVI-LFM), The core idea to enrich the representation of the infrared modality with textual descriptions automatically generated by VLMs. Specifically, pre-trained VLM to extract textual features from texts generated by VLM and augmented by LLM, and incrementally fine-tune the text encoder to minimize the domain gap between generated texts and original visual modalities. Meanwhile, to leverage
incorporate
enhance the infrared modality with extracted textual representations, modality alignment capabilities of VLMs and VL.M-generated feature-level filters. This enables the text model to learn complementary features from the infrared modality, ensuring the semantic structural consistency between the fusion modality and the visible modality. Furthermore, we introduce modality joint learning align features across all modalities, ensuring that textual features maintain stable semantic representation of overall pedestrian appearance during complementary information learning. Additionally. modality ensemble retrieval strategy is pro- posed to leverage complementary strengths of each query modality improve retrieval effectiveness and robustness. Extensive experiments three expanded VI-ReID datasets demonstrate that our method significantly improves the retrieval performance, paving the way for the utilization of large foundation models in downstream multi-modal retrieval tasks.
Introduction
Person Re-ldentification (Rell)) aims to retrieve images of the same identity across different cameras, which is an important task for intelligent surveillance and urban security [4 Although RGB- based methods [22 16 & 23 33 25 69,
63 68 13 have shown promising results during the daytime, their performance significantly declines night, as RGB cameras fail to capture sufficient information about individuals in low-light conditions. Infrared cameras can capture pedestrian appearances in dark environments. Therefore, Disible-Infraree Person Re-ldentification (VI-ReID) is proposed to match images of individuals captured by visible and infrared cameras, enabling 24-hour surveillance. However, the absence of crucial information, such as color, in infrared images results in significant differences between infrared and visible modalities, posing major challenge for VI-ReID.
55
47
"Equal Contribution.
1 Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous
Ours
Time Cost
Fast Generation
VLM
Labor Cost
Original
Captioon
Rephrased Caption
Automatic Process
L.M
Fixed Amnotation
Dy Dynamic Caption
fr
Agha
Triined Visial Encoder
Param-free Fusson
Eneoder
Netwrk
Rusion
Agha
Acle
Prlor-Info
Driven
Modulle
Emihed
Knowweddge
Prior
Sensttive
Text Fioooler
VLM
Variation
Figure 1: Illustration of our idea. Existing methods rely manual annotations, complex architectures and prior-knowledge-based optimization enrich infrared modality, leading to significant time and labor cost, additional parameters and data sensitivity. In contrast, our method employs VLMs and LLMs to automatically generate dynamic text, enhancing the robustness against text variation. Addtionally, we fine-tune a pre-trained VLM through aligning features across all modalities, enabling the framework to create fusion features semantically consistent with visible modality in parameter- free manner.
Most existing VI-RelD methods, including supervised methods (59
57 38
20
56
60]. semi-supervised [39 45) methods and unsupervised methods 51
primarily focus on mining modality-shared features, while paying less attention to compensating for information absence in the infrared modality, which limits further improvements in cross-modal retrieval performance.
In real-world scenarios, based on the visible modality, human descriptions can provide rich informa- tion, such color, serving vital auxiliary clues. However, existing methods that utilize auxiliary text [9] to enhance the infrared modality, as shown in Fig. [ heavily rely on human annotation to collect fixed text descriptions, resulting in significant time and labor costs. Moreover, they depend prior knowledge, such as pre-defined color vocabularies or hyper-parameters. to design complex loss functions and modules with additional parameters for modality alignment and fusion. This reliance increases sensitivity to data variations and reduces the effectiveness of the fusion process.
Recent advancements in large foundation models 2 particularly LLMs and VLMs, demonstrate significant potential for multi-modal retrieval tasks. This motivates us to explore a feasible solution to complement missing vital information with off aff-the-shelf large foundation models. To this end, propose a Text-enhanced VI-ReID framework driven by Large Foundation Models (TVI-LEM) which comprises Modality-Speciffe Caption (MSC), Incremental Fine-tuning Strategy (IFS), and Modality Ensemble Retrieval (MER). The core idea is to enrich infrared representations with generated text, which is cross- modality retrieval approach bolstered by heterogeneous text descriptions. Specifically, the MSC employs an LLM to augment "L.M-genertted texts, creating dynamic descriptions of visible and infrared images, reducing labor and time costs, while enhancing the model's robustness against text variations. Then, IFS incorporates pre-trained VLM to extract features from text generated by MSC, and incrementally fine-tunes the text encoder minimize the domain gap between the generated texts and the original visual modalities. To enhance the infrared modality with extracted textual representations, IFS leverages the modality alignment capabilities of VLMs and VL.M-generated feature-level filters to create a fusion modality. This enables the text model to learn complementary features from the infrared modality, ensuring semantic structural consistency between the fusion modality and the visible modality. Furthermore, IFS introduces modality joint learning to align features across modalities, ensuring that textual features maintain a stable semantic representation of the overall pedestrian appearance during complementary information learning. Additionally, MER is introduced to leverage complementary strengths of each query modality, forming ensemble queries to further improve retrieval performance. Finally, by integrating the above three modules for dynamic text generation, semantic alignment, and the integration complementary queries, our method effectively addresses the information absence in the infrared modality, significantly improving cross-modal retrieval performance.
The main contributions can be summarized as follows:

---

26 and LLMs such as We design a Text-enhanced VI-ReID framework driven by Large Foundation Models (TVI-LFM). It enriches infrared representations with generated textual descriptions, effectively mitigating the absence of critical information, e.g. color, in the infrared modality and significantly improving the performance of cross-modal retrieval.
We propose the IFS that fine-tunes pre-trained VLM to align generated texts with original images. It creates fusion modality to learn complementary information from the infrared modality and jointly aligns features across all modalities. This ensures a stable semantic consistency of the text, the fusion, and the visible modality during complementary information learning.
We propose the MER that leverages the complementary strengths of all query modalities, forming ensemble queries to further improve the performance of cross-modality retrieval bolstered by heterogeneous text descriptions.
We introduce three extended VI-ReID datasets with VLM-generated textual descriptions for every image. Extensive experiments these expanded datasets demonstrate the competitive performance of our TVI-LFM framework, paving the way for the utilization of large foundation models in downstream multi-modal retrieval tasks.
2 Related Work
2.1 Visible-Infrared Person Re-Identification
VI-ReID aims to retrieve images across visible and infrared modalities, but it suffers from the absence
of critical information, such as color, in infrared modality. Previous methods (59 4 64
27
57
primarily focus on mining modality-shared information and optimizing features extracted by CNNs or Transformers [54] but pay less attention to compensate for the missing vital information in the infrared modality. which limits further improvements in retrieval performance. Some methods 9 5 [ explore auxiliary information compensation. Specifically, [9] uses coarse descriptions as textual identity labels, while [5] and [ integrate attribute embeddings with visual features. These methods heavily rely handcrafted annotations for each identity. Furthermore and require prior knowledge, such as and loss functions. This results in additional parameters and increased sensitivity to variations in auxiliary data. In contrast, we propose a framework that automatically generates dynamic textual descriptions for VI-ReID datasets and fine-tunes pre-trained VLM to align the generated texts with the original images. By leveraging modality alignment capabilities and feature-level filters of the VLM, our framework creates a fusion modality that enables the text model to learn complementary features from the infrared modality, while maintaining semantic consistency with the visible modality.
and pre-defined color vocabulary, to design complex modules
2.2
Large Foundation Model
Large foundation models 12). pre-trained on extensive datasets, have shown great potential in downstream tasks. Recent advancements in VLMs like 44 24 34
67 [8], demonstrate remarkable data generation and visual alignment capabilities. For instance, BLIP (4 excels at generating textual captions from images, and can be fine-tuned to accommodate various image styles such as infrared images. Vicuna 667. pre-trained on extensive text data, is great customized text generation and understanding with prompts. CLIP (34 's pre-training on large-scale image-text pairs enables its basic capability to align text-image semantics. It can also be fine-tuned on downstream moosl retrieval tasks [19] Our approach integrates generative VLMs and L.MSs for automatic text generation and dynamic augmentation. And it incorporates a pre-trained VLM into the VI-RelD system to extract features from texts and utilize them to enrich infrared representations.
35
2.3 Multi-modal Analogical Reasoning in VI-ReID
As demonstrated in 300, language analogical reasoning in the language embedding space can be represented using vector arithmetic. For example, the analogy "man is to woman as king is to ?" can be solved by finding the word whose embedding vector closest to:
T'king" m'man" wooman"
(1)
Subsequently, 21 investigates an identical regularity manifested in the multi-modal vector space. For instance, the relationship can be expressed as:
T Umagerbbus car) - Tuue" + 3.re"" 2 Uimagotred car)-
(2)
We establish this property by aligning features across all modalities, subsequently mapping them into unified embedding space. This enables us to create fusion features in parameter-free free manner.
3

---

Modal Soeaal-peeiillccco Cuption
Incrementul Fine
Eusemble Retrieaal
Modallty Joint Learning
Gallery Ranking list
L Lis Laor
Fluh
Gir
4
Merge
LLM
Len
Augracutation
fi
Filter
Fras
Prye
Figs
Semantic Filtered
145
Grgh
Pask
Pall
Attrivato
Veuttore
Atain
P
f-
Featore
Poge
Figure 2: Illustration of our TVI-LFM, including Modality-Speede Speciffi Caption (MSC), Incremental Fine-tuning Strategy (IFS), and Modality Ensemble Retrieval (MER). MSC utilizes fine-tuned VLMs as modal-specific captioners and employs an LLM for augmentation. IFS fine-tunes pre-trained VLM to create fusion features semantically consistent with visible features. MER leverages the strengths of all query modalities, forming ensemble queries to improve the retrieval performance.
Proposed Method
Task Setting. Humans can provide textual descriptions based on visible modality. Containing critical information, such as colors, they can serve as auxiliary clues to identify individuals. There- fore, we propose the cross-modal retrieval bolstered by heterogeneous text descriptions. Let Negb
D (D D... Y} denote the sample set, where D = (V J= images, D = Nir represents Nir infrared images, and Y = Ji denotes the label set. During inference, the i-th query sample q; consists of an infrared image V text description Tj generated from V,,, and randomly selected text T generated from visible images of the same identity V4, represented qi [V T3 Tighh The gallery contains visible images represented as (Vig)) N₂ 1 Finally, compute similarity ranking lists between each qi and Vgz E G results.
i=1 denotes Nrgb visible
Overview. Detailed in Fig. 2 TVI-LFM contains MSC, IFS and MER. MSC employs two fine-tuned Blips 24 to automatically generate textual descriptions from visible and infrared images and utilizes LLM for augmentation. IFS trains VI-ReID backbone to extract vision features, and incrementally fine-tunes a pre-trained CLIP
to align generated texts with original images. Then, creates fusion modality to learn complementary features from the infrared modality and jointly aligns features across all modalities. It ensures stable semannic consistency of the text, the fusion and the visible modality during complementary information learning. Additionally, MER leverages the strengths of each query modality, forming ensemble queries for more accurate retrieval.
34
3.1 Modal-Specific Caption (MSC)
MSC utilizes fine-tuned VLMs automatically generate text from visible and infrared images and employs off-the-sheif L.LM for textual augmentation, consequently creating dynamic descriptions for VI-RelD datasets. This module reduces the time and labor costs of manual annotations while enhancing the system's 's robustness against the variations in auxiliary text.
VLM based Textual Generation. Currently, there no publicly available large-scale VI-ReID datasets with image-level text annotations. Thus, to reduce labor and time costs, we fine-tune two VLMs to automatically generate text descriptions with critical information, such color, for every visible and infrared image. Consequently. we construct three expanded datasets: Tri-SYSU-MMOI, Tri-LLCM, and Tri-RegDB, each derived from the original datasets [46] 65| 31 respectively.
1) RGB Captioner: First, train a Blip (24) on a large-scale pedestrian image-text dataset [4 as the RGB captioner Grgh, generate text descriptions for each visible image Vig D Digg-
IR Captioner: Then, randomly select visible and infrared images pairs in SYSU-MMOT training split for every identity, then apply the RGB captioner in step I to generate textual descriptions for

---

every visible image in these pairs. Then, remove color-related terms from these generated texts by regular expression filters, and build fetrared-fittered text pairs dataset with filtered text descriptions and corresponding infrared images in the same expanded visible-infrared image pairs. Finally we fine-tune the Blip [24] step again on the Infrared- Text (filtered) dataset as the Captioner Gir. which is able to generate text descriptions without color for each infrared image V E D
3) Text Expanding: Finally, utilize the two refined modality-specific captioners in previous steps to
generate text descriptions Tigo = Grgo (Vigh)) and Tü Grgb(Viig) from original images.
The statistics and samples visualization of the expanded datasets Tri-LLCM, Tri- Tri-RegDB and Tri- SYSU-MM01 are shown Appendix A Through this expansion process, the framework can automatically generate text descriptions for VI-ReID datasets without large-scale manual annotations.
LLM based Textual Augmentation. To ensure that the framework can extract robust representations from generated descriptions against text variations while preserving original semanties of sentences, we propose the I.L.-based textual augmentation module applied during the training stage. This module regenerates diverse descriptions by rephrasing the original text for the same target. In detail, given an original description, the module employs an LLM to rephrase it, producing an augmented textual description. The LLM is guided by the prompt "Rephrase the person's 's description above using similar words. Answer:". The augmentation is applied as follows:
[1m prompt), p T 1-p"
m E (rgb, ir),
where Ty denotes the text descriptions generated from visible or infrared images, Tit represents the augmented text Th and p = 0.5 reflects that each description variant is equally probable. Benefiting from the powerful prompt-driven text generation capability of LLM, this approach diversifies the textual descriptions while maintaining their original meanings. This forces the model to focus on the core person appearance information, thus enhancing the robustness of our system against text variation. Moreover, we can also apply this augmentation method directly on existing frameworks involving text data processing, without changing the original structure.
3.2 Incremental Fine-tuning Strategy (IFS)
IFS incrementally fine-tunes a CLIP 33 based on the frozen visual features extracted by a trained VI-ReID backbone, to minimize the domain gap between the generated texts and original visual modalities, namely, to align the complementary feature across vision and text, thereby creating fusion features semantically consistent with visible modality. The detailed steps are follow:
Features Extraction. During the training stage, each batch contains NB selected quads as inputs, Na
2=1- Here, Ti and Tig. are augmented texts generated from the i-th image pair V and righ- which is randomly selected belonging to identity ys. IFS first utilizes the Channel Augmentation (CA) 57 strategy to train a dual-stream ResNet-50 [15] as the VI-RelI)
denoted as {(Vi. Vi Tit Trg&) )
rrgh
backbone, detailed in Appendix B Then utilize the trained backbone to extract infrared features f!(())
and visible features fry0 denoted as:
[(()) = PF and SIO = F (F (Vig.)):
where FRGB and F denote the visible/infrared specific layers respectively. F$ denotes the shared layers of the visual backbone. Meanwhile, IFS incorporates CLIP (34] FV FVLM to extract the "text
features" f2 from visible images texts and "filter features" (3) from infrared images texts:
frgb =FY
T()
FVLM (Tigh) and f T()) =F (T
Feo
Semantic Filtered Fusion (SFF). Meanwhile, to enhance the infrared modality with the generated texts, we propose the SFF module that leverages the text-visual alignment capability of VLM and VI.M-generated filter features to create fusion features.
Benefiting from the large-scale pre-training image-text pairs, the CLIP 34 possesses powerful text-visual alignment capability, ensuring that the features extracted from the generated texts can T(i)
be aligned with the features of original images. Therefore, the text feature STgb can work as an
alternative for the visible feature fl(()) Similarly, the filter feature f (1) can work as an alternative for
5

---

Target description
boy short
boy with
weening rund
wearing a mund reck.
Prgo
short sleeves alitt
pray
with jagas pants. pairod atres,
jogaer paired black bag back.
frr
fir
Frge
Figure 3: The Visualization of SFF. With the aligned features of generated texts and original images, SFF creates fusion features semantically consistent with visible modality by arithmetically adding the textual complementary information for infrared modality the infrared features.
the infrared feature f(r Next, we formulate the complementary features for the infrared modality by decomposing the visible feature Frih and the text feature T as:
Fryb = (3)) +
((()) and Sryb = f T()) comp
comps
(6)
where fommp denotes the visual complementary feature for the infrared modality. Similarly, Foomp denotes the textual complementary feature for the infrared modality. Finally, with Eq. ( and the alignment of features of generated texts and original images, benefiting from the pre-trained CLIP, the representation of fusion features f with the same semantic structure as the visible modality can be derived, represented as:
f S/60
S/(())
comp
(S
(Prgh
(T)))
(7)
FT
FF
By leveraging the powerful text-visual alignment capability of CLIP and VLMMgennrated filter features, SIF approximates f/ - - with f-16" - Thus, the framework can create fusion features by adding this textual complementary features to the infrared features, as shown in Fig This allows the text model to learn complementary features from the infrared modality, while ensuring semantic structural consistency between the fusion modality and the visible modality.
Modality Joint Learning (MJL). Furthermore, MJL is proposed to optimize the pre-trained VLM. incrementally fine-tunes a CLIP text encoder 1% L-YLM based on the infrared and visible features extracted from the frozen VI-ReID backbone with parameters 01, fg and 63, thus further aligning the fusion modality and the visible modality. This training strategy and its effectiveness in avoiding conflicts during the visual and textual part representation learning are discussed in the Appendix
This method utilizes classic RelD loss for fine-tuning, which eliminates the prior knowledge reliance, such as hyper-parameters and predefined vocabulary, during optimization. The loss consists of cross-entropy loss Lag and weighted regularized triplet loss Twerz (59], denoted
Lettal = L.a B": ;8:))))))))))))))) T.ort B3; (b), B" {(Frrg
T() Fryb f (F((), (.)))
(8)
where Og denotes the trainable parameters of VLM, each batch B contains the frozen visible
feature f!(()) the frozen infrared feature (()) the trainable text feature f the trainable fusion
feature (P(() and identity label yı. During optimization, MJL pulls all these features belonging to the same identity ys together while pushing them away from whom belonging to Vy, E {vly # yc aligning the semantics across all modalities. Since we solely optimize the F. FYLM this process can be considered as aligning the text features and the fusion features with visible features, under the supervision an identity label y and the regularization from infrared features.
By aligning the textual complementary features for infrared modality TTii) integrated the fusion features with the corresponding part f in the visible features, MJI effectively reduces the discrepancy between fusion and visible modality. Additionally, it ensures the overall semantic
T() consistency of text features Frgo with visible features f/(() during complementary information
learning, which enables the following MER to form effective ensemble queries.

---

Thereby, considering the established two alignments above while taking into account the feature decomposition in Eq. 66. the alignment between infrared features ((()) and filter features fir can be derived and represented as:
fir f1gh)
((t)
f
comp
rgb
ccomp'
(9)
(()
Consequently, by establishing the aligmment between generated texts with original images, we successfully minimize the domain gap between the generated texts and original visual modalities, namely, align each part of the visual and textual complementary features, thereby enabling the alignment between fusion features and the visible features, leading improved retrieval accuracy.
According to Eq. (6). (T can be decomposed as trainable f((()) and foomp, while f can be decomposed as frozen fir and trainable f Considering that frozen infrared features f(( also participate in aligning with the text features fr and the fusion features fP((), it can be regarded as
TG)) regularization to constrain the complexity of textual complementary features fromp while aligning with filter features fir enabling the framework to construct better fusion features.
3.3 Modality Ensemble Retrieval (MER)
MER leverages the complementary modalities strengths and rich semantics in query representations mined from IFS to create an ensemble query feature f(f) for more accurate retrieval, represented as:
(fl frgb + /
(10) where fusion feature F() provides a combined feature with semantic structure the same as the visible features, serving as the primary matching modality. Infrared feature f((() provides contiguous visual semantics. Their similarity with visible images can serve as a supplementary reference for texture and shape information. Text feature T(() aligned with visible features in MJL, provides descriptive information such as the color and the pattern of the clothes. The similarity between text features and visible features serves as reference for these key matching features. Therefore, when encountering challenging scenarios that are hard to distinguish person's 's clothing or shape but distinguishing color is feasible, or vice versa, the ensemble query feature f(²) can leverage the features of two modalities in addition to the primary matching contribution from fusion modality to explore their similarities in visual texture and key attributes respectively. Thus, the similarity score
q
Siy between i-th query sample and j-th gallery sample is defined as f( F Thus, the strengths of all query modalities can be integrated into the final similarity score, consequently enhancing the accuracy against hard cases of retrieval. In fact, by plugging Eq. (10) into fa f an equivalent definition for the similarity between two high- digeesiinnioon features is derived, represented as:
8ij (f 1(0) ST
+F(()) (
(11)
concat (()
( ST FF((] - concat (( Sigh S 1/33
is equivalent to increasing the feature dimension for retrieval but use ensemble features with less dimensions and computational cost. Due the larger distances between classes in higher dimensional feature space, the models can distinguish features of different identities the ensemble feature space more easily, therefore utilizing ensemble queries can further improve the retrieval performance.
4 EXPERIMENTS
4.1 Experimental Settings
Datasets. We evaluate our framework on Tri-SYSU-MM01, Tri-RegDB, and Tri-LLCM. These datasets with text descriptions for each visible and infrared image expanded from the original VI-ReID datasets [46] 65)
(5,311 31], utilizing fine-tuned Blip (2 as captioner. The splits of the training
set and testing for each dataset are available in Appendix D
Evaluation Protocols. line with established VI-RelD settings [59 we assess the performance of the infrared query and the fusion query using Rank-k matching accuracy, mean Average Precision
7

---

Table 1: Ablation study fusion query (I + R) about each component the performance of
Tri-SYSU-MM01 and Tri-LLCM datasets. Rank-1 (%), mAP(%) and (%) are reported.
7+ T + R
Tr--YYUU-MOI
Tri-LICCM
SFF
MUL
LLM
MER
R1
mAP
mINP
mAP
mlNP
72.52
69.15
55.93
52.63
58.82
55.43
77.00
73.73
61.50
54.73
60.95
57.64
83.97
80.40
69.46
56.76
63.58
60.35
/
84.17
80.72
70,02
57.13
64,06
60.72
84.88
81.32
70.57
57.09
63.87
60.62
84,90
81.47
70.85
58.19
65.08
61.83
(mAP), and mean Inverse Negative Penalty (mINP) 159 within our TVI-LFM framework. To get stable performance on Tri-SYSU-MME and Tri-LLCMM, we evaluate our model 10 times with random splits of the gallery set; as for Tri-RegDB, we evaluate our model on trials with different train/test splits and report the average performance on each dataset.
Implementation Overview. We utilize dlual-strean ResNet-50 [58] pre-trained on ImageNet [6) as the visual backbone and transformer in CLIP R4 as the textual encoder. Training involves visible and infrared images alongside text descriptions generated by two modality-specialized fine-tuned Blip [24] models. All text descriptions are augmented by vicuna-7b [67] with random rephrasing strategy. Incremental fine-tuning is applied by fixing the visual parameters while tuning the textual part of the framework. All details are described in Appendix B
4.2 Ablation Study
To thoroughly evaluate the effect of each component of our proposed method, we conduct compre- hensive ablation studies on Tri-LLCM and Tr--SYS--MM0L. These studies involve gradually adding the proposed modules to baseline, systematically removing specific modules from our framework and assessing their impact on performance. The overall experimental setup remained consistent, with only the module under evaluation being modified.
Effect of Semantic Filtered Fusion. By leveraging text-visual modality alignment capability of VLM and VLM -egeeeere filter features, SEF fuses textual complementary information with infrared features to create fusion features semantically consistent with visible modality. Compared to the baseline, the method obtains 4.48% Rank-1 improvement in Tr-SSYS-MMOI and 2. 10% Rank-1 I improvement in Tri-LLCM, as shown in Table The results demonstrate that the module effectively integrates information from different modalities.
Effect of Modality Joint Learning. In cooperation with SFF, MJL aligns features across all modalities to minimize the domain gap between the generated texts and original visual modalities, thereby mitigating the fusion-visible modality difference. Based on the experimental results in Table and compared to the baseline with SFF, adding MJL gains significant enhancement 6.97% Rank-T improvement, 6.67% mAP improvement, and 7.96% mINP improvement in TU--SYSU-MMO), and 2.03% Rank-1 improvement, 2.63% mAP improvement, and 2.71% mINP improvement Tri-LLCM. This result demonstrates the effectiveness of MJL, which greatly minimizes the modality gap.
Effect of Modality Ensemble Retrieval MER leverages the complementary advantages of different query modalities to construct ensemble query features thereby improving the retrieval accuracy. The results demonstrate its effectiveness, in Table I incorporating MER provides an additional improvement of 0.91% % in Rank-1, 1, 0.92% in mAP, and 1.11% in mINP in the Tri-SYSU-MM01 dataset over the baseline with MJL+SFF. Similarly, on the Tri-LLCM dataset, MER achieves 0.33% Rank-1 improvement, 0.29% mAp improvement, and 0.27% mINP improvement.
Effect of LLM based Textual Augmentation. To extract robust text representations against text variation. we implement LLM based augmentation module by randomly rephrasing the generated descriptions for dynamic expression. As shown in Table [ incorporating further improves the overall performance and robustness against text variation. It also works well with other modules, achieving 84.90% Rank-1 and 58.19% Rank-1 in Tri-SYS--MM0 and Tri-LLCM respectively.
4.3 Comparison with the State-of-the-art Methods
We present a comprehensive comparison TVI-LFM against state-of-the-an methods on different Our evaluation includes a variety of metrics: Rank-
datasets as outlined in Table 2 and Table g
8

---

Table 2: Comparison with the state-of-the-an methods the proposed Tri-SYSU-MMOL.
All Search
Indoor Search
Methods
Venue
Type
R-1
mAP
mINP
R-1
mAP
mlNP
Zero-Padding
46
ICCV-17
14.80
15.95
20.58
26.92
HCML [6
AAAI-18
14.32
16.16
24.52
30.08
emGAN
BCCA--1
27.80
42.19
AlignGAN
4R]
ICCV-19
42.40
40.70
45,90
54,30
AGW
TPAMT-21
47.65
35.30
62.97
59.23
DDAG 55
ECCV-20
54.75
53.02
39.62
61,02
67.98
62.61
CM-NAS
ICCV-21
1
61.99
60.02
67.01
72.95
DART 33
CVPR-22
68.7
66.3
82.0
73.8
CAJ 57
ICCV-21
69.88
66.89
53.61
76.26
80.37
76.79
DEEN 65
CVPR-23
74.70
71.80
80.30
83.30
SAAT 10
ICCV-23
75.90
77.03
83.20
88.01
MSCLNet 69
BCCV-22
76.99
71 64
78. 78.49
81.17
SGIEL 11
CVPR-23
77.12
72.33
82.07
82.95
PartMix 20
CVPR-23
77.78
74.62
81.52
84.38
YYDS g
Arxiv-24
1+T R
74,60
70.35
56.01
81,35
83.64
79.56
VI-ReID Backbone
1 R
69.89
66.74
53.34
76.91
80.64
76.70
TVI-LFM
I+T
84.90
81.47
70.85
89.06
90.78
88.39
Table 3: Comparison with the state-of-the-arl methods on the proposed Tri-RegDB and Tri-LLCM.
Tri- BBi-RegDB
Tri-LLCM
Methods
Venue
Type
mAP
mINP
R-1
mAP
mlNP
DDAG 58
ECCV-20
68.06
61,80
48.62
40.3
48,4
AGW
TPAMI-21
70.49
51.24
43.6
CAJ 57
ICCV-21
84.8
77.8
61.56
48.8
56,6
I R
DART
CVPR-22
82.0
73.8
52.2
MMN 66
MM-21
87.5
80.5
52.5
58.9
DEEN R5
CVPR-23
89.5
83.4
54.9
62.9
YYDS
9
Arxiv-24
11T R
90.95
84.22
70.12
58.13
64.91
61.77
VI-ReID Backbone
I I-> R
89.51
83.51
69.65
53.53
59.77
56.40
TVI-LFM
I+T
85.92
72.73
58.19
(R-1), mean Average Precision (mAP), and mean Inverse Negative Penalty (mINP) [59]. For fair comparison, re-run YYDS on the proposed expanded datasets with the same image size: 288x 144.
Performance on Tri-SYSU-MM01 Dataset As shown in Table with the enhancement of generated text, TVI-LFM greatly improves the performance of the VI-ReID backbone and outperforms all previous methods under All Search' and Indoor Search' conditions. Specifically, TVI-LFM achieves significant improvements in Rank-1, reaching 84.90% and 89.06% respectively, compared to the next best result of 77.78% by PartMix (20) in All Search and 83 83.20% by SAAI [10] in Indoor Search. Furthermore, in terms of mAP, TVI-LFM posts scores of 81.47% and 90. 90.78%, which are substantial increases from the previous high scores 77.03% and 88.01%, %, respectively.
Performance Rri-RegDB and Tri-LLCM Dataset Table outlines method's performance on the two datasets. the Tri-RegDB dataset, TVI-LFM obtains a Rank-1 of 91.38% and an mAP of 85.92%, higher than the prior top scores of 90.95% in Rank- and 84.22% in mAP by YYDS. In the Tri-LLCM dataset, our method leads with a Rank- of 58. 19% and an mAP of 65.08%, surpassing the prior top scores of 58.13% in Rank-1 1 and 64.91% % in mAP, both held by YYDS.
4.4 Visualization
Feature Distribution Visualization. To explore the reason why our method is effective, utilize t-SNE [42] 2D feature space and visualize cosine distances of the intra-class and inter-class features on TTi-SYSU-MMOL dataset. From the (a) to (d) in Fig. the t-SNE feature distribution shows that our method greatly enhances the ability of distinguishing features from different identities with text and reduces extreme outliers
same identity and samples with too large cross-modal discrepancy, For the feature distance distribution shown in Fig. 4 (e-h), which corresponds to the 2D t-SNE 42 feature distribution, the inter-class and intra-class distance distributions are increasingly well separated, particularly noting that the excessive intra-class distance is also significantly reduced.

---

(a) Initial Distribution
(b) VI-ReID Backbone Distribution
(c) Bascline Distribution
(d) TVI-LFM Distribution
Iatur
(c) Initial Distance
(1) VI-ReID Backbone Distance
(g) Buseline Distunce
TVI-LFM Distance
Figure 4: First row (a-d) shows the t-SNE feature distribution of the 20 randomly selected identities, triangles means infrared features (w/wo textual enhancement), circles means visible features. Differ- ent colors indicate different identities. Figures in the second row (c-h) represent the intra-class (blue) and inter-class (green) distances of infrared features (w/wo textual fusion) and visible features.
Retrieval Result. To intuitively present the performance of our method, we visualize some retrieval results of the VI-RelI) backbone, baseline and our method the Tri-SYSUMMM01 dataset in Appendix C For the same query image, our method significantly enhances retrieval performance utilizing generated descriptions compared baseline and VI-ReID backbone.
Conclusion
To alleviate the absence of detailed color information in the infrared modality, this paper presents a VI- RelD framework driven by Large Foundation Models (TVI-LFM) to enrich the infrared representation with VLM -en-eeeeertted textual descriptions, which is moddltty retrieval approach bolstered by heterogeneous text descriptions. To enhance the infrared modality with text, MSC utilizes one off-the-shelf LLM augment VLM-ggnerated text descriptions. Then, IFS incorporates a pre-trained
VLM extract features from generated texts, and incrementally fine-tunes the text encoder to align generated texts and original visual modalities. enhance the infrared modality with extracted text features, IFS leverages modality alignment capabilities of VLMs and VL.M-generated feature-level filters to create fusion modality. This enables the text model to learn complementary features from the infrared modality, ensuring semantic structural consistency between the fusion modality and the visible modality. Furthermore, IFS introduces modality joint learning to align features of all modalities, maintaining stable semantic representation of the overall pedestrian appearances for text features during complementary information learning. Additionally, MER leverages complementary strengths of query modalities, forming ensemble queries further improve retrieval performance. Extensive experiments on three expanded VI-ReID datasets demonstrate the competitive performance of TVI-LFM, paving the way for the utilization of large foundation models in downstream multi- modal retrieval tasks.
Limitations and future research. While the proposed TVI-LFM shows promising performance, the retrieval accuracy hinges on text quality, and the performance on hard datasets, such as LLCM(65). still have rooms for improvement. High-quality text enhances retrieval accuracy by improving text- vision correspondences during training and providing precise information for infrared compensation during inference. Therefore, for future improvements, I) image augmentations during generator fine-ture fine-tuning; 3) progressive generation strategies focusing on ien-ggrained attributes, could be introduced enhance text quality, thereby improving the accuracy of cross-modality retrieval bolstered by heterogeneous textual descriptions.
advanced generative models; 2)
Acknowledgements. This work is partially supported by National Natural Science Foundation of China under Grant (621761 76188, 62361 166629, 13),,113). Key Research and Development Project of Hubei Province (2022BADI75) and Postdoctoral Fellowship Program of China Postdoctoral Science Foundation (GZC20241268) The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.
10

---

References
[1] Zheng Aihua, Pan Peng, Li Hongchao, Li Chenglong, Luo Bin. Tan Chang, and Jia Ruoran. Progressive attribute embedding for accurate cross-modality person re-id. In ACM MM, 2022.
[2] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, and et. al. On the opportunities and risks of foundation models. ArXiv, 2022.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, and et. al. Language models are few-shot learners. In NeurlPS, 2020.
[4] Cuiqun Chen, Mang Ye, Mcibin Qi. Jingjing Wu, Jianguo Jiang, and Chia-Wen Lin. Structure- aware positional transformer for visiblc-infrared person re-identification. IEEE TIP, 2022.
[5] Zhuxuan Cheng, Huijie Fan, Qiang Wang, Shiben Liu, and Yandong Tang. Dual-stage attribute embedding and modality consistency learning-base baaeed visible-infrared person re-identification. Electronics, 2023.
[6] Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, and Yuyu Huang, Cross-modality person re-identification with generative adversarial training. IJCAI, 2018.
[7] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpon vision-language models with instruction tuning. ArXiv, 2023.
[8] Neng Dong, Shuanglin Yan, Hao Tang, Jinhui Tang, and Liyan Zhang. Multi-view information integration and propagation for occluded person re-identification. IF, 2024.
[9] Yunhao Du, Zhicheng Zhao, and Fei Su. Yyds: Visible-infrared person re-identification with coarse descriptions. ArXiv, 2024.
[10] Xingye Fang, Yang Yang, and Ying Fu. Visible-infrared person re-identification via semantic alignment and affinity inference. In ICCV, 2023.
[11] Jiawei Feng, Ancong Wu, and Wei-Shi Zheng. Shape-erased feature learning for visible-infrared person re-identification. In CVPR, 2023.
[12] Chaoyou Fu, Yibo Hu, Xiang Wu, Hailin Shi, Tao Mei, and Ran He. Cm-nas: Cross-modality neural architecture search for visible-infrared person re-identification. In ICCV, 2021.
[13] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang. Houqiang Li, and Dong Chen. Unsupervised pre-training for person re-identification. CVPR, 2021.
[14] Yunpeng Gong, Liqing Huang, and Lifei Chen. Person re-identification method based color attack and joint defence. In CVPR, 2022.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[16] Shuting He, Hao Luo, Pichao Wang. Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In ICCV, 2021.
[17] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, 2017.
[18] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, and et. al. Mixtral of experts. ArXiv, 2024.
[19] Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. In CVPR, 2023.

---

[20] Minsu Kim, Seungryong Kim, Jungin Park, Seongheon Park, and Kwanghoon Sohn. Partmix: Regularization strategy learn part discovery for visible-infrared person re-identification. In CVPR, 2023.
[21] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. ArXiv, 2014.
[22] He Li, Mang Ye, Cong Wang. and Bo Du. Pyramidal transformer with conv conv-patchify for person re-identification. In ACM MM, 2022.
[23] Huafeng Li, Yiwen Chen, Dapeng Tao, Zhengtao Yu, and Guanqiu Qi. Attribute-alined domain-invariant feature learning for unsupervised domain adaptation person re-identification. IEEE 77FS, 2021.
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-image pre-training for unified vision-language understanding and generation. ArXiv, 2022.
[25] Mingkun Li, Chun-Guang Li, and Jun Guo. Cluster- Cluster-guided asymmetric contrastive learning for unsupervised person re-identification. JEEE TIP, 2022.
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[27] Jialun Liu, Yifan Sun, Feng Zhu, Hongbin Pei, Yi Yang, and Wenhui Li. Learning memory- augmented unidirectional metrics for cross mods--mo person re-identification, In CVPR, 2022.
[28] Min Liu, Yeqing Sun, Xueping Wang, Yuan Bian, Zhu Zhang, and Yaonan Wang. Pose-guided modallty-invariant feature alignment for visible infrared object re-identification. IEEE TIM, 2024.
[29] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. CVPR, 2019.
[30] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In ACI., 2013.
[31] Dat Nguyen, Hyung Hong, Ki Kim, and Kang Park. Person recognition system based on combination of body images from visible light and thermal cameras. Sensors, 2017.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, and et. al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
[33] Xuelin Qian, Yanwei Fu, Tao Xiang, Wenxuan Wang, Jie Qiu, Yang Wu, Yu-Gang Jiang, and
X. Xue. Pose-normalized image generation for person re-identification. In ECCV, 2017.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI. 2021.
[35] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. IJCV, 2014.
[37] Jiangming Shi, Xiangbo Yin, Yeyun Chen, Yachao Zhang, Zhizhong Zhang, Yuan Xie, and Yanyun Qu. Multi-memory matching for unsupervised visible-infrared person re-identification. In ECCV, 2024.
[38] Jiangming Shi, Xiangbo Yin, Yachao Zhang, Zhizhong Zhang, Yuan Xie, and Yanyun Qu. Learning commonality, divergence and variety for unsupervised visible-infrared person re- identification. ArXiv, 2024.
12

---

[39] Jiangming Shi, Yachao Zhang, Xiangbo Yin, Yuan Xie, Zhizhong Zhang, Jianping Fan, Zhongchao Shi, and Yanyun Qu. Dual pseudo-labels interactive self self-training for semi-supervised visible-infrared person re-identification. In ICCV, 2023.
[40] V Subramanyam. Niranjan Sundararajan, Vibhu Dubey, and Brejesh Lall. littd-20k: Dense captioning for text-image reid. ArXiv, 2023.
[41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, and al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 2023.
[42] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. JMLR, 2008
[43] Guan' an Wang. Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, and Zengguang Hou. Rgb- infrared cross modslityy person re-identifiiation via joint pixel and feature alignment. In ICCV, 2019.
[44] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-texx transformer for vision and language. ArXiv, 2022.
[45] Ziyu Wei, Xi Yang, Nannan Wang, and Xinbo Gao. Semi-supervised learning with heteroge- neous distribution consisteney for visible infrared person re-identification. IEEE TIP, 2024.
[46] Ancong Wu, Wei-Shi Zheng, Hlong-Xing Yu, Shaogang Gong, and Jianhuang Lai. Rgb-infrared cross-modality person re-identification. In ICCV, 2017.
[47] Dongming Wu, Mang Ye, Gaojie Lin, Xin Gao, and Jianbing Shen. Person re- re-deenifiicatinn by context-aware part attention and multi-head collaborative learning. IEEE TIFS, 2022.
[48] Zesen Wu and Mang Ye. Unsupervised visible-infrared person re-identification via progressive graph matching and alternate learning. In CVPR, 2023.
[49] Bin Yang. Jun Chen, Cuiqun Chen, and Mang Ye. Dual consistency learning for unsupervised visible-infrared person re-identification. IEEE TIFS, 2024.
[50] Bin Yang. Jun Chen, and Mang Ye. Towards grand unified representation learning for unsuper- vised visible-infrared person re- re-identification. In ICCV, 2023.
[51] Bin Yang. Jun Chen, and Mang Ye. Shallow-deep collaborative learning for unsupervised visible-infrared person re-identification. In CVPR, 2024.
[52] Bin Yang, Mang Ye, Jun Chen, and Zesen Wu. Augmented dual-contrastive aggregation learning for unsupervised visible-infrared person re-identification. ACM MM, 2022.
[53] Mouxing Yang, Peng Huang Zhenyu, Hu, Taihao Li, Jiancheng Lv, and Xi Peng. Learning with twin noisy labels for visible-infrared person re-identification. CVPR, 2022.
[54] Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng, David Crandall, and Bo Du. Transformer for object re-identification: A survey, ArXiv, 2024.
[55] Mang Ye, Xiangyuan Lan, and Qingming Leng. Modality-aware collaborative learning for visible thermal person re-identification. In ACM MM, 2019.
[56] Mang Ye, Xiangyuan Lan, Jiawei Li, and P C Yuen. Hierarchical discriminative learning for visible thermal person re-identification. In AAAI, 2018.
[57] Mang Ye, Weijian Ruan, Bo Du, and Mike Zheng Shou. Channel augmented joint learning for visible-infrared recognition. In ICCV, 2021.
[58] Mang Ye, Jianbing Shen, David Crandall, Ling Shao, and Jiebo Luo. Dynamic dual-attentive aggregation learning for visible-infrarde person re-identification. In ECCV, 2020.
13

---

[59] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. Deep learning for person re-identification: A survey and outlook. IEEE TPAMI, 2022.
[60] Mang Ye, Zesen Wu, Cuiqun Chen, and Bo Du Channel augmentation for visible-infrared re-identification. IEEE TPAMI, 2024.
[61] Xiangbo Yin, Jiangming Shi, Yachao Zhang, Yang Lu, Zhizhong Zhang, Yuan Xie, and Yanyun Qu. Robust pseudo-label learning with neighbor relation for unsupervised visible-infrared person re-identification. ArXiv, 2024.
[62] Zhiming Luo Yansong Qu Rongrong Ji Min Jiang Yunpeng Gong, Zhun Zhong, Cross-modality perturbation synergy attack for person re-identification. ArXiv, 2024.
[63] Xuanmeng Zhang, Minyue Jiang, Zhedong Zheng, Xiaoping Tan, Errui Ding, and Yi Yang. Understanding image retrieval re-ranking: graph neural network perspective. ArXiv, 2020.
[64] Yiyuan Zhang, Sanyuan Zhao, Yuhao Kang, and Jianbing Shen. Modality synergy complement learning with cascaded aggregation for visible-infrarce person re-identification. In ECCV, 2022.
[65] Yukang Zhang and Hanzi Wang. Diverse embedding expansion network and low-light cross- modality benchmark for visible-infrared person re-identification. In CVPR, 2023.
[66] Yukang Zhang, Yan Yan, Yang Lu, and Hanzi Wang. Towards unified middle modality learning for iisileeinfrrared person re-identification. In ACM MM, 2021.
[67] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. 7i Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-hench and chatbot arena. NeurIPS, 2023.
[68] Zhihui Zhu, Xinyang Jiang, Feng Zheng, Xiaowei Guo, Feiyue Huang, Weishi Zheng, and Xing Sun. Viewpoint-await aware loss with angular regularization for person re-identification. ArXiv, 2019.
[69] Chang Zou, Zeqi Chen, Zhichao Cui, Yuehu Liu, and Chi Zhang. Discrepant and multi-instane instance proxies for unsupervised person re-identification. In ICCV, 2023.
14

---

A Details of Expanded Datasets
black holk. seertigg black tull kerves oner oneruced that deramm yumss mutched Ias clethess with shoes"",
with her meening a full sleass lenggh shirt polred with full langth auass
wich Brown longthy oper carrying on complete [Eckes, palrod with biue jeons pants welll sperting ginng an black of her right shoulder.
ledy with enn deter sporting cristing comp pireeeee... -deesed topy ddlition an ancake's. ap sportng an cummen wing bag
blue
man with short hair weening ing half diocve dark round- shar poined with black length jeams pants shows, he also weering pewringgaane glasses
man Ei hair weating a ocreet, shvi meteefolti parts he has menthed cerrying on shouller.
a boy with short black har wearing true shirt pared with black carto followed by graphic shoes, carrying black backpack on tas
hoy shoet hain eearii g.a half sleeved round. neck shin, shorts pa ned wet cocual gh oooss i Weveing heripare shouldeer.
shoes.
Figure 5: Visualization of the data samples selected from the expanded three datasets.
Table 4: Dataset statistics
Datasets
#ID
#RGB
#IR
#Text
Tri-LLCM
1064
25626
21141
46767
Tri-RegDB
412
4120
4120
8240
Tri-SYSU-MM01
491
30071
15792
45863
All the fine-tuning process of VLMs can be found in documentations from huggingface: https
thuggingface. co/ co/docs/ ttt//transfrrrroo wwtssformer/mmmee watrss/manneeeee
The generator model we use refers to the official implementation released in huggingface: https wttp:aaaccc.caaaasfeeeeeeeeeeeee htp.porr//oppppmaaoooooooooooagg
captioning-oaigge
B Implementation Details
We implement our framework in PyTorch 33 utilizing single NVIDIA RTX 3090 GPU for training. For visual backbone training, it takes about 9GB memory for training and about 3GB memory for testing, about 9 hours are needed for training on Tri-SYSU-MMO and Tri-LLCM, about hour for smaller TBi-RegDB. For incremental fine-tuing, it takes about 5GB memory for training and about 3GB memory for testing, about hour are needed for fine-tuning on Tri-SYSU-MMO1 and Tri-LL.M, about 10 miniutes for smaller Tri-RegDB. Each batch consists of 8 identities, with each identity containing 4 visible images, 4 infrared images, 4 text descriptions generated from visible images, and 4 text descriptions generated from infrared images. All input images resized to 3 x 288 x 144, with full augmentation strategy the same as CAJ [57]. All text descriptions are generated by two modality-spenn specialized fine-tuned VLMs and augmented by the proposed LLM rephrasing augmentation with a probability of 0.5, here we use vicuna- 7b [67] as our LLM model, use Blip 24 as VLM model, whose tuning process can be found in Sec. 3.1 We employ dual-stream resnet50 model (58] pre-trained on ImageNet (36) as the visual backbone and a transformer model with parameters derived from CLIP (34] the textual backbone. For incrementally fine-tuning our TVI-LFM,, at first, we should get an available well-trained visual backbone. Here we utilize the augmentation method (57) to train the visual backbone for 120 epochs by cross-entropy loss and weighted regularized triplet loss [59]. finally get the well-trained visual backbone. Then we integrate the well-trained VI-ReID model and fine-tune the text encoder from CLIP (34) and simple ReID bottleneck (29) applied for each feature for 20 epochs. We use the Adam 11 for optimization. For the TYU-SYSU-MO01 and Tri-LLCM datasets, in both visual and textual parts, the learning rate is set to 3.5e-4 and the weight decay to 5e-4. For the Tri-RegDB dataset, the learning rate for the visual part is 2 with weight decay of 5e-4, and for the textual part, the learning rate le-5 with weight decay of 4e-5. The learning rate rises up to the initial value by a linear warm-up scheme for the first 10 epochs, then decays by a linear scheme with a decay-facta of 0. 1. at the milestones of 40, 60, and 100 epochs.
15

---

C
Retrieve Result Examples w/wo Text
Method
Target Description
Generated Filter
IR Overy
Results
VI-RelID Backbone
N/A
N/A
short black hair boy wearing short sleeves t shirt with grey jogger pants, paired with green shoes...
Baseline
N/A
Concecceeee ceceesss Seceeeeees...................... short black hair boy wearing short sleeves red t shirt with grey jogger pants, paired with green shoes...
short hair boy wearing short sleeves t shirt
Qvery
Top
Ours
with jogger pants, paired with shoes
Method
Target Description
Generated Filter
Results
VI-ReID Backbone
N/A
N/A
man with black hair, wearing green half sleeves
Quary
Baseline
shirt that he paired with
N/A
dark black colour
man with black hair, wearing green half sleeves t- shirt that he paired with
with hair, wearing a half sleeves
Ours
-siirt that he paired with
dark black colour
Figure 6: Visualization of the rank-5 retrieval results obtained by the VI-ReID backbone, the baseline, and our method on the proposed Tri-SY MM-0Y.
The VI-ReID backbone and the baseline still includes misidentifications. But method fully leverages complementary information from textual data, significantly enhancing retrieval performance through semantic filtered fusion.
D Assets Details
This section provides the necessary details for the data assets utilized in our research: SYSU-MM01, LLCM, and RegDB.
SYSU-MM01 46
The SYSU-MM01 dataset was created by researchers at Sun Yat-sen University (SYSU). Ancong Wu, et al. "RGB-IR Person Re- Re-Identiiication by Cross-Modality Similarity Preservation" (2020) the seminal paper associated with this dataset.
Source and Citation:
data splits: The training set contains 22,258 visible images and 11,909 infrared images of 395 identities. The testing set contains 96 identities, with 3,803 infrared images for query and 301 (single-shot) randomly selected visible images as the gallery set.
URL: The dataset be accessed through GitHub repository: httpss:/ggiit SSYSU-MM111 MMO1, where users must agree to the data release agreement.
We cannot find the license SYSU-MMOI uses, but the author requires signing the usage agreement notice and contact him through e-mail to get the dataset. The detailed usage agreement refers to the github url mentioned above.
License:
16

---

LLCM 65
Source and Citation: The LLCMM dataset was introduced by researchers from Xiamen University. Yukang Zhang and Hanzi Wang's 's paper "Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible- Infrared Person Re- identification" (2023) discusses this dataset.
data splits: The training set contains 30,921 images of 713 identities, and the test set contains 13,909 images of 351 identities.
URL: The dataset is available on GitHub httffbbbttts:://iihtuu http://ttt...coo2///.0//////////
License: CC-BY 4.0
Code: We use its code for feature visualization.
RegDB 31
Source and Citation: The RegDB dataset was developed at Dongguk University from the paper named "Person Recognition System Based on a Combination of Body Images from Visible Light and Thermal Cameras".
data splits: The training set contains 206 identities and the testing set contains 206 identities. There are 10 visible images and 10 infrared images for each person.
-URL: We only find the paper's 's doi https: org/ 030..339///00 s1703060
License: CC-BY 4.0
E
Discussion of Incremental Fine-tuning Strategy
Table 5: The impact of incrementally fine-tuning the framework based frozen, well-trained visual backbone versus training from scratch is evaluated for two scenarios: infrared query (I + RR and fusion query (I + T R) the performance of Tri-SYSU-MMO1 and Tri-LLCM. To specifically analyze the influence on visual perception capability and fusion feature modeling, we exclude the MER strategy from the fusion query to eliminate the effect of combining original features from both the infrared modality and text modality.
Tri- MMM
Tri-LLCM
I
R1
mAP
mINP
R1
mAP
mINP
VI-ReID Backbone
69.89
66.74
53.34
53.53
59.77
56.40
From Scratch
6..46.15.43
61.31 J5.43
46.9416.40
49.29 14.24
55.78 13.99
12,12.
Tr-SYSU-MMOI
Tri-LLCM
1+T +R
RI
MAP
mINP
RI
mAP
mINP
TVI-LFM
84.17
80.72
70.02
57.13
64,06
60.72
From Scratch
84.03
19.85
68.06.1..99
55.47 L1.66
62.23
58.86..1.88
To optimize the whole framework, we first train a simple VI-ReID backbone, then incrementally fine-tune the VLM textual encoder based on the well-trained frozen backbone to inherit its visual perception capability and integrate text information for infrared modality compensation. If we train the whole framework from scratch, as shown in the Table 5 above, the performance of the VI-ReID backbone suddenly declines by 5.43% and 4.24% in Rank- in the two datasets respectively, indicating the loss of visual perception capability, thereby the performance textually enhanced task (I + T + R) is also affected, with a decline of 0.14% Rank- in Ci-SYSU-MMOO) and 1.66% Rank-1 1 in Tri-LLCM. This demonstrates the importance of incrementally fine-tuning strategy, which avoids the potential performance influence caused by conflicts of modeling visual features and textual enhanced infrared features optimization.
F Broader Impacts
Our TVI-LFM framework offers significant advancements in urban security by enhancing person re- identification in low-light conditions, boosting surveillance effectiveness. It automates text generation from and RGB images, reducing annotation workload and improving text robustness, aiding multi-modal research and smart security system development. However, it's crucial to address
17

---

environmental impact concerns related to large models' energy consumption and the privacy risks associated with re-identification technology. Governments and regulatory bodies must enact stringent regulations to prevent misuse and ensure identification accuracy to avoid societal disruptions.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate clearly main claim that leveraging large foundation models to generate dynamic textual modalities effectively addresses the lack of critical information, such color, in infrared modality. The experimental results across all three expanded VI-ReID datasets show great improvements on the accuracy of our proposed cross-modality retrieval bolstered by heterogeneous text descriptions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
19

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We don't have proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
This paper provides clear and comprehensive description of the proposed TVI-LFM architecture in Sec. 3 and Fig. 2 the method of expanding the existing open- source dataset in Sec. 3.1 as well as the complete implementation details of constructing the whole framework in Appendix
Justification:
B training and testing, along with the detailed steps of the experiments in Sec. 4 This ensures the replicability of our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
20

---

(a) If the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our data and code soon in httts://githhbb cOw/WHU-U/W
TVI
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wwwbllccggu.dds/oodeubbbaaa
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All of experimental settings are shown in Sec.
while the analysis of 4.1
experiments results can be found in Ablation Study in Sec. 4.2 and Comparison with
state-of-the-an methods in Sec. 43
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We don't have error bars, but in order to get stable performance, we evaluate our model for 10 times with random split of the gallery set all experiments on Tri-SYSU- MM01 and Tri-LLCM datasets; for RegDB different training/testing splits, and finally we report our model's average performance on each dataset, the same existing related works did.
evaluate our model on the 10 trials with
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative етror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All the details can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hht.ghg..r.s..aarrr....oo//o///r
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the Broader Impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
23

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The pretrained language model 667 mentioned above we used are safe and come from open source community, and we don't post any new pre-trained language model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes of course, we cite the author and owners for all used assets. And we also respect and follow all the license and terms of use explicitly mentioned. The detail of data and code assets we used are shown in Appendix D
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question:
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
24

---

Justification: All the documentations of original datasets can be viewed at the github urls in Appendix [ Documentations and statistics of the expanded data will also be provided at this githuu url.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work didn't relate to any crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "fVIt6vnaNu", "reviewer": "Reviewer_368Y", "summary": "To address the loss of performance in Visual_infrared re-identification wrt to visual , the authors propose a novel text-enhanced VI-ReID framework driven by Foundation Models (TVI-FM) , which enriches infrared representations with automatically generated textual descriptions. This framework incorporates a pretrained multimodal language vision model (LVM) to extract and fine-tune textual features, minimizing the domain gap between texts and visual images. Additionally, modality joint learning and an ensemble retrieving strategy are introduced to align and leverage features from all modalities, enhancing retrieval effectiveness and robustness.", "questions": "Section 3.2.2 need a careful revision to present the solution is a clearer way. \n\nHow the text data has been aligned with image modality in pre-train? Which text?\n\nIs N_{sum} different from N_i. If so, how? If not why using this notation?\n\nWhat is the difference between MES and MER in the ablation", "limitations": "Although the limitations are addressed in Appendix D, this section is underdeveloped and lacks depth. The discussion fails to provide a clear and comprehensive strategy for improving the quality of the textual information to enhance overall performance. It is unclear at which specific step in the methodology this improvement should be implemented. More detailed explanations and actionable insights are needed to understand how enhancing textual quality can positively impact the system’s effectiveness. Clarifying these points would significantly strengthen the paper’s contribution and provide a more robust framework for future research.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper presents an interesting approach to enriching the limited features of infrared imagery by incorporating textual information, effectively extracted and fused using a strategy based on Language Vision Models (LVMs) and Large Language Models (LLMs). This proposed method stands out as it leverages advanced models to bridge the gap between modalities, enhancing the overall effectiveness of Visible-Infrared Person Re-identification (VI-ReID).\nThe detailed analysis of intra- and inter-class distributions is particularly noteworthy, providing valuable insights into the data characteristics and the impact of the proposed approach. This statistical evaluation underscores the robustness and depth of the methodology, highlighting its potential to significantly improve the performance of VI-ReID systems.\nFurthermore, the results demonstrate good performance on the adopted dataset, outperforming recent state-of-the-art solutions. This achievement validates the effectiveness of the proposed framework. The paper's findings contribute meaningfully to the field and offer a promising direction for future research in enhancing VI-ReID using advanced textual and visual fusion techniques.", "weaknesses": "The presentation of the methodology lacks sufficient clarity, making it challenging to follow the authors' logic. In particular, Section 3.2, which discusses the incremental fine-tuning strategy, is especially confusing. The connection between the textual descriptions and the proposed architecture shown in Figure 2 is not clearly established. Including notations for the features as they are outputted at different steps would greatly improve understanding.\n\nSection 3.2.2 is overly complex and difficult to read. The sentence, “we employ the fine-tuned LVM in Section A to generate textual description,” is misleading because it references a Section A that does not exist within the document. This ambiguity needs to be addressed to avoid reader confusion. Additionally, the explanation of how features are fused is not clearly articulated, adding to the overall lack of coherence in this section.\n\nFurthermore, there are numerous language errors throughout the text that must be corrected. These mistakes detract from the readability and professionalism of the paper, making it harder to take the work seriously. Overall, the paper requires significant revisions to improve its clarity and readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the soundness, competitive performance, and adequate experiments of our method. Our method is a novel exploration of applying foundation models to down-stream data-intensive multimodal tasks. It uses LVM-generated text to enrich infrared representations and employs an end-to-end ReID loss to fine-tune the LVM text encoder, minimizing discrepancies between visible and fusion features thereby achieving competitive performance across three expanded VI-ReID datasets. We are grateful for feedback on the clarity and logical coherence of our presentation. In response, we will refine our manuscript to highlight contributions, correct misleading references and improve readability, especially in Section 3.2 and fusion process. We have adjusted the framework, moving LLM augmentation to \"Text Generation,\" and added elaborations of each module, as detailed in Figure 1 of the submitted rebuttal PDF.\n\n**Q1: Suggestions for clear presentation of Section 3.2, especially fusion process.**\n\n**A:** Incremental Fine-tuning Strategy (IFS) includes Fusion Module and Modality Joint Learning, integrating text into the infrared modality for improved cross-modal retrieval.\n\n**Fusion Module.** As shown in Figure 2 from the rebuttal material, the fusion process is defined as:\n$$\nf^I_{rgb} = f^I_{ir} + f^I_{comp} = f^I_{ir} + (f^I_{rgb} - f^I_{ir}) \\approx f^I_{ir} + (f^T_{rgb} - f^T_{ir}) = f^I_{ir} + f^T_{comp} \\triangleq f^{fusion}_{rgb}\n$$\n\nWhere $f^I_{ir}$ and $f^I_{rgb}$ are infrared and visible features; $f^T_{ir}$ and $f^T_{rgb}$ are text features of visible and infrared image; $f^T_{comp}$ and $f^I_{comp}$ are text and visual complementary information for infrared modality, respectively.\n\nThe visible feature $f^I_{rgb}$ decomposes into the infrared feature $f^I_{ir}$ and its complementary feature $f^I_{comp}= f^I_{rgb} - f^I_{ir}$, and similarly, $f^T_{comp} = f^T_{rgb} - f^T_{ir}$. Using foundation model's basic text-visual alignment capability, visible text features $f^T_{rgb}$ and infrared text features $f^T_{ir}$ are roughly equivalent to visible $f^I_{rgb}$ and infrared $f^I_{ir}$ features respectively. Thus, $f^T_{comp}$ is roughly equal to $f^I_{comp}$. We can create fusion features by adding $f^T_{comp}$ to $f^I_{ir}$, roughly equivalent to visible features.\n\n**Modality Joint Learning.** Freezing visual encoders, we fine-tune a LVM text encoder with ReID loss to jointly align semantics across all modalities. The loss includes a cross-entropy loss $L_{id}$ and a weighted regularized triplet loss $L_{wrt}$, defined as:\n$$\nL_{total} = L_{id}(f^*) + L_{wrt}(f^*) \\quad f^* \\in \\\\{f^ T _ {rgb},f ^ I _ {rgb},f^ I_{ir},f^ {fusion}_{rgb}\\\\}\n$$\nThe fusion feature consists of the combination of frozen infrared features and text complementary features. With IFS, we can further align text and visual complementary features, thus we can further optimize fusion features, reducing fusion-visible discrepancy and improving cross-modal retrieval accuracy.\n\n**Q2: Misleading reference of Section A that does not exist within the document.**\n\n**A2:** The reference is detailed in Appendix A, about the fine-tuning process of two modality-specific text generators. We will add explanation of this process in main text and correspondingly correct the reference to ensure better coherence.\n\n**Q3: How the text data has been aligned with image modality in pre-train? Which text?**\n\n**A3:** \n\nCLIP text encoder possesses text-image alignment capability benefited from pre-training on the large-scale image-text pairs (WebImageText) via contrastive learning: \n$$\n\\mathcal{L} = -\\sum_{i=1}^{N} \\left[ \\log \\frac{\\exp(v_i ^T \\cdot t_i) / \\tau)}{\\sum_{j=1}^{N} \\exp(v _ i ^T \\cdot t_j) / \\tau)} \\right]\n$$\nWhere $ v_i $ and $ t_i $ are the $ i $-th image-text features pair in a batch, $ \\tau $ is the temperature parameter, and $ N $ denotes the number of image-text pairs in the batch. Using this basic capability of CLIP, visible text and infrared text features are roughly aligned with respective visual features, which is further aligned adapting to VI-ReID task in following optimization.\n\n**Q4: Is N_{sum} different from N_i? If so, how? If not, why using this notation?** \n\n**A4:** They are equal because each fusion feature is created based on the corresponding infrared feature. We use these two notations to distinguish the features of different modalities more clearly, so the subscript same as their belonging modality is used for the number of features of all modalities.\n\n**Q5: Difference between MES and MER in ablation.**\n\n**A5:** MES (Modality Ensemble Searching) and MER (Modality Ensemble Retrieving) describe the same module in Section 3.3. We will unify all names of this module to MER.\n\n**Q6: Strategy for improving the quality of generated text.**\n\n**A6:** To improve generated text quality, we can apply several strategies during the generator fine-tuning process (Appendix A):\n\n**Better data for Generator Fine-tuning**: During generator fine-tuning process, we can use augmentations (e.g., flipping, brightness adjustments) for more diverse images, enhancing text generation against varying image quality.\n\n**Stronger Generative LVM**: Use advanced LVMs with larger parameters thereby capturing more useful visual information in images to generate textual descriptions with better quality.\n\n**Progressive Generation Strategy**: Fine-tune generators on images with attribute annotations to focus more on fine-grained attributes rather than sentence modeling, then use LLMs to reorganize them into high-quality descriptive sentences.\n\n**Q7: How enhanced text quality boosts system effectiveness?**\n\n**A7:** During training, high-quality text allows the model to learn better text-vision correspondences, maximizing the utilization of complementary information to create fusion features. During retrieval, high-quality text provides accurate information for infrared compensation, enabling more accurate cross-modal retrieval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urAl87Rx5U", "reviewer": "Reviewer_BZVu", "summary": "Visible-infrared person re-identification often underperforms due to the significant modality differences, primarily caused by the absence of detailed information in the infrared modality. This paper investigates a feasible solution to empower the VI-ReID performance with off-the-shelf foundation models by proposing a text-enhanced VI-ReID framework to compensate for the missing information in the infrared modality.", "questions": "The comparison settings in Table 4 show significant differences. For instance, in Tri-SYSU-MM01, YYDS presents the I+T->R results, while in Tri-RegDB and Tri-LLCM experiments, YYDS shows the I->R results. The rationale behind these settings needs further discussion.", "limitations": "The method utilizes text information to complement the infrared modalities. This approach's reliance on the accuracy of text information generation may pose a limitation, which the authors should briefly discuss.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The figures and tables in this paper are detailed, and the proposed method is logical and well-founded. This method reduces the cost of manual labeling, effectively addresses the issue of missing infrared modality information, and offers some insights for this community. Extensive experiments demonstrate that the proposed method improves retrieval performance on three expanded cross-modality re-identification datasets, paving the way for utilizing LLMs in downstream data-demanding tasks. The new test setting proposed in the paper seems to be a composed image retrieval with some real-world applications.", "weaknesses": "1. The structure of the article needs adjustment. Some content in the appendix, such as Datasets Expansion and Task Settings, should be moved to the main text to enhance readability.\n2. Compared with the previous text enhancement method YYDS, the advantages of this paper should be further explained.\n3. There are some spelling errors in the paper (line 131). It is recommended that the authors conduct a thorough check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive recognition of the detailed tables, clear figures, and the soundness of our framework. We also appreciate your constructive comments and will revise and clarify the suggested points to improve the quality of the paper writing.\n\n**Q1: Some content in the appendix should be moved to the main text to enhance readability.**\n\n**A1:**  As suggested, we will add details of the proposed task setting and an introduction about the generation process into the main text to ensure the coherence and readability.\n\n**Q2: Compared with the previous text enhancement method YYDS, the advantages of this paper should be further explained.**\n\n**A2:** \n\n**YYDS**\n\n- **Manually collected** descriptions for images.\n- **Prior-information** for text-visual complementary information **alignment,** like **pre-defined** color vocabulary **dictionary**; **complicated** architecture with **much additional parameters** for information extraction and fusion, resulting **potential** **information loss**.\n- **Fixed** auxiliary **descriptions** for training, resulting **sensitivity** to text variation.\n\n**Our framework** \n\n- **Automatically** generated text from visible and infrared images.\n- Feature level fusion module without **additional parameter. End-to-end** ReID loss fine-tunes LVM, guiding semantic alignment across all modalities **without prior-information**, significantly mitigating fusion-vision discrepancy and achieve **more acurrate cross-modal retrieval.**\n- Employs **LLM random rephrasing** to create **dynamic** descriptions for training, improve the robustness against textual variation.\n\n\n\n**Q3: There are some spelling errors in the paper (line 131).** \n\n**A3:** We will conduct a thorough review and the correction of spelling errors throughout the document to ensure professionalism and clarity.\n\n**Q4: Varied comparison settings in Table 4, especially the differences in results for YYDS in Tri-SYSU-MM01 vs. Tri-RegDB and Tri-LLCM.**\n\n**A4:** We will standardize the experimental setups for YYDS across all tests to use the \"I+T->R\" configuration. Both YYDS and our method employ joint text and infrared sample retrieval, whereas other methods solely use infrared queries.\n\n**Q5: Briefly discuss the reliance on the accuracy of text information generation, which may pose a limitation.**\n\n**A5:**\n\n- The quality of generated text can indeed affect model performance, particularly when the original images' (for generation) quality or generators' capabilities are suboptimal. \n- However, even on challenging LLCM and lower-resolution RegDB, with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0frNjmPJW", "reviewer": "Reviewer_EhC5", "summary": "This paper incorporates a pretrained multimodal language vision model (LVM) to extract textual features and incrementally fine-tune the text encoder to minimize the domain gap between generated texts and original visual images. Meanwhile, to enhance the infrared modality with text, this paper employs LLM to augment textual descriptions. Furthermore, the authors introduce modality joint learning to align features of all modalities. Additionally, a modality ensemble retrieving strategy is proposed to consider each query modality for leveraging their complementary strengths to improve retrieval effectiveness and robustness.  Adequate experiments verify the validity of the method.", "questions": "See Weakness", "limitations": "This paper only focuses on how to do but not analyze the insight why the proposed method is effective. And the limitation of this method is not introduced.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors utilize existing multimodal models to enhance cross-modal retrieval performance. The method is sound, and the experiments are adequate to effectively demonstrate the effectiveness of the proposed approach. \nPROS:\n1. This paper leverages Language Vision Models (LVMs) to automatically generate textual modality, which enriches the representations of the infrared modality and reduces the cost of human annotation.\n2. The proposed Incremental Fine-tuning Strategy (IFS) and Modality Ensemble Retrieving (MER) can improve the robustness and accuracy of existing VI-ReID systems by complementing the infrared modality with information from generated text.\n3. The experimental results shows that the proposed method achieve a significant gain than SOTA.", "weaknesses": "1. This paper is not easy to follow. The method is not introduced clearly, and some key steps are not detailed. \nThe motivation and advantages of each module in the paper should be clarified further to enhance understanding.\n2. It's important to outline the limitations of existing text-assisted ReID methods like YYDS and highlight the differences and advantages of this paper's method in comparison.\n3. The authors should provide further clarification about the motivation behind the new testing setting. \n4. There appears to be an imbalance in the content distribution among several modules. It is recommended that the authors adjust these accordingly.\n5. The authors should thoroughly discuss the challenges that remain unresolved by current text-assisted VI-ReID methodologies. Furthermore, they need to clearly outline how their approach addresses these issues comprehensively.\n6. The quality of the generated text descriptions may affect the performance of the model, and if the generated texts are inaccurate, which may lead to degradation of retrieval performance.\n7 How are the voting scores  obtained?  It should  be expressed further using formula.\n8 There are some typos in this paper: 1) “... utilizing textwe employ...” in line 131 should be “... utilizing text, we employ...”; 2)  “conbined with” in line 179 should be “combined with”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of the adequate experiments, the soundness and competitive performance of our work. We also appreciate the constructive comments on the motivation, rationale, and content distribution balance, which is valuable for improving our paper writing. \n\n**Q1: Motivation and advantages of each module**.\n\n**A1:** For better presentation, we adjust the framework as shown in Figure 1 (the submitted rebuttal pdf). LLM augmentation is now in \"Text Generation\" and elaboration of the proposed method is included in rebuttal pdf for better understanding. The motivation and advantages of each module are as follows:\n\n- **Text Generation.** Existing methods rely on **fixed manual annotations** for training, incurring **labor and time costs** and is **sensitive to text variation**. Our method uses fine-tuned language vision models to **automatically** generate text and employs LLM random rephrasing to create **dynamic** descriptions for training, enhancing our framework's **robustness** against text variation.\n- **Incremental Fine-tuning Strategy (IFS)** includes **Fusion Module** and **Modality Joint Learning** to integrate text into the infrared modality for improved cross-modal retrieval. Existing methods require **prior-information** like pre-defined color vocabularies for text-visual complmentary information alignment, complicated architecture with **much additional parameters** for information extraction and fusion, causing potential **information loss**. Our method creates **fusion** features at **feature level** by **arithmetic operations** and fine-tunes LVM with **end-to-end ReID loss** to jointly **align semantics** from all modalities **without prior-information**, mitigating the fusion-visible discrepancy and achieving **more accurate** cross-modal retrieval.\n- **Modality Ensemble Retrieving (MER).** Features of different modalities focusing on distinct information, motivated by this, we fully uses of features from all query modalities to form ensemble representations, boosting retrieval accuracy and robustness against challenging retrieval cases.\n\n**Q2: Comparison of existing text-assisted ReID methods like YYDS & and this paper's method.**\n\n**A2:** \n\n**YYDS:**\n- **Manually collected** descriptions for images.\n- **Prior-information** for text-visual complementary information **alignment,** like **pre-defined** color vocabularies; complicated architecture with **much additional parameters** for information extraction and fusion, causing **potential information loss**.\n- **Fixed** auxiliary **descriptions** for training, **sensitivity** to text variation.\n\n**Our framework:** \n- **Automatically** generated text from visible and infrared images.\n- Feature level fusion module without **additional parameter. End-to-end** ReID loss fine-tunes LVM, guiding semantic alignment across all modalities **without prior-information**, significantly mitigating fusion-vision discrepancy and achieving **more acurrate cross-modal retrieval.**\n- Employs **LLM random rephrasing** to create **dynamic** descriptions for training, improving the robustness against textual variation.\n\n**Q3: Motivation behind the new testing setting.**\n\n**A3:** Humans perceive objects as visible images, and eyewitness descriptions are based on these perceptions. These descriptions, rich in information complementary to infrared modalities, serve as auxiliary clues for retrieval. Given the variability in eyewitness descriptions of the same target, our task setting allows any description of visible images of the same identity to be used with infrared features for retrieval, mimicking the varied visual perceptions of human eyewitnesses.\n\n**Q4: Imbalance of content distribution among several modules.**\n\n**A4:** For better content distribution balance, we will re-organize the content about task definition, text-generation, and add more details of each module.\n\n**Q5: Challenges unresolved by current text-assisted VI-ReID methodologies. How the proposed approach addresses these issues comprehensively?**\n\n**A5:** \n- **Sensitive to the text variation.** Our method utilizes LLM based random rephrasing to create dynamic text for training, significantly enhancing the robustness against text variation.\n- **Struggling text-infrared information integration and fusion-visible features alignment.** By fine-tuning LVM with end-to-end ReID loss to align semantics across all modalities, we simultaneouly achieve text-vision alignment for better infrared compensation and fusion-visible alignment for more accurate retrieval.\n\n**Q6: The quality of generated text descriptions may affect the retrieval performance.**\n\n**A6:** \n- The quality of generated text affects model performance when original images' (for text generation) quality or generators' capabilities are suboptimal. \n- However, even on challenging LLCM and lower-resolution RegDB with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.\n\n**Q7: Didn't introduce limitations.**\n\n**A7:** We introduced the limitation about impact of text quality in Appendix D. More detailed discussion and direction for future improvements will be added in the revision. \n\n**Q8: Using formula to express voting scores.**\n\n**A8:** The scores are difined as:\n$$\nscore = f_{ensemble} \\cdot f^I _{rgb}\n= (f^I _{ir} + f^T _{rgb} + f^{fusion} _{rgb})/3 \\cdot f^I _{rgb}\n= ([f^I _{ir}, f^T _{rgb}, f^{fusion} _{rgb}] \\cdot [f^I _{rgb}, f^I _{rgb}, f^I _{rgb}])/3\n$$\nThe score equivalently represents the cosine similarity between the visible feature and a concatenated infrared-text-fusion feature, increasing the feature dimension. This higher-dimensional space increases the distance between identities, enhancing identity discrimination.\n\n**Q9: There are some typos in this paper.**\n\n**A9:** We will revise all mentioned and other found typos as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "49dOEjxbOn", "reviewer": "Reviewer_hNfL", "summary": "This paper proposes a text-enhanced VI-ReID framework driven by Foundation Models (TVI-FM). VI-ReID often lags behind RGB-based ReID due to the inherent differences between modalities, particularly the absence of information in the infrared modality. This paper enriches the representation of the infrared modality by integrating automatically generated textual descriptions. Extensive experiments on three expanded cross-modal re-identification datasets demonstrate significant improvements in retrieval performance.", "questions": "According to the description in Task Settings, the method in this paper utilizes text information from heterogeneous modalities of the same individual during testing, which aligns more closely with real-world conditions. This appears to be a new test setting, and the authors should further elaborate on what constitutes \"real-world conditions\" and discuss its plausibility.", "limitations": "The structure of the paper requires adjustments, particularly in refining details within specific methods. Additionally, the insights should be clarified and made more understandable.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is a good attempt to use textual information from heterogeneous modalities to enhance cross-modal retrieval performance. This paper is methodologically sound, clearly presented, and able to provide the following contributions: \n\na). The proposed text-enhanced VI-ReID framework driven by Foundation Models (TVI-FM) enriches the representation of infrared modality with the automatically generated textual descriptions, reducing the cost of text annotations and enhancing the performance of cross-modality retrieval.\n\nb). This paper develops an Incremental Fine-tuning Strategy (IFS) to employ LLM to augment textual descriptions and incorporate a pre-trained LVM to extract textual features, leveraging modality alignment capabilities of LVMs and feature-level filters generated by LVMs to enhance infrared modality with information fusion and modality joint learning.\n\nc). Extensive experiments demonstrate that the proposed improves retrieval performance on three expanded cross-modality re-identification datasets, paving the way for utilizing LLMs in downstream data-demanding tasks.", "weaknesses": "a). Some key elements in the appendix should be included in the main text, such as obtaining a multimodal model capable of generating text from two visual modalities and the definition of the new task.\n\nb). The introduction is somewhat lengthy and verbose, making the method appear repetitive. It should be simplified to refine the key ideas, avoiding repetition in the method overview.\n\nc). There are some grammatical errors, and the tenses are inconsistent. The authors should further strengthen the correctness of their writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the positive recognition of the soundness, clearly presentation of our framework and also appreciate your detailed comments aimed at improving our paper writing. We believe our revisions will address your suggestions. Thank you for the valuable feedback for guiding these improvements.\n\n**Q1: How testing settings align with \"real-world conditions\"?**\n\n**A1:** Humans perceive objects as visible images, and eyewitness descriptions are based on these perceptions. These descriptions, rich in information complementary to infrared modalities, serve as auxiliary clues for retrieval. Given the variability in eyewitness descriptions of the same target, our task setting allows any description of visible images of the same identity to be used with infrared features for retrieval, mimicking the varied visual perceptions of human eyewitnesses.\n\n\n\n**Q2: Some key elements in the appendix should be included in the main text.**\n\n**A2:** Thanks for the suggestion, we will add details of the proposed task setting and an introduction about the generation process into the main text to ensure the coherence and readability.\n\n**Q3: The introduction is somewhat lengthy and verbose, making the method appear repetitive.**\n\n**A3:** We will streamline the introduction as suggested to reduce redundancy and focus more on the key ideas and innovations of our approach\n\n**Q4: There are some grammatical errors, and the tenses are inconsistent.**\n\n**A4:** We will make revisions to correct all grammatical errors and ensure the consistency of simple present tense throughout the document.\n\n**Q5: The structure of the paper requires adjustments, particularly in refining details within specific methods. Additionally, the insights should be clarified and made more understandable.**\n\n**A5:** We will adjust the structure of the paper and add detailed motivation, rationale, and insights into the confusing parts of the methodology, ensuring that the details and insights of our method are clearly communicated and easily understandable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jeE8aWvGdR", "reviewer": "Reviewer_88CP", "summary": "The authors aim to tackle the challenge of lacking detailed information in the infrared modality by employing foundation models. Their proposed method includes an Incremental Fine-tuning Strategy (IFS) and Modality Ensemble Retrieving (MER). These techniques enhance the representation of the infrared modality through automatically generated textual descriptions, thereby lowering the cost of text annotations and boosting the performance of cross-modality retrieval.", "questions": "The results of YYDS in Tables 3 and 4 do not align with those reported in the original paper. The authors need to provide a further explanation for this discrepancy.", "limitations": "Authors are encouraged to add descriptions of limitations to their papers", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1）The authors explore a viable solution to enhance VI-ReID performance using readily available foundation models. \n\n2）The solution is well-conceived, and the experiments are comprehensive. \n\n3）The paper is well-structured, featuring clear diagrams and lucidly presented ideas.\n\n4）The appendix material provides detailed information about the methodology, and the extensive experiments effectively validate the proposed approach.", "weaknesses": "1) The main content lacks a description of the data generation process. It is recommended to replace the baseline description with details of the data generation process.\n\n2) The task setting should be introduced in the introduction, which can adequately support the rationale for using text enhancement in cross-modality retrieval tasks, improving the quality of the paper.\n\n3) The font size in Figure 2 and the tables needs adjustment.\n\n4) Several writing errors in the paper need correction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback regarding the clear architecture figure, feasibility, and soundness of our method.  We also acknowledge and appreciate the constructive criticisms for improving certain aspects of our paper writing.\n\n**Q1: Explain the discrepancy of the misaligned results of YYDS in Tables 3 and 4 with those reported in the original paper.**\n\n**A1:** We primarily explored the performance of existing models on the proposed automatically expanded tri-modality datasets, at the mainstream image resolution of 144*288, and under the newly proposed task setting.  Consequently, YYDS was tested on our data and setting, yielding dissimilar results from the original paper.  Extensive experiments demonstrate that our method performs better and is more robust under the proposed task.\n\n**Q2: The main content lacks a description of the data generation process.**\n\n**A2:** Thanks for the suggestion, we will add an introduction about the generator fine-tuning process for text generation in the main text to ensure the coherence and readability. \n\n**Q3: The task setting should be introduced in the main text.**\n\n**A3:** Thanks for the suggestion.  We will add a detailed explanation of the task setting to the introduction.\n\n**Q4: The font size in Figure 2 and the tables need adjustment.  Several writing errors in the paper need correction.**\n\n**A4:** We will correct the noted writing errors and adjust the font size in Figure 2 and the tables to enhance readability.\n\n**Q5: Authors are encouraged to add descriptions of limitations to their papers.**\n\n**A5:** The limitations were initially detailed in Appendix D.\n\n- The quality of generated text can indeed affect model performance, particularly when the original images' (for generation) quality or generators' capabilities are suboptimal. \n- However, even on challenging LLCM and lower-resolution RegDB, with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.\n- To provide more valuable insights to the community, we will also add a discussion of potential ways to improve the quality of generated text, like progressive generation strategy and images augmentation for fine-funing VI-ReID specialized description generators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their positive feedback on clear diagrams ($\color{red}{R\\#88cp}$)($\color{red}{R\\#BZVu}$), methodology feasibility ($\color{red}{R\\#hNfL}$)($\color{red}{R\\#88cp}$)($\color{red}{R\\#EHC5}$), competitive performance ($\color{red}{R\\#EHC5}$)($\color{red}{R\\#368Y}$), comprehensive experiments ($\color{red}{R\\#88cp}$)($\color{red}{R\\#EHC5}$), interesting approach and detailed visualization analysis ($\color{red}{R\\#368Y}$). We hope this rebuttal allows $\color{red}{R\\#EHC5}$, and $\color{red}{R\\#368Y}$ to update the scores. The code will be released.

Meta-review: Five experts in the field reviewed this paper. Their recommendations are Weak Reject, Strong Accept, Accept, Strong Accept, and Strong Accept. Overall, the reviewers appreciated the paper because it addresses a new problem (applying foundation models to VI-ReID tasks), proposes an innovative approach, and shows effectiveness in its comprehensive experimental evaluations. Based on their feedback and the author’s satisfactory rebuttal that addresses the initial concerns, I recommend it for acceptance. The reviewers raised some valuable issues and concerns in the Weaknesses that should be addressed in the final camera-ready version of the paper. In particular, the overall structure and clarity of the paper should be significantly improved in the final version.  The motivation, details and advantages of each module in the paper should be clarified to enhance understanding. The task setting should also be introduced from the outset, which can adequately support the rationale for text enhancement in cross-modality retrieval. Finally, it's important to outline the limitations of state-of-the-art text-assisted ReID methods to justify your method.  The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)