Semi-supervised Visible-Infrared Person Re-identification via Modality Unification and Confidence Guidance

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 10:04:15

Abstract: Semi-supervised visible-infrared person re-identification (SSVI-ReID) aims to match pedestrian images of the same identity from different modalities (visible and infrared) while only annotating visible images, which is highly related to multimedia and multi-modal processing.
Existing works primarily focus on assigning accurate pseudo-labels to infrared images but overlook the two key challenges: erroneous pseudo-labels and large modality discrepancy. To alleviate these issues, this paper proposes a novel Modality-Unified and Confidence-Guided (MUCG) semi-supervised learning framework. Specifically, we first propose a Dynamic Intermediate Modality Generation (DIMG) module, which transfers knowledge from labeled visible images to unlabeled infrared images, enhancing the pseudo-label quality and bridging the modality discrepancy. Meanwhile, we propose a Weighted Identification Loss (WIL) that can reduce the model's dependence on erroneous labels by using confidence weighting. Moreover, an effective Modality Consistency Loss (MCL) is proposed to narrow the distribution of visible and infrared features, further narrowing the modality discrepancy and enabling the learning of modality-unified features. Extensive experiments show that the proposed MUCG has significant advantages in improving the performance of the SSVI-ReID task, surpassing the current state-of-the-art methods by a significant margin.
The code will be available.

Main paper: 
 related work 2.1 supervised visible-infrared person reid: 
 Supervised visible-infrared person ReID (SVI-ReID) aims to match infrared images with visible images of pedestrians under nonoverlapping cameras. Recently, some works [21, 42, 57] try to mine modality-invariant information by using complex network structures or generation methods to alleviate modality discrepancy. [40] starts the first attempt by proposing a zero-padding one-stream network toward automatically evolving modality-specific nodes. [11] utilize the modality-sharing layer to develop shared knowledge and improve the modality invariance of deep representation. Additionally, a channel enhancement (CA) method is introduced in [47] to uniformly generate color-independent images by randomly swapping color channels.Although the supervised VI-ReID methods mentioned above have achieved good results, they require a large amount of crossmodality identity annotations, which hinders the rapid deployment of new scenes. Manual annotation requires a high cost, especially for infrared images. In this work, we investigate the semi-supervised visible-infrared person ReID task, which does not require infrared identity annotation and is of great significance for deploying VI-ReID in the real world.The goal of unsupervised domain adaptation (UDA) is to enhance learning of the unlabeled target domain through labeled source domains. It can be roughly divided into three categories, i.e. finetuning [2, 5] , GAN transferring [8, 18, 39] , and joint training [6, 13, 60] . Fine-tuning methods first train the model using labeled source data and then fine-tune the pre-trained model on the target data using pseudo-labels [58] . GAN transfer methods disentangle features into id-related and id-unrelated features [64] or use GAN to transfer the style of images [8] . Joint training methods combine the source data and target data and use the ImageNet network to train from scratch [20] . However, these methods ignore the bridging between two domains, that is, using the similarity between the two domains to learn domain invariant information. The task of this paper is similar to unsupervised domain adaptive ReID [37, 43] . Labeled visible images are the source domain and unlabeled infrared images are the target domain. UDA-VI-ReID aims to transfer learned knowledge from labeled visible images to unlabeled visible infrared images and match images of the same person captured by both visible and infrared cameras. In addition, the unsupervised domain adaptation ReID task is a homogeneous retrieval task, while the semi-supervised VI-ReID task is a heterogeneous retrieval task. The domain difference between visible and infrared images is greater than that in the UDA ReID task, making it a significant challenge.The pseudo-labeling method is a supervised paradigm that learns from both unlabeled and labeled data simultaneously which uses the class with the highest prediction probability as the pseudolabel. According to the assumption of semi-supervised learning [1, 24, 25] , the decision boundary should pass through areas with sparse data to avoid dividing dense sample data points on both sides of the decision boundary. This means that the model needs to make low entropy predictions on unlabeled data, i.e. minimizing entropy. Pseudo-labels can effectively reduce class overlap, leading to clearer class boundaries and more compact learned classes. UPS [32] proposal of high confidence pseudo-labels may not necessarily be correct, while low confidence pseudo-labels are basically incorrect. Based on the above content, when selecting a subset of pseudo-label predictions, we choose high-confidence predictions as positive examples and low-confidence predictions as negative examples. Self-tuning method [38] proposes using a pseudo-label group comparison mechanism to mitigate the impact of noisy labels. FixMatch [34] , ConMatch [22] , and FlexMatch [52] all use thresholds to select high-confidence pseudo-labels for training.F i MLP W v W r F v F r F ap r F ap v DConv DConv GAP GAP F v GAP Residual Block4 Stage-4F i DIMG Module F rIn addition, [37] formulates the label assignment task as an optimal transportation (OT) problem, treating unlabeled samples as suppliers and pseudo-labels as demands. Through the optimal transportation plan, the supplier samples are transported to the demand side at the lowest cost. In this paper, we apply OT to the infrared data label allocation problem. This method can force infrared samples to be assigned to equally sized subsets, avoiding grouping samples together. Furthermore, the quality of pseudo-labels is closely related to the calibration error (i.e. the predictive ability) of the model. This paper proposes an effective WIL to reduce the impact of erroneous pseudo-labels on the model. 
methodology: 
 In this section, we first introduce the model architecture of the proposed Modality-Unified and Confidence-Guided (MUCG) semisupervised VI-ReID. Then, we elaborate on the design of the Dynamic Intermediate Modality Generation (DIMG) Module, Weighted Identification Loss (WIL), and Modality Consistency Loss (MCL) in detail. Finally, we adopt a multi-loss strategy to jointly optimize the proposed semi-supervised VI-ReID method.Figure 2 provides an overview of the proposed MUCG method. The inputs of MUCG are labeled visible images and unlabeled infrared images, which are fed into the DIMG module to generate intermediate modality features. Under the semi-supervision setting, we can only access the labels Y 𝑣 of visible images. For unlabeled infrared images, we initially randomly generate pseudo-labels for them. Then, we introduce the optimal transport assignments [37, 58] to update pseudo-labels,EQUATIONwhere diag(•) denotes the square diagonal matrix with the elements of vector on the main diagonal, P is the softmax output of the infrared image classifier, 𝛾 is a parameter that controls the smoothness of the mapping, 𝛼 and 𝛽 represent class prior uniform distribution vector and sample prior uniform distribution vector respectively. Through them, it is possible to force the assignment of infrared samples to equally sized subsets. The infrared pseudo-labels Y 𝑟 are as follows,EQUATIONwhere argmax(•) is used to find the index of the maximum value in each row of P * , determine the most likely category of each sample, thereby generating an infrared pseudo-label Y 𝑟 . Inspired by the work of PCB [36] in extracting discriminative features, we horizontally divide the feature map F 𝑔 into three parts F 𝑝1 , F 𝑝2 , F 𝑝3 , each of which is fed into the classifier to learn local knowledge. In addition, to reduce the modality discrepancy and eliminate the negative impact of noisy pseudo-labels, we propose a novel Weighted Identification Loss (WIL) and a Modality Consistency Loss (MCL).Unlike unsupervised visible ReID problems, visible and infrared images have significant appearance discrepancies in the SSVI-ReID task. We draw inspiration from works [6, 62] , which show that As shown in Figure 2 , we generate intermediate modality features by mixing visible and infrared features. The DIMG module we proposed can be inserted after the hidden stage in the backbone network . This module takes the output features (F 𝑣 , F 𝑟 ) of visible and infrared images (X 𝑣 , X 𝑟 ) in stage-3 as input and generates two weight factors (W 𝑣 , W 𝑟 ). We can mix visible and infrared features with these two weighting factors to dynamically generate intermediate modality features.In each mini-batch during the training stage, we combine samples into n sample pairs based on labels. For each sample pair (X 𝑣 , X 𝑟 ), both samples have the same label (pseudo-label). After obtaining their feature maps F 𝑣 , F 𝑟 ∈ R ℎ×𝑤×𝑐 , we use the large convolution kernel of depth-wise convolution to extract discriminative features from the visible and infrared modalities. Following [4] , we set the kernel size to 63. Then, we apply average-pooling to both features, resulting in 1 × 1 × 𝑐 dimensional features (F 𝑣 𝑎𝑝 , F 𝑟 𝑎𝑝 ), and their output feature vectors are summed and inputted into 𝑀𝐿𝑃 consisting of two fully connected layers to generate two weighting factors:EQUATIONwhere 𝛿 (•) is a softmax function, W 𝑣 and W 𝑟 are weighting factors for visible and infrared features, respectively. Weighting factors are used to dynamically fuse the features of two modalities. Therefore, the formula for generating intermediate modality features can be written as follows:EQUATIONThen, the intermediate modality features and original features are fed together into the network. Our proposed DIMG module can learn in an effective joint training scheme, rather than undergoing arduous training on GANs or reconstructed images. By utilizing appropriate intermediate modalities to connect the visible and infrared domains, visible knowledge can be better transferred to the infrared domain and improve the discriminative ability of the model in the infrared domain. However, relying solely on the DIMG module is not enough to fully address all the challenges in the SSVI-ReID task. Especially in small datasets, the problem of noisy labels during training has become a challenge that we must face. To address this challenge, we further propose weighted identification loss.Unlike other semi-supervised learning methods [22, 34, 52 ] that only select high-confidence samples during the sample selection stage, we use all samples for training due to the small size of the VI-ReID datasets. However, the inevitable inclusion of noisy labels in pseudo-labeled samples can significantly reduce model performance. To alleviate this issue, we propose a Weighted Identification Loss (WIL) that utilizes confidence weighting to mitigate the impact of incorrect labels. Drawing inspiration from work [44] , we utilize the memory effect of deep neural networks (DNN) to calculate the correct labeling confidence for each sample by simulating the loss distribution. The loss distribution of each sample in all training data is fitted by a two-component Gaussian mixture model, as shown below:EQUATIONwhere 𝜂 𝑘 and 𝜑 (𝐿 𝑖𝑑 |𝑘) are the mixture coefficient and probability density of the 𝑘-th component, respectively. 𝐿 𝑖𝑑 is the identification (cross-entropy) loss. Based on the memory effect of DNN, we can calculate the correct annotation confidence 𝑤 𝑘 for each sample 𝑘:EQUATIONwhere 𝑚 is the posterior probability over the small mean value component. Therefore, the proposed WIL can be expressed as follows:EQUATIONwhere 𝑥 𝑘 is the input image feature, 𝑦 𝑘 is the corresponding label, and 𝑝 (𝑦 𝑘 |𝑥 𝑘 ) is the prediction probability that 𝑥 𝑘 is recognized as class 𝑦 𝑘 . However, as pointed out in [32] , low-confidence pseudolabels are largely incorrect, so we set a certain threshold. When the confidence is below this threshold, the sample is treated as a negative sample for learning. So, the proposed WIL is as follows:EQUATIONwhereEQUATION𝜏 is a threshold for positive and negative labels, and we set it to 0.1. 𝑤 𝑝 𝑘 is the positive learning weight, and 𝑤 𝑛 𝑘 is the negative learning weight. For visible images, since their labels are known and correct, we set 𝑤 𝑘 to 1. For infrared images, the proposed WIL can enable all pseudo-label samples to play a role in the training process while more accurately evaluating the confidence of pseudolabels and weighting the loss function accordingly, reducing the negative impact of noisy labels on model training.Despite WIL's ability to optimize the model's handling of noisy labeled samples, the inherent differences between visible and infrared modalities continue to hinder the model's feature extraction and matching capabilities. Consequently, in this section, we delve deeper into strategies to mitigate the discrepancies between these modalities, aiming to enhance the model's performance in SSVI-ReID tasks. To alleviate the impact of cross-modality on model performance, we can reduce the distance between each visibleinfrared image pair with the same identity. Specifically, 𝑁 identities are randomly sampled from the dataset, and 𝑃 visible images and 𝑃 infrared images are sampled for each identity to form a mini-batch with 2 × 𝑁 × 𝑃 images. Then, to enhance the similarity between visible and infrared features, we define the following loss function: where 𝐹 𝑣 𝑛,𝑝 and 𝐹 𝑟 𝑛,𝑝 represent the normalized feature of the 𝑝-th visible image and infrared image of 𝑛-th identity respectively in each mini-batch.EQUATIONHowever, due to semi-supervised settings, there are incorrect infrared pseudo-labels. Paired narrowing of the distance between visible and infrared images will further reduce the distance between indistinguishable erroneous infrared images and visible images, affecting model performance. What's more, although this paired loss will reduce the modality gap of cross-modality images, it may lead the network to focus more on some details, such as posture and accessories, rather than identity features. Based on this, we calculate the centers of the visible and infrared features of the same identity,EQUATIONwhere 𝐶 𝑣 𝑛 and 𝐶 𝑟 𝑛 represent the centers of the visible and infrared features of the 𝑛-th identity respectively. By narrowing the distance between their centers, the modality gap between visible and infrared modalities can be narrowed, while avoiding the negative impact of a small amount of incorrectly labeled features. Therefore, the proposed modality consistency loss can be written as follows:EQUATIONwhere 𝜙 (•) is a linear kernel, variables are mapped to vectors in Hilbert Space through kernel functions. We project features onto Hilbert Space to measure the distance between them. As shown in Figure 3 , it is obvious that the optimization of MCL would make two modality features similar by bridging the modality gap by reducing the distance between the visible-infrared feature centers of the same identity. The proposed modality consistency loss not only reduces the modality discrepancy between visible and infrared images, but also narrows the feature gap within the same modality, encouraging the compact distribution of features with the same identity within each modality.The original visible and infrared images are fed together into the two-stream ResNet50 [14] backbone network, along with the generated intermediate features, to help optimize the network. In the proposed MUCG, in addition to the proposed 𝐿 𝑊 𝐼 𝐿 and 𝐿 𝑀𝐶𝐿 , we also combined the triplet loss 𝐿 𝑇 𝑅𝐼 [16] and the adversarial loss 𝐿 𝐷 [37] to jointly optimize the network together.EQUATION𝐿 𝑇 𝑅𝐼 is used in VI-ReID tasks, as it helps to minimize intra-class similarity and maximize inter-class similarity in metric learning. 𝐿 𝐷 is an adversarial loss in domain adaptation, assisting the model in learning modality-invariant features. The total loss of the proposed MUCG is defined as:EQUATIONwhere 𝜆 𝑊 𝐼 𝐿 and 𝜆 𝑀𝐶𝐿 are two trade-off hyper-parameters. Overall, the proposed method provides a comprehensive solution for SSVI-ReID, utilizing multiple loss functions and modalities to enhance the performance of the model. 
experiments 4.1 datasets: 
 The proposed method is evaluated on three challenging VI-ReID datasets, i.e., SYSU-MM01 [40] , RegDB [30] The proposed method is implemented with PyTorch. The model is trained for 80 epochs in total. We use ResNet-50 [14] pre-trained on the ImageNet [7] as the backbone to extract image features. Following [55, 56] , for the SYSU-MM01 dataset, the input images are resized to 384 × 192. In each mini-batch, we randomly select 4 visible images and 4 infrared images from 6 identities for training.For the RegDB and LLCM datasets, the input images are resized to 288 × 144. In each mini-batch, we randomly select 4 visible images and 4 infrared images from 8 identities for training. In the training stage, the input images are randomly flipped and erased with 50% probability [61] , while visible images are extra randomly grayscale with 50% probability. The model is optimized by the Adam optimizer with an initial learning rate of 3.5 × 10 -3 . The learning rate is incorporated with a warm-up strategy [28] and decayed 10 times at epoch 20 and epoch 50 [37] . The hyper-parameter 𝜆 𝑊 𝐼 𝐿 is set to 0.1. The hyper-parameter 𝜆 𝑀𝐶𝐿 is set to 5 on the SYSU-MM01 and LLCM datasets, and to 100 on the RegDB dataset.We compare our method with three related VI-ReID settings to demonstrate its effectiveness, i.e., fully-supervised VI-ReID (SVI-ReID), unsupervised domain adaptation ReID (UDA-ReID), and semi-supervised VI-ReID (SSVI-ReID). Following [37] , for UDA-ReID methods [12, 13, 29, 59] , we use ground-truth labeled visible data as the source domain and unlabeled infrared data as the target domain. Following [43] , for visible-infrared UDA-ReID methods [37, 43] , we use other labeled visible data as the source domain and unlabeled VI-ReID data as the target domain. The experimental results on the SYSU-MM01 and RegDB datasets are reported in Table 1 and the results on the LLCM dataset are reported in Table 2 .Comparison with Fully-supervised Methods: The proposed MUCG only with ground-truth visible data outperforms several fully supervised VI-ReID methods on the SYSU-MM01 and RegDB datasets and achieves comparative results on the LLCM dataset.The results indicate that the proposed MUCG can effectively utilize unlabeled infrared image information to improve model performance. However, there remains a certain gap between the proposed MUCG and the state-of-the-art fully supervised results. Comparison with Unsupervised Domain Adaptation Methods: As we can see, the state-of-the-art UDA-ReID methods [12, 13] cannot achieve good results under semi-supervised VI-ReID settings due to the huge modality discrepancy. Although some UDA-ReID methods use stronger monitoring signals than ours, the accuracy is far lower than our method. On the other hand, UDA-VI-ReID [37] and [43] achieve better results than the traditional UDA-ReID [12] and [13] . This is because the traditional UDA-ReID method heavily relies on the labeled source domain, making the model less distinguishable for infrared data. Our MUCG can help the model alleviate modality gaps and achieve excellent performance. Specifically, compared to TAA [43] , mAP gains of 23.5% and 20.7% are achieved on the SYSU-MM01 and RegDB datasets, respectively.Comparison with Semi-supervised Methods: In the same experimental setting (SSVI-ReID), our method outperforms existing methods [48, 49] . Both OTLA [37] and DIPS [33] focus on handling infrared pseudo-labels, while neglecting the handling of the modality gap, and their handling of pseudo-labels is not comprehensive enough. OTLA focuses on generating pseudo-labels while neglecting the calibration of noisy labels. DIPS focuses on the calibration of noisy pseudo-labels. Compared with OTLA, our MUCG achieved 20.6% and 22.0% gains on the SYSU-MM01 dataset, 37.0% and 34.9% gains on the RegDB dataset, and 7.7% and 7.0% gains on the LLCM dataset, respectively in Rank-1 and mAP. MAUM-50 and MAUM-100 use 50 and 100 IR identities respectively to train the VI-ReID model. Our MUCG does not require IR data annotation and performs better than MAUM.The Influence of Different Components: To evaluate the contribution of each component to MUCG, we conduct some ablation studies on the SYSU-MM01 dataset. The overall settings remain the same, while only the module under demonstration is added or removed from MUCG. As shown in Table 3 , by incorporating 4 , it can be observed that when optimizing by 𝐿 𝑊 𝐼 𝐿 , the network achieves the best performance, surpassing the 𝐿 𝑊 𝐼 𝐿-by 1.3% and 1.9% on the SYSU-MM01 dataset, respectively in Rank-1 and mAP. To demonstrate that using feature centers of the same identity to measure the distribution of visible and infrared modalities is more effective than using one-to-one corresponding visible-infrared features to measure the distribution, we conduct experiments to compare the results of using 𝐿 𝑀𝐶𝐿- and 𝐿 𝑀𝐶𝐿 . As shown in Table 4 , it can be observed that the network achieves the best performance when optimizing by 𝐿 𝑀𝐶𝐿 , surpassing the 𝐿 𝑀𝐶𝐿-by 1.8% and 1.9% on the SYSU-MM01 dataset, respectively in Rank-1 and mAP.The 5 , DIMG after stage-3 can achieve the best performance, which indicates that after stage-3, the proposed DIMG can better transfer visible knowledge to the infrared domain.The Influence of the Hyper-parameters 𝜆 𝑊 𝐼 𝐿 and 𝜆 𝑀𝐶𝐿 : To evaluate the influence of the two hyper-parameters, we give quantitative comparisons and report the results in Figure 4 . As we can see, the best performance is achieved when 𝜆 𝑊 𝐼 𝐿 is set to 0.1 and 𝜆 𝑀𝐶𝐿 is set to 5, respectively. Pseudo-label Analysis: We conduct an analysis experiment to evaluate the accuracy of pseudo-labels. As shown in Figure 6 , as the training continues, the pseudo-label accuracy of the semisupervised setting is iteratively improved. It can achieve an accuracy of 83.6% on the SYSU-MM01 dataset, surpassing OTLA's [37] 54.8%. Compared with OTLA, we penalize noisy labels while improving the model's discrimination ability for infrared images. As pseudo-labels are generated through model prediction, enhancing the performance of the model will significantly boost the accuracy of these labels.To investigate the reasons for the effectiveness of MUCG, we visualize inter-class and intra-class distances on the SYSU-MM01 dataset, as shown in 5 shows that the intra-class distances of MUCG are significantly smaller compared to the distances of baseline features. Therefore, MUCG can effectively reduce the distances between visible and infrared images. To further validate the effectiveness of the proposed MUCG, we plot the t-SNE distribution of the MUCG feature representations in the 2D feature space for visualization. As shown in Figure 7 (a) and 7 (b), the proposed MUCG method can significantly shorten the distance between images corresponding to the same identity in visible and infrared modalities, and effectively reduce modality discrepancy. 
conclusion: 
 In this paper, we investigate the semi-supervised visible-infrared re-identification (SSVI-ReID) task, which can reduce the cost of cross-modality annotation. We propose a novel modality-unified and confidence-guided semi-supervised VI-Reid learning framework. We have also proposed three modules: DIMG, WIL, and MCL. DIMG can dynamically generate appropriate intermediate modality features, which helps improve the model's discrimination ability in the infrared domain and reduce modality discrepancies between visible and infrared modalities. In addition, we use the WIL to reduce the negative impact of incorrect labels on the model, and we use the MCL to narrow the distance between visible and infrared modality features. Extensive experiments have shown that MUCG outperforms the state-of-the-art semi-supervised methods and some fully supervised methods.  
: 
 in practice. Therefore, [40] propose a challenging cross-modality visible-infrared ReID (VI-ReID) task.The cross-modality VI-ReID task [10, 23, 50] solves the problem of person ReID under poor lighting conditions, aiming to match nighttime infrared person images captured by infrared cameras with visible person images. At present, significant progress has been made in VI-ReID. Some widely used VI-ReID techniques [35, 40] strive to identify distinct embedding spaces that minimize the gap between different modalities at the embedding level. Nevertheless, the significant modality gap poses a challenge for these methods in locating appropriate embedding spaces. Alternatively, there are image-level approaches [9, 56] that aim to transform images from one modality to another, effectively bridging the modality gap between visible and infrared images. Despite their success in reducing the modality gaps, the generated cross-modality images are usually accompanied by some noises. An important factor for the above methods to achieve good results is their well-annotated cross-modality training sets. However, annotating cross-modality ReID data is extremely time-consuming and requires extremely high costs. Additionally, the lack of color information in infrared images makes it more difficult to annotate cross-modality images manually. These problems motivate us to train a cross-modality ReID model using labeled visible data and unlabeled infrared data.Therefore, the investigation of the semi-supervised VI-ReID (SSVI-ReID) task holds significant importance. It aims to learn modality-invariant knowledge from labeled visible data and unlabeled infrared data, thereby achieving cross-modality pedestrian image retrieval. However, existing single-modality UDA-ReID methods (using labeled visible images as the source domain and unlabeled infrared images as the target domain) suffer from crossmodality discrepancy, making it difficult to directly learn modality invariant features. Besides, the current semi-supervised VI-ReID methods [33, 37] primarily focus on how to correctly assign pseudolabels. OTLA [37] focuses on the assignment of infrared pseudolabels. DIPS [33] generates pseudo labels dependently on multimodel collaboration, which might lead to reduced efficiency. They often neglect the negative impact of noisy pseudo-labels and modality discrepancy. Therefore, as shown in Figure 1 , how to eliminate the negative impact of noisy pseudo-labels and transfer the learned knowledge from visible modality to infrared modality under semisupervised settings is the key to the SSVI-ReID task.In this paper, we propose a new Modality-Unified and Confidence-Guided (MUCG) semi-supervised method for VI-ReID without the labels of infrared images. To address the issue of noisy labels and the modality discrepancy between the labeled visible and unlabeled infrared images, we propose the following three modules. Firstly, we propose a Dynamic Intermediate Modality Generation (DIMG) module that generates intermediate modality features by mixing the features of visible and infrared modalities. Using intermediate modality features to improve the discriminative ability of the model for unlabeled infrared images. Secondly, to reduce the negative impact of noisy pseudo-labels, we propose a Weighted Identification Loss (WIL) to calculate the confidence of pseudo-labels. By assigning different weights to different pseudo-labels, the WIL can ensure that the model pays more attention to reliable labels during the training process, while reducing dependence on unreliable labels. Finally, to address the issue of cross-modalities discrepancy, we propose an effective Modality Consistency Loss (MCL) to minimize the distances between visible and infrared modalities. The three modules, DIMG, WIL, and MCL focus on enhancing the model's adaptability to modality differences, reducing the impact of noisy labels, and enhancing feature alignment, respectively, thus solving the issues of noisy labels and modality discrepancies. The proposed method significantly improves the overall performance of the model in the SSVI-ReID task. Specifically, the MUCG method achieves a Rank-1 accuracy of 68.8% on the SYSU-MM01 dataset, 86.9% on the RegDB dataset, and 51.9% on the LLCM dataset, surpassing the current state-of-the-art semi-supervised methods.The main contributions can be summarized as follows:(1) We propose a novel modality-unified and confidence-guided semi-supervised VI-ReID framework that exclusively relies on the annotation of visible images, offering a cost-effective solution.(2) We design a dynamic intermediate modality generation module, which can effectively enhance the model's discriminative ability of unlabeled infrared images.(3) We propose a weighted identification loss and a modality consistency loss, alleviating the negative impact of noisy pseudolabels and narrowing the modality gap between visible and infrared.(4) The proposed method outperforms other state-of-the-art methods for the semi-supervised VI-ReID task on three challenging datasets, as demonstrated by extensive experiments. 
 
 Appendix:  


Decision: This paper proposes a Modality Unification and Confidence Guidance (MUCG) method for semi-supervised visible-infrared person re-identification (SSVI-ReID), addressing pseudo-label errors and modality differences. The Dynamic Intermediate Modality Generation (DIMG) module creates intermediate modality features by blending features from visible and infrared images to bridge the gap between modalities and enhance the discriminative capability for unlabeled infrared images. The Weighted Identification Loss (WIL) leverages the memory effect of deep neural networks to calculate the correct labeling confidence for each sample, reducing the negative impact of incorrect labels. The Modality Consistency Loss (MCL) further minimizes the feature distribution gap between visible and infrared modalities by calculating the distance between feature centers of the same identity, optimizing feature similarity across modalities. Extensive experimental results show that the proposed method has achieved significant performance improvement on semi-supervised VI-ReID tasks.

Initially, this paper received three positive scores and one negative review. After the rebuttal, all the reviewers have updated the score to Weak Accept. Therefore, I would like to accept this paper. The authors may carefully prepare the final camera ready version according to the reviewer comments. Some writing issues could be improved and missing references should be added.

Reviews: review: paper_summary: paper_summary: The article presents a semi-supervised visible-infrared person re-identification (SSVI-ReID) approach called the Modality-Unified and Confidence-Guided (MUCG) framework. This framework is designed to address the challenges of erroneous pseudo-labels and significant modality discrepancies by introducing innovative components. The Dynamic Intermediate Modality Generation (DIMG) module creates intermediate modality features by blending features from visible and infrared images to bridge the gap between modalities and enhance the discriminative capability for unlabeled infrared images. The Weighted Identification Loss (WIL) leverages the memory effect of deep neural networks to calculate the correct labeling confidence for each sample, reducing the negative impact of incorrect labels. The Modality Consistency Loss (MCL) further minimizes the feature distribution gap between visible and infrared modalities by calculating the distance between feature centers of the same identity, optimizing feature similarity across modalities. The experiments conducted on multiple datasets demonstrate that MUCG surpasses current state-of-the-art methods in SSVI-ReID tasks, offering an effective semi-supervised learning strategy, especially beneficial for scenarios with limited annotation resources. This work advances research in cross-modality person re-identification and provides new insights and tools for future studies in related fields.
strength_weakness: strength_weakness: 1. This paper innovatively combines modality unification and confidence-guided learning to address the unique challenges of semi-supervised visible-infrared person re-identification. The introduction of the Dynamic Intermediate Modality Generation (DIMG) module is particularly novel, as it effectively generates intermediate features that bridge the gap between visible and infrared modalities, enhancing the model's capability to handle unlabelled infrared images.
2. The theoretical approach is solid and well-founded, integrating the DIMG with Weighted Identification Loss (WIL) and Modality Consistency Loss (MCL) to systematically reduce the impact of erroneous pseudo-labels and modality discrepancies. This combination ensures the model can learn more robust and unified features, which is crucial for cross-modality tasks.
3. The paper demonstrates correctness and thoroughness through comprehensive experiments. The evaluation is extensive, with the MUCG framework being tested on multiple challenging datasets (SYSU-MM01, RegDB, LLCM), and consistently outperforming existing state-of-the-art methods. The use of standard evaluation metrics (Cumulative Matching Characteristics and mean Average Precision) further validates the robustness and applicability of the proposed approach.
limitations: limitations: Although the Weighted Identification Loss (WIL) aims to mitigate the impact of noisy labels, the reliance on pseudo-labels still introduces a vulnerability, especially if the initial pseudo-labels are of low quality. This can lead to a propagation of errors, which is a common challenge in semi-supervised learning frameworks.
2. the paper could improve in terms of clarity and depth of certain methodological explanations. For example, the specifics of how the Dynamic Intermediate Modality Generation (DIMG) module operates could be elaborated more comprehensively to provide better insights into its inner workings and effectiveness. This would help in understanding the potential limitations or constraints of this module in different contexts.
3. The paper does not delve deeply into the computational complexity or efficiency of the proposed method. Given that practical applications often require real-time processing, an analysis of the computational overhead introduced by the DIMG, WIL, and MCL modules would have been valuable. Without this, it is difficult to gauge the method's feasibility for deployment in resource-constrained environments.
paper_summary: paper_summary: This paper introduces a Modality-Unified and Confidence-Guided (MUCG) semi-supervised VI-ReID framework. The authors propose three modules - Dynamic Intermediate Modality Generation (DIMG), Weighted Identification Loss (WIL), and Modality Consistency Loss (MCL), addressing the issues of erroneous pseudo-labels and large modality discrepancy between visible and infrared images. Compared with existing methods, the performance is significantly improved.
strength_weakness: strength_weakness: 1. This paper introduces a DIMG module that effectively bridges the gap between visible and infrared modalities, and enhances the model's discriminative ability for infrared modality, thereby elevating the pseudo-label quality and narrowing the modality gap.
2. This paper also introduces WIL and MCL. WIL can reduce the model's dependency on erroneous labels through confidence weighting, while MCL further minimizes the distribution gap between visible and infrared features, enabling the model to learn modality-unified features.
3. Experimental results demonstrate that the proposed MUCG method achieves significant performance improvements on the semi-supervised VI-ReID task, surpassing current state-of-the-art methods. This indicates that the proposed method is highly effective and practical in addressing the semi-supervised VI-ReID problem.
limitations: limitations: 1. The Fig.1 lacks explanations, which may confuse readers when understanding the content of the figure. Suggest the authors add annotations in Figure 1 and clearly explain the meaning and direction of the arrows so that readers can have a more intuitive understanding of the process and structure of the model.
2. What is the calculation method for confidence weight in WIL? How to determine appropriate weight values to ensure that the model can balance the contribution of samples with different confidence levels during the training process?
3. Although MCL helps to narrow the modality gap, it remains unclear whether it can maintain sufficient robustness in the presence of a large number of erroneous labels. Is it necessary to combine other techniques (such as data cleaning, label correction, etc.) to further enhance the model's performance?
4. In the experimental section, apart from demonstrating performance improvements, it would be beneficial to include an analysis of the characteristics of different datasets and discuss the key factors that influence performance, thereby enhancing the persuasiveness of the experimental results.
paper_summary: paper_summary: This paper proposes a Modality Unification and Confidence Guidance (MUCG) semi-supervised VI-ReID method, addressing pseudo-label errors and modality differences in existing work. The proposed Dynamic Intermediate Modality Generation (DIMG) module effectively transfers annotated visible image knowledge to unlabeled infrared images. Meanwhile, by introducing Weighted Identification Loss (WIL) and Modality Consistency Loss (MCL), the robustness and performance of the model are further improved. The experimental part also fully verified the effectiveness of the proposed method, which has significant advantages compared to existing methods.
strength_weakness: strength_weakness: 1. From the content of the paper, it appears that the logic is clear and the structure is rigorous. The DIMG module can effectively solve the problems of pseudo label errors and modal differences. The proposal of WIL and MCL has further improved the performance of the model.
2. In the experimental section, the authors conduct sufficient validation and compare it with existing methods. The experimental results show that the proposed method has achieved significant performance improvement on semi-supervised VI-ReID tasks, fully demonstrating its effectiveness and superiority.
limitations: limitations: 1. When describing the DIMG module, its working principle and implementation details can be further elaborated. For example, explain how it enhances the quality of pseudo labels through knowledge transfer, and how it bridges modal differences.
2. Please explain why MCL utilizes feature centers to reduce intra-class variance, instead of operating directly on individual sample features. What advantages or considerations underlie this design choice?
3. In the title of Figure 7, please indicate the specific dataset used for the experiment depicted in the figure for clarity.
4. There is a small writing error: in Table. 2, the "VIR" should be written as "VIS".
paper_summary: paper_summary: This paper focuses on semi-supervised VI-ReID tasks. A DIMG module is proposed to bridge the modality gap at the feature level by dynamically generating intermediate features. A WIL loss is proposed to assign different weights to instances based on the confidence of their pseudo-labels. Additionally, an MCL loss is introduced to narrow the modality gap between cross-modality centroids.
strength_weakness: strength_weakness: (1) The proposed method achieves remarkable performance, which surpasses the SOTAs by a considerable margin;\
(2) The extensive experiments demonstrate the effectiveness of each component of the proposed method. The experiments involving the detailed design of each module are comprehensive. The illustrations and visualizations are appropriate.
limitations: limitations: (1) The intermediate modality generation strategy is commonly employed in existing methods [1,2]. It is important to clearly elucidate the distinctions and advantages of DIMG compared to these methods.\
(2) The confidence-guided WIL loss appears to be inspired by [3]. The main novelty lies in the improvement from WIL to WIL. Nevertheless, the extent of this improvement seems limited.\
(3) This paper integrates PCB [4] into their baseline. It might be unfair to compare with other methods that do not utilize PCB. Therefore, the authors are suggested to provide the performance gain achieved by PCB.

[1] Yu H, Cheng X, Peng W, et al. Modality unifying network for visible-infrared person re-identification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11185-11195.\
[2] Zhang Y, Yan Y, Lu Y, et al. Towards a unified middle modality learning for visible-infrared person re-identification[C]//Proceedings of the 29th ACM international conference on multimedia. 2021: 788-796.\
[3] Shi J, Zhang Y, Yin X, et al. Dual pseudo-labels interactive self-training for semi-supervised visible-infrared person re-identification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11218-11228.\
[4] Sun Y, Zheng L, Yang Y, et al. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline)[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 480-496.\