Learning Commonality, Divergence and Variety for Unsupervised Visible-Infrared Person Re-identification

Authors: jiangming  shi
xiangbo  yin
yachao  zhang
zhizhong  zhang
yuan  xie
yanyun  qu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 04:03:16

Abstract: Unsupervised visible-infrared person re-identification (USVI-ReID) aims to match specified persons in infrared images to visible images without annotations, and vice versa. USVI-ReID is a challenging yet underexplored task. Most existing methods address the USVI-ReID through cluster-based contrastive learning, which simply employs the cluster center to represent an individual. However, the cluster center primarily focuses on commonality, overlooking divergence and variety. To address the problem, we propose a Progressive Contrastive Learning with Hard and Dynamic Prototypes for USVI-ReID. In brief, we generate the hard prototype by selecting the sample with the maximum distance from the cluster center. We reveal that the inclusion of the hard prototype in contrastive loss helps to emphasize divergence. Additionally, instead of rigidly aligning query images to a specific prototype, we generate the dynamic prototype by randomly picking samples within a cluster. The dynamic prototype is used to encourage variety. Finally, we introduce a progressive learning strategy to gradually shift the model's attention towards divergence and variety, avoiding cluster deterioration. Extensive experiments conducted on the publicly available SYSU-MM01 and RegDB datasets validate the effectiveness of the proposed method.

Main paper: 
 introduction: 
 Visible-infrared person re-identification (VI-ReID) aims at matching the same person captured in one modality with their counterparts in another modality [1] [2] [3] . It has recently gained attention in computer vision applications like video surveillance [4] and image retrieval [5] [6] [7] . With the development of deep learning [8] [9] [10] [11] , VI-ReID has achieved remarkable advancements [12] [13] [14] . However, the development of existing VI-ReID methods is still limited due to the requirement for expensive-annotated training data [15, 16] . To mitigate the problem of annotating large-scale cross-modality data, some semi-supervised VI-ReID methods [17] [18] [19] are proposed to learn modalityinvariant and identity-related discriminative representations by utilizing both labeled and unlabeled data. For this purpose, OTLA [17] proposed an optimal transport label assignment mechanism to assign pseudo-labels for unlabeled infrared images while ignoring how to calibrate noise pseudolabels. DPIS [18] integrates two pseudo-labels generated by distinct models into a hybrid pseudo-label for unlabeled infrared data, but it makes the training process more complex. Although these methods have gained promising performances, they still rely on a certain number of manual-labeled data.Several USVI-ReID methods [20] [21] [22] [23] have proposed to tackle the issues of expensive visible-infrared annotation through contrastive learning. These methods create two modality-specific memories, one for visible features and the other for infrared features. During training, these methods consider the memory center as a prototype and minimize the contrastive loss across the features of query images and prototype. Then, these methods aggregate the corresponding prototypes based on similarity. However, the centroid prototype only stores the commonality of each person, neglecting the divergence [24] [25] [26] , which causes the pseudo-labels generated by the cluster to be unreliable. Just like a normal distribution, to better reflect the data distribution of a dataset, we need not only the mean but also the variance.In this paper, we argue that an important aspect of contrastive learning for USVI-ReID, i.e. the design of the prototype, has so far been neglected, and propose progressive contrastive learning with hard and dynamic prototype (PCLHD) method for the USVI-ReID. Firstly, we design a Hard Prototype Contrastive Learning (HPCL) to mine divergent yet meaningful information. In contrast to traditional contrastive learning methods, we choose the hard samples to serve as the hard prototype. In other words, the hard prototype is the one that is farthest from the memory center. The hard prototype encompasses distinctive information. Furthermore, we introduce the concept of Dynamic Prototype Contrastive Learning (DPCL), we randomly select samples from each cluster to serve as the dynamic prototype. DPCL effectively accounts for the intrinsic variety within clusters, enhancing the model's adaptability to varying data distributions. Early clustering results are unreliable, and utilizing hard and dynamic prototype at this stage may lead to cluster degradation. Therefore, we introduce progressive contrastive learning to gradually focus on divergence and variety.The main contributions are summarized as follows:• We propose a progressive contrastive learning with hard and dynamic prototype method for the USVI-ReID. We reconsider the design of prototypes in contrastive learning to ensure that the model stably captures commonality, divergence, and variety. • We propose Hard Prototype Contrastive Learning for mining divergent yet significant information, and Dynamic Prototype Contrastive Learning for preserving the intrinsic variety in sample features. • Experiments on SYSU-MM01 and RegDB datasets demonstrate the superiority of our method compared to existing USVI-ReID methods, and PCLHD generates higher-quality pseudo-labels than other methods.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
supervised visible-infrared person reid: 
 Visible-infrared person re-identification (VI-ReID) has drawn much attention in recent years [27] [28] [29] [30] [31] [32] .Many VI-ReID methods focused on mitigating huge semantic gaps across modalities have made advanced progress, which can be classified into two primary classes based on their different aligning ways: image-level alignment and feature-level alignment. The image-level alignment methods focus on reducing cross-modality gaps by modality translation. Some GAN-based methods [33, 34] are proposed to perform style transformation for aligning cross-modality images. However, the generated images unavoidably contain noise. Therefore, X-modality [35] and its promotions [36, 37] align cross-modality images by introducing a middle modality. Mainstream feature-level alignment methods [38] [39] [40] focus on minimizing cross-modality gaps by finding a modality-shared feature space. However, the advanced performances of the above methods build on large-scale humanlabeled cross-modality data, which are quite time-consuming and expensive, thus hindering the fast application of these methods in real-scenes.The existing unsupervised single-modality person ReID methods can be roughly divided into two classes: Unsupervised domain adaption (UDA) methods, which try to leverage the knowledge transferred from labeled source domain to improve performance [41] [42] [43] [44] , and fully unsupervised Compared with the UDA methods, the USL methods are more challenging. Recently, clustercontrast learning [46] has achieved impressive performance by performing contrastive learning at the cluster level. However, cluster-contrast with a uni-proxy can be biased and confusing, which fails to accurately describe the information of a cluster. To this end, the methods [47, 48] proposed maintaining multi-proxies for a cluster to adaptively capture different information within the cluster. The above methods are mainly proposed to solve the single-modality ReID task, but they are limited to solving the USL-VI-ReID task due to large cross-modality gaps.Unsupervised visible-infrared person ReID (USVI-ReID) has attracted much attention due to the advantage of not relying on any data annotation. Some UDA methods [49, 17] use a well-annotated labeled source domain for pre-training to solve the USVI-ReID task. Some fully unsupervised methods [23, 22] adopt contrastive learning to boost performance, which mainly follow a two-step loop paradigm: generating pseudo-labels using the DBSCAN algorithm [50] to create memory banks with clustering centers and establishing cross-modality correspondences based on these memory banks. However, pseudo-labels are often inaccurate and rigid, CCLNet [51] leverages the text information from CLIP to afford greater semantic monitoring insights to compensate for the rigidity of pseudo-labels. Moreover, reliable cross-modality correspondences are vital to USVI-ReID, thus PGM [23] proposes a progressive graph matching framework to establish more reliable crossmodality correspondences. However, cluster centers mainly present common information while lacking distinctive information, which results in ambiguous cross-modality correspondences when meeting hard samples [52, 53] .3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: This work relies on DBSCAN to generate pseudo-labels. However, for extremely large-scale datasets, DBSCAN's performance may be limited, which could affect the overall effectiveness of our approach. To address the limitation, we plan to explore hierarchical clustering in future research to better handle large-scale datasets.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation and overview: 
 Given a USVI-ReID dataset D = {V, R}, where V = {V i } Nv i=1 represents the visible images and R = {R j } Nr j=1 denotes the infrared images. V i and R j represent the set of images corresponding to the i-th and j-th class. N v and N r denote the number of visible and infrared clusters, respectively. In the USVI-ReID task, the purpose is to train a deep neural network to obtain modality-invariant and identity-related features for matching pedestrian images with the same identity.We propose a Progressive Contrastive Learning with Hard and Dynamic Prototype (PCLHD) method for USVI-ReID, which mainly contains online encoder, momentum encoder, and progressive contrastive learning strategy with centroid prototype, hard prototype, and dynamic prototype, as shown in Fig. 1 . The online encoder is a standard network, updated through back-propagation. The momentum encoder mirrors the structure of the online encoder, updated through the weights of the online encoder. The clustering is used to generate pseudo labels for creating cluster-aware memory, and we employ DBSCAN for clustering. PCLHD primarily focuses on representation learning, and we use PGM [23] to aggregate cross-modality memory.Following the USVI-ReID methods [22, 54] , we use centroid prototype contrastive learning to optimize the online encoder in the first state, which includes memory initialization and optimization.Memory Initialization. Let ϕ 0 be the online encoder that transforms the input image to an embedding vector. At the beginning of each training epoch, all image features are clustered by DBSCAN [50] and then each cluster's representations are stored in visible memoryM RGB ={cm v 1 , cm v 2 , • • • , cm v Nv } and infrared memory M IR ={cm r 1 , cm r 2 , • • • , cm rNr }, as follows:EQUATIONEQUATIONwhere | • | denotes the number of instances belonging to specific cluster.During training, we update the two modality-specific memories by a momentum updating strategy [46] . We treat the memory center as a centroid prototype and optimize the feature extractor ϕ 0 using contrastive learning with the centroid prototype, computed as:EQUATIONEQUATIONEQUATIONwhere cm v(r) iis the positive centroid prototype, denoting a query and the prototype shares the same identity. The τ is a temperature hyper-parameter.To ensure that the prototype effectively captures divergence within a identity, we devise a novel hard prototype for contrastive learning, which is referred to as Hard Prototype Contrastive Learning. HPCL is designed to provide a comprehensive understanding of personal characteristics, which benefits its handling of hard samples [47] . We use the online encoder ϕ 0 to extract feature representations, and select k samples that are farthest from the memory center as the hard prototype:EQUATIONhm r j = arg max ∀r∈Rj ϕ 0 (r) -cm r j .Theorem 1. The information entropy of hard sample prototypes is greater than the information entropy of centroid prototypes, thereby preserving greater divergence within the hard memory.Given a set of features {f 1 , f 2 , . . . , f Nc } for class c. The entropy H(cm c ) can be approximated by the entropy of the distribution of the sample means. Considering that cm c is a convex combination of the sample features f i , we have:EQUATIONBy the convexity of entropy, we have:EQUATIONThis inequality implies that the entropy of the centroid prototype is generally lower due to the averaging effect, which reduces the divergence among the samples, leading to lower entropy. Given that hm c is the sample with the maximum individual entropy among the set {f 1 , f 2 , . . . , f Nc }, it follows that:EQUATIONThen, we construct contrastive loss with the hard prototype to minimize the distance between the query and the positive hard prototype while maximizing their discrepancy to all other cluster hard prototypes, as follows:EQUATIONEQUATIONEQUATIONwhere hm v(r) iis the positive hard prototype representation and the τ is a temperature hyper-parameter.Finally, we update the two modality-specific memories with a momentum-updating strategy:EQUATION)where α is a momentum coefficient that controls the update speed of the memories. t and t -1 refer to the current and last iteration, respectively.The hard prototype contrastive learning has two main advantages: For intra-class feature learning, it ensures that the learning process does not just focus on the shared characteristics within a cluster but also considers the diverse elements, which are often more informative. For inter-class feature learning, it is also beneficial for increasing the distances between different persons. In contrast, centroid prototypes tend to average features, lacking diversity, which can affect the network's ability to extract discriminative features.Inspired by MoCo [55] and DPM [56] , we design dynamic prototype contrastive learning in order to preserve the intrinsic variety in sample features. DPCL comprises an online encoder ϕ 0 and a momentum encoder ϕ m . The momentum encoder mirrors the structure of the online encoder, which is updated by the accumulated weights of the online encoder:EQUATION)where β is a momentum coefficient that controls the update speed of the momentum encoder. t and t -1 refer to the current and last iteration, respectively. The momentum encoder ϕ m is updated by the moving averaged weights, which are resistant to sudden fluctuations or noisy updates [55] .We use the momentum encoder ϕ m to extract feature representation and store them in visible memoryDM RGB ={dm v 1 , dm v 2 , • • • , dm v Nv } and infrared memory DM IR ={dm r 1 , dm r 2 , • • • , dm r Nr }.We randomly select M visible/infrared samples from each cluster, denoted as X v i and X r j .as follows:EQUATIONEQUATIONWe select visible dynamic prototype dm v i from DM RGB . In the same cluster, we select the sample farthest from the query image as the prototype. In different clusters, we choose the sample closest to the query image as the prototype:EQUATIONwhere y q and y i represent the pseudo label of the query image and the dynamic prototype, respectively. ∥•∥ denotes Euclidean norm. We obtain infrared prototype dm r j through the same method. The overall optimization goal of DPCL is as follows:EQUATIONEQUATIONEQUATIONwhere dm v(r) iis the positive dynamic prototype representation, i.e., the query image and dynamic prototype have the same identity.DPCL promotes a flexible and adaptable learning process, aiming to minimize discrepancies between samples and their respective dynamic prototypes, rather than rigidly aligning query images with a fixed prototype.In the initial training phases, representations are generally of lower quality. Introducing hard samples at this period could be counterproductive, potentially leading the model optimization in an incorrect direction right from the start [47, 57] . To address this issue, we introduce the Progressive Contrastive Learning, which forms the overall loss function:EQUATIONwhere λ is the loss weight, E CPCL is a hyper-parameter.some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiment: 
 We conduct extensive experiments to validate the superiority of our proposed method. First, we provide the detailed experiment setting, which contains datasets, evaluation protocols, and implementation details. Then, we compare our method with many state-of-the-art VI-ReID methods and conduct ablation studies. In addition, to better illustrate our method, we also exhibit further analysis.If not specified, we conduct analysis experiments on SYSU-MM01 in the all-search mode. Dataset. We evaluate our method on two common benchmarks in VI-ReID: SYSU-MM01 [71] and RegDB [72] . SYSU-MM01 is a large-scale public benchmark for the VI-ReID task, which contains 491 identities captured by four RGB cameras and two IR cameras in both outdoor and indoor environments. In this dataset, 22,258 RGB images and 11,909 IR images with 395 identities are collected for training. In the inference stage, the query set consists of 3,803 IR images with 96 identities and the galley set contains 301 randomly selected RGB images. RegDB is collected by an RGB camera and an IR camera, which contains 4,120 RGB images and 4,120 IR images with 412 identities. To be specific, the dataset is randomly divided into two non-overlapping sets: one set is used for training and the other is for testing.Evaluation Protocols. The experiment follows the standard evaluation settings in VI-ReID, i.e., Cumulative Matching Characteristics (CMC) [73] and mean Average Precision (mAP).Implementation Details. We adopt the feature extractor in AGW [58] , which is initialized with ImageNet-pretrained weights to extract 2048-dimensional features. During the training stage, the input images are resized to 288×144. We follow augmentations in CAJ [1] for data augmentation. In one batch, we randomly sample 16 pseudo identities, and each pseudo identity samples 16 instances. We set M to be 16 for computational convenience. The number of epochs is 100, in which the first 50 epochs are trained by contrastive loss with the centroid prototype. For the last 50 epochs, we train the model by contrastive loss with both the hard and dynamic prototypes. E CP CL is 50. At the beginning of each epoch, we utilize the DBSCAN [50] algorithm to generate pseudo labels. During the inference stage, we use the momentum encoder ϕ m to extract features and take the features of the global average pooling layer to calculate cosine similarity for retrieval. The momentum value α and β is set to 0.1 and 0.999, respectively. The temperature hyper-parameter τ is set to 0.05 and the weighting hyper-parameter λ in Eq.( 23) is 0.5.To comprehensively evaluate our method, we compare our method with 18 supervised VI-ReID methods, 3 semi-supervised VI-ReID methods, and 9 unsupervised VI-ReID methods. The comparison results on the SYSU-MM01 and RegDB are reported in Tab. 1.Comparison with USVI-ReID Methods. As shown in Tab. 1, our method achieves superior performance compared with state-of-the-art USVI-ReID methods. MMM [54] is proposed to establish reliable cross-modality correspondences and is also the current best-performing method. Our method with MMM can achieve 65.9% in Rank-1 and 61.8% in mAP, which surpasses that of MMM by a large margin of 4.3% and 3.9%. Notably, our method even without MMM gains the best performance with 64.4% in Rank-1 and 58.7% in mAP. Although existing USVI-ReID methods mentioned in Tab. 1 have made great progress in the USVI-ReID task, the neglects of divergence and variety hinder their further improvement. They overlook divergence and variety, which often constitutes hard samples. Thus, we propose progressive contrastive learning with hard and dynamic prototypes to mine hard samples, which can guide the model to learn more robust and discriminative features.Comparison with SSVI-ReID Methods. There are three SSVI-ReID methods proposed to alleviate the problem of labeling cost by using a part of annotations. Remarkably, our method achieves superior performance without any annotations, outperforming all existing SSVI-ReID methods that utilize partial annotations. Moreover, the results suggest that our method can significantly reduce the dependency on manual annotations.Comparison with SVI-ReID Methods. Surprisingly, our method without annotation outperforms several SVI-ReID methods, e.g., DDAG [39] , AGW [58] , NFS [59] , LbA [60] . This shows the immense competitiveness of PCLHD compared to SVI-ReID methods that rely on complete data annotations. The superior performance of PCLHD mainly benefits from the hard prototype and dynamic prototype contrastive learning. Additionally, we have to acknowledge that a significant disparity still exists between PCLHD and the state-of-the-art fully-supervised results.We conduct ablation studies on the SYSU-MM01 dataset in both all-search and indoor-search modes to show the effectiveness of each component in our method. The results are shown in Tab. 2.Baseline Settings. We use PGM [23] as our baseline. Although PGM has achieved a promising performance on the USVI-ReID task, the neglect of hard samples hinders its further improvement.The HPCL is proposed to mine divergence. As shown in Tab. 2, When adding the HPCL on Baseline, the performance improves a large margin of 5.8% in Rank-1 and 5.1% in mAP, respectively. It shows that divergence can be effectively mined using hard prototype contrast learning, facilitating the model to learn more discriminative features.The DPCL is proposed to mine variety. The results show that Rank-1 accuracy can be improved by 2.8% in Rank-1 and 2.7% in mAP when adding the DPCL on Baseline, which confirms that contrastive learning with dynamic prototype can learn variety.Effectiveness of PCL. PCL is introduced to smoothly shift the model's attention from commonality to divergence and variety. The results show that Rank-1 accuracy can be improved by about 1% in Rank-1 and mAP compared to adding simultaneously the HPCL and DPCL on the Baseline. This confirms that progressive contrastive learning plays a valuable role in assisting HPCL and DPCL. Surprisingly, contrastive learning with both hard and dynamic prototypes significantly exceeds the baseline by a large margin of 8.1% in Rank-1 and 7.0% in mAP. The HPCL and DPCL can complement each other to learn divergence and variety, which effectively guides the network to learn more robust and discriminative features.Hyper-parameter analysis. Hyper-parameter λ is a weighting parameter to trade-off L HP CL and L DP CL . Fig. 2 (a) presents the results under different values of λ. We can observe that when λ is small, i.e., L DP CL contributes more to the model, the performance degrades. However, when λ is large, i.e., L HP CL contributes heavily to the model, the model both achieves superior performance. Note that when λ = 1, i.e., the proposed method is trained without DPCL, the performance drops significantly. λ is finally set to 0.5 and our method achieves the best performance of 64.4% in Rank-1. Moreover, we also analyze the effect of the number of hard samples at hard prototype. As shown in Fig. 2 (b), we vary the k from 1 to 3 and keep the other hyper-parameters fixed, which shows that PCLHD achieves the best performance when k = 1. Hard samples are distributed in multiple directions, so multiple hard samples cannot be represented by a single prototype. This is why using more hard samples as prototypes leads to a decline in overall performanceThe ARI metric. Following MMM [54] , we utilize the Adjusted Rand Index (ARI) metric for clustering evaluation. The larger the ARI value, the higher the clustering quality. In Fig. 2 (c), "RGB" and "IR" denote the ARI values of visible and infrared clusterings, which can measure the quality of visible and infrared pseudo-labels. "ALL" means the ARI values of overall clusterings, which can evaluate the reliability of cross-modality correspondences. PCLHD surpasses other methods significantly on all of the mentioned ARI values, which demonstrates PCLHD can effectively mine divergence and variety to improve clustering quality.Visualization. As shown in Fig. 3 , we visualize the t-SNE map of 10 randomly chosen identities from SYSU-MM01. Compared to the baseline, the distribution of the same identity from the same modality is more compact and the distance of the same identity from different modalities is closer together. Moreover, some hard samples in the baseline are incorrectly clustered, while these hard samples are well clustered in our PCLHD, which shows the effectiveness of the proposed PCLHD. 
conclusion and limitation: 
 In this paper, we propose a novel method for USVI-ReID called Progressive Contrastive Learning with hard and dynamic prototype (PCLHD), which learns commonality, divergence and variety. To be specific, we design Hard Prototype Contrastive Learning to mine divergent yet significant information and Dynamic Prototype Contrastive Learning to preserve intrinsic variety features. Furthermore, we introduce a progressive learning strategy to incorporate both HPCL and DPCL into the model.Extensive experiments demonstrate that PCLHD outperforms state-of-the-art USVI-ReID methods.This work relies on DBSCAN to generate pseudo-labels. However, for extremely large-scale datasets, DBSCAN's performance may be limited, which could affect the overall effectiveness of our approach.To address the limitation, we plan to explore hierarchical clustering in future research to better handle large-scale datasets.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification:The paper provide open access to the code, with sufficient instructions to faithfully reproduce the main experimental results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The paper specify all the training and test details. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are properly credited and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [No] Justification: The datasets used in this paper, SYSU-MM01 and RegDB, are publicly available and widely used in research. These datasets were collected by their original creators and made accessible for research purposes. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [No] Justification: Since we are using datasets that are already publicly available and have beenextensively used in previous research, and given that the content does not involve sensitivepersonal information, this study did not undergo an independent IRB review. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This work was developed using publicly available datasets and aims to enhance the capabilities of VI-ReID, which plays a vital role in scenarios where traditional ReID systems fail, such as in low-light or nighttime conditions. VI-ReID offers significant benefits in improving security and surveillance by enabling more reliable identification across varying environmental conditions. Importantly, this work raises no ethical, safety, or environmental concerns, and no harm was inflicted on living beings during the research. However, we acknowledge the risk of misuse, particularly privacy invasion if used to track individuals in public spaces without appropriate regulation. While VI-ReID does not directly identify specific individuals, its unauthorized deployment could still result in significant privacy violations. Therefore, public surveillance systems using VI-ReID should be controlled by authorized entities, ensuring proper regulatory frameworks and adherence to ethical standards.NeurIPS Paper Checklist 


Decision: This paper proposes an unsupervised approach to Infrared-Visible person re-identification. The authors propose to use progressive contrastive learning with hard and dynamic prototypes. Hard prototypes are used to find divergent samples, while dynamic ones are used to conserve variability. Reviewers raised a number of technical issues, most of which the authors responded to convincingly (or at least partially) during the rebuttal period. On the positive side, reviewers praise the clarity of technical presentation and motivation, the excellent experimental results, and the innovative use of contrastive learning to enhance discrimination.

The two Ethics Reviewers and *all four reviewers* note that there is absolutely no discussion of limitations of the work and potential societal impacts. During the rebuttal period the authors demonstrated very little willingness to engage at all with the broader implications of surveillance technologies. Although the reviewers (and one Ethics Reviewer) note that the work stands on its own technical merits, this lack of engagement to the point of defensiveness is unacceptable. The authors are encouraged to seriously reflect on these issues in any final version of this work.

Reviews: review: paper_summary: paper_summary: The paper proposes a method for unsupervised visible-infrared person ReID. Towards this goal, it extends the PGM method with a progressive contrastive learning with hard and dynamic prototypes. Initially a typical centroid prototype contrastive learning approach is used. A mixed batch of images is encoded through an online encoder and contrastive learning is used to update this encoder using contrastive learning with the centroid prototypes computed by DBSCAN clustering the training data and updating it with a momentum-updating strategy throughout the training. The paper hypothesizes that additionally hard samples should be used as prototypes to find divergent samples and dynamic prototypes should be used to preserve variability during training. Hard prototypes are selected as those samples within a cluster that are farthest away from the centroid prototypes and dynamic prototypes are random samples from a cluster and encodes them with an encoder that keeps a momentum version of the online weights. Since the hard and dynamic prototypes might not be well-defined at the start of training, the proposed method initially uses the standard centroid prototype contrastive learning and after a certain period of training, changes to a loss function based on hard and dynamic prototype contrastive learning. This setup shows significant improvements of the PGM baseline, but also over the current state of the art.
strength_weakness: strength_weakness: - L215 states that each batch contains 16 person IDs and 16 samples per person. Is this a confusion and it should be based on the assignments from the clustering? Or are these actually person IDs from the annotations? This is kind of a big point, so please make sure to answer this in the rebuttal. I checked some other papers and there similar things are mentioned, but if this is truly based on annotations, the approach CANNOT be called unsupervised, unless you show you can do without annotations here!

- The novelty isn't that great, the base method PGM covers most of the approach and the real novelty lies in a slight modification to the contrastive learning setup for a part of the training. The dynamic prototype learning, as the paper states, is inspired by MoCo and when searching a bit for hard examples in contrastive learning, there is also quite some literature available. Nevertheless, in this specific setting, there is a contribution of showing the improvements are possible.
- I find the related work very weak! The paper is completely focused on the contrastive learning aspect of the PGM method, however, the related work section is purely focused on Visible-Infrared and unsupervised person ReID. Why is there no discussion about unsupervised learning in general, specifically with a focus on contrastive learning and its modifications?!
- While the performance boost in general is quite good, figure 2a does raise the question how stable it is w.r.t to some of the hyperparameters. Then again a fixed value of 0.5 isn't too unexpected and I guess it might have even been an initial default value. Nevertheless an actual validation set for these kinds of ablations would have been valuable, but I guess also uncommon in the community.
- Overall the discussion of the results is often a bit meaningless and just describes the values in the tables, partially even in a bit of a weird fashion. For example L238-240 points out that the proposed method "remarkably" is able to outperform even some semi-supervised methods, but quite some other unsupervised methods in the table also do so. Then in L241-246 it's pointed out that even some supervised methods from 2021 are outperformed, but almost every other supervised method outperforms every single unsupervised method in the table.
questions: questions: - Why are the "Baseline" results in table 2 so different from the PGM results? I get the fluctuation of ~1% for All Search, but why is Indoor Search so much better in your training setup?
- Why did you duplicate almost every single equation just for visible and infrared? This is a major waste of space and could have been used for actually relevant discussions.
- In figure 2c, it very much confused me to see a line between the data points since the different methods are in no way connected.
- Is figure 3 generated based on training of test images?
limitations: limitations: The authors claim "We have not found the limitations of the work." but then again the paper is far from solving the problem. Given that this indicates there are obviously limitations, I find it somewhat lazy to not put some effort into coming up with a meaningful discussion of what still needs to be improved.

And then this is actually outrageous: "There is no societal impact of the work performed." The complete paper is about person re-identification, now even beyond the visible spectrum. Performing this kind of research can have major effects on how effective surveillance is and it's ridiculous to state that there is no impact! I will not judge someone for doing this research out of scientific curiosity, but thinking it has no impact and propagating that belief is not acceptable.
paper_summary: paper_summary: The paper is presented in the field of unsupervised visible-infrared person re-identification (USVI-ReID) which aims to match individuals in infrared images to those in visible images without annotations. Existing methods often use cluster-based contrastive learning, which fails to account for divergencies by focusing solely on commonalities. To overcome this, the authors propose a Progressive Contrastive Learning method with Hard and Dynamic Prototypes, which emphasizes divergencies and feature variety through the use of hard prototypes and a progressive learning strategy.
strength_weakness: strength_weakness: The concept of farthest equal diversity in the HPCL (equations 6 and 7) is based on a strong assumption. Distance is a weak measure and a poor indicator of how the samples within a cluster are actually spread out. While variance is a simple metric, it captures the statistical distribution of the samples within the cluster more effectively.

The DPCL section is not clearly articulated. Specifically, the concept of the query and its application in equation 19 are obscure and require further clarification.

The loss function L_PCLHD  relies on the constant E_CPCL , which is arbitrarily set to 50 without any justification or ablation study to support this choice. From the results presented, it appears that all the ablation studies commence with 50 epochs of CPCL.

Additionally, although it is common practice to integrate the proposed methodology into unsupervised solutions, I believe that knowing the number of classes in advance and setting the number of clusters accordingly constitutes a form of supervision. While it is less intensive than labeling every sample with its respective class, it is still a form of supervision. In my opinion, these methods should be categorized under "unlabeled samples" rather than "unsupervised" learning.
questions: questions: Did the authors explore alternative methods for representing diversity aside from using the farthest sample? If such exploration was undertaken, it would be beneficial to discuss the findings. Conversely, if no alternative methods were considered, does this represent a limitation of the proposed solution? Understanding the rationale behind the chosen approach or the constraints of the methodology would add valuable context to the study.

How does the performance of the proposed solution vary with different values of the E_CPCL constant? Was any ablation study conducted to investigate this aspect? An examination of the impact of varying E_CPCL  on the solution's effectiveness would provide deeper insights into the robustness and adaptability of the proposed method. Such analysis would be crucial for validating the consistency and reliability of the results.
limitations: limitations: The limitations of the proposed approach are not adequately addressed. The weaknesses related to assumptions (e.g. diversity) and constant values definitions, should have brought the authors to a deeper ablation study and thus comprehension of the limitations of the proposed solution.
paper_summary: paper_summary: This paper proposes a method for unsupervised visible-infrared person re-identification called Progressive Contrastive Learning with Hard and Dynamic Prototype. This method focuses on capturing commonality, divergence, and variety. By introducing hard prototype contrastive learning to emphasize divergence and dynamic prototype contrastive learning to preserve intrinsic feature variety, the approach addresses limitations of existing methods. The proposed progressive learning strategy further enhances model robustness and prevents cluster deterioration. Experimental results on two benchmark datasets, SYSU-MM01 and RegDB, show significant improvements in performance, with notable increases in mAP and Rank-1 accuracy compared to the current state-of-the-art methods.
strength_weakness: strength_weakness: 1.While introducing hard prototype contrastive learning can capture divergences, selecting hard samples may lead to model instability during the early training stages. 
2.Although a progressive contrastive learning strategy is introduced to mitigate the above problem, balancing the impact of hard samples remains a challenge.
3.The strategy for selecting dynamic prototypes might require more complex implementation and higher computational resources on larger datasets.
4.Although the paper mentions that the code will be released after acceptance, it currently lacks detailed instructions for reproducibility. 
5.The accuracy of pseudo-labels directly impacts the model training. Some parameters in DBSCAN are very important and should be analyzed further.
questions: questions: 1.Is L_CPCL involved in the optimization process during the second stage?
2.Is the comparison fair with GUR without camera information?
limitations: limitations: The method relies on the DBSCAN clustering algorithm to generate pseudo-labels. For large-scale datasets, DBSCAN might not perform well, thereby affecting the overall method's performance.
paper_summary: paper_summary: This paper proposed a Progressive Contrastive Learning with Multi-Prototype method for USVI-ReID, which learns both shared and diverse information. The Hard Prototype Contrastive Learning method aims to mine distinctive yet significant information, while the Dynamic Prototype Contrastive Learning method is designed to preserve intrinsic diversity features. The experimental results significantly outperform existing methods, validating the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1) From Table 2, it is evident that the method is based on a relatively strong baseline. Does the paper use different baselines, like ADCA, still achieve good results?
2) The paper introduces a momentum backbone in the PCLHD method but does not fully explain its advantages. 
3) Cross-modality associations are especially important for USVI-ReID, what is the association strategy used in this paper? Some details need to be supplemented for clarity.
questions: questions: What insights does this method offer for other multimodal tasks? How can the approach be adapted to enhance performance in different multimodal scenarios?
limitations: limitations: Although the method shows significant performance improvement over other unsupervised methods, there is still a considerable gap when compared to fully supervised methods.

Comments: title: Response to Ethics Review Concerns
comment: Dear Area Chairs and Reviewers 

The recommendation from Ethics Reviewer #ZWH demonstrates a serious bias against surveillance technologies. We contend that technology itself is neutral, and its impact is determined by its specific use case and management policies. It is widely accepted that the primary purpose of technologies is to benefit society and promote progress. Furthermore, many artificial intelligence technologies, particularly in computer vision, are ultimately applied in surveillance systems. This should not negate the valuable contributions made by researchers in these fields.

As the Ethics Review Chairs stated, “ethics reviews are meant to be educational, not prohibitive. Our goal is not to police submissions, but instead to prompt reflection.” Therefore, we sincerely hope that the Area Chairs and Reviewers will not consider this biased opinion, to ensure a fair and objective evaluation of the paper’s technical and scientific contributions. For a detailed discussion of Ethics Reviewer #ZWH’s other comments, please refer to the reply for Ethics Reviewer #ZWH.

Warmest regards,

Authors
Dear Reviewer mgXA,

Thank you for your previous feedback and for mentioning your intention to raise the score. We noticed that the score has not been updated. As the discussion phase comes to an end, we want to ensure that we have thoroughly addressed all of your concerns. If there are any additional points of concern that you believe would contribute to the improvement of our manuscript, please let us know. We genuinely appreciate your expertise and the dedicated time you have devoted to reviewing our work.

Warmest regards,

Authors
We sincerely appreciate your response and for your willingness to improve the score.
Thank you for the response, which addressed my concerns.
Dear Reviewer edeS,

We sincerely appreciate your insightful review, as it plays a pivotal role in enhancing the quality of our manuscript. Furthermore, we have carefully considered your feedback to refine our paper further. As the discussion phase comes to an end, we want to ensure that we have thoroughly addressed all of your concerns. If there are any additional points of concern that you believe would contribute to the improvement of our manuscript, please let us know. We genuinely appreciate your expertise and the dedicated time you have devoted to reviewing our work.

Warmest regards,

Authors
After reviewing the rebuttal carefully, the authors have addressed my previous concerns in detail. I have also considered the opinions of other reviewers:\
(1)I agree with the other reviewers that the authors should add the limitations discussed in the general rebuttal to the original manuscript.\
(2)I agree more with Ethics Reviewer fgoD than with Ethics Reviewer zWHs. I argue this work is largely technical and does not present direct ethical concerns.\
Considering the opinions of other reviewers, I have decided to increase the score from 7 to 8.
Dear Reviewers,\
Thanks very much for your time and valuable comments. We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirm whether you have any further questions? We are very glad to provide answers and revisions to your further questions.\
Best regards and thanks,\
Authors of # 1045
Thank you for your feedback. We appreciate your acknowledgment that our method is developed through experiments on publicly available datasets and does not raise direct issues of ethical, safety, or environmental concerns.\
We acknowledge that any technology can be misused, but we believe risks of ReID can be managed. For example, tracking individuals in public spaces with ReID technology could raise privacy concerns. However, public surveillance systems using ReID are typically controlled by authorized institutions, which reduces the risk of misuse. We will add a discussion on the ethical and social implications in future revisions of our paper to better align with NeurIPS guidelines.
Thank you for your feedback. We understand your concerns, which reflect broader issues that exist across all AI technologies. From a dialectical perspective, AI can be used for both good and ill. However, as the renowned scholar Fei-Fei Li has also stated, “I believe that humanity can effectively address the challenges posed by AI, as long as we adhere to human-centered artificial intelligence.” Besides, the question of whether person re-identification (ReID) is inherently guilty is beyond the scope of this paper, which focuses on the technology itself rather than policy and regulation.\
Here are the answers to your concerns.\
**C1: The work in this paper is akin to performing research on improving facial recognition, as such, my position is that it should not be published.** \
A1: ReID offers greater privacy compared to facial recognition technology. ReID uses structured data instead of personal photos. Specifically, it uses feature vectors from images at a given time, represented as numerical values. Besides, this technology only matches the same individual and cannot identify specific identities.\
**C2: Now considering where infrared technology is most often used (surveillance and military purposes) it seems questionable, at best whether such work should be published at scientific and academic venues. In my opinion such work has no home in either scientific or academic venues.**\
A2: ReID has received significant attention in computer vision for a long time. Many significant ReID studies [1-7] have been published in NeurIPS. ReID cannot be published, many published papers in this field would also need to face similar scrutiny. \
[1] HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception. Junkun Yuan et al. NeruIPS 2023.\
[2] TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification. Shengcai Liao et al. NeruIPS 2021.\
[3] Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID. Yixiao Ge et al. NeruIPS 2020.\
[4] The Dilemma of TriHard Loss and an Element-Weighted TriHard Loss for Person Re-Identification. Yihao Lv et al. NeruIPS 2020.\
[5] Learning Disentangled Representation for Robust Person Re-identification. Chanho Eom et al. NeruIPS 2019.\
[6] FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification. Yixiao Ge et al. NeruIPS 2018.\
[7] Deep Neural Networks with Inexact Matching for Person Re-Identification. Arulkumar Subramaniam et al. NeruIPS 2016.\
**C3: The purpose largely should be to the benefit of society, and (again in my opinion) it is hard to justify how surveillance infrastructures or military applications have improvements for society, narrowly or broadly.**\
A3: It’s important to clarify that our research has always been aimed at using this technology for benefit society, not for military or harmful applications. For example: 1) Helping locate missing persons, such as lost children in public spaces. 2) Enhancing customer behavior analysis, optimizing store layouts, and improving personalized services. 3) Advancing AR experiences through precise tracking of individuals and interactive content integration.\
**C4: The authors do not discuss the limitations of their method or the social implications.**\
A4: We greatly value all reviewers' feedback and will be adding a limitation section as shown in the general response to the manuscript. We believe this addition will provide a more complete understanding of our work.\
**C5: The authors do not discuss the social implications.**\
A5:We acknowledge that any technology can be misused, but we believe risks of ReID can be managed. Specifically, tracking individuals in public spaces with ReID technology could raise privacy concerns. However, public surveillance systems using ReID are typically controlled by authorized institutions, which reduces the risk of misuse.\
**C6: The technology itself lends directly to surveillance, weaponization, and breaches of human rights.**\
A6: As ethics reviewer fgoD noted “This method is largely technical, and is developed through experiments on publicly available datasets. As such, the work developed in this paper does not raise direct issues of ethical, safety, or environmental concerns. No creatures were harmed in the making of this research.” In fact, Our research does not raise direct issues of ethical, safety, or environmental concerns.
We sincerely thank you once again for your valuable comments. We have carefully considered your concerns and have provided our responses below.

**C1: Could you actually elaborate on W3? You plan to add additional related work, but maybe give a list of the papers you plan to discuss here? As is, I can only hope you will come up with a meaningful discussion and that doesn't feel very convincing.**

A1: Thank you for your follow-up question. We plan to additional related work about unsupervised learning in general, specifically with a focus on contrastive learning and its modifications. Specifically, we will discuss the following related works:\
Contrastive learning has achieved remarkable advancements in visual representation learning by pulling positive samples closer together and pushing negative samples further apart in feature space[1,2,6]. In contrastive learning, positive and negative samples play a crucial role and have a direct impact on the model's performance. Therefore, some data augmentation methods[2,3] have been proposed to augment positive pairs to learn discriminative features. For negative pairs, which have received less attention in contrastive learning, MoCo[1,4,5] series construct a memory bank to maintain numerous negative samples for increasing model’s performance. However, these methods define samples from different instances as negative, which is not feasible for ReID. In ReID datasets, many samples have the same identity, causing many positive samples to be wrongly labeled as negative.\
[1]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.\
[2]Xuyang Zhao, Tianqi Du, Yisen Wang, Jun Yao, and Weiran Huang. ArCL: enhancing contrastive learning with augmentation-robust representations. ICLR, 2023.\
[3]Demirel B U, Holz C. Finding order in chaos: A novel data augmentation method for time series in contrastive learning. NeurIPS, 2023.\
[4]Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint, 2020.\
[5]Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.\
[6]Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi. Vision-language pre-training with triple contrastive learning. CVPR, 2022. \
**If you have any more relevant works in mind, we would greatly appreciate your suggestions.**

**C2: Also, as another reviewer highlighted, will you be adding a limitations section to the paper?**\
A2: We greatly value the reviewers' feedback and will be adding a limitation section as shown in the general response to the manuscript. We believe this addition will provide a more complete understanding of our work.

**C3: You have promised to add quite some things to the final paper and I'm wondering how all of this will fit and which parts will be sacrificed and moved to the supplementary to make all of this fit.**\
A3:To accommodate the additional content, we plan to move the discussion of Unsupervised Single-Modality Person ReID in the related work section, as well as Figure 2a and its related discussion, to the supplementary material. This will allow us to maintain the core content of the paper while ensuring that all important details are still accessible.

**C4: In another comment you wrote "For instance, in image-text matching tasks, hard and dynamic prototypes help capture divergence and variety in visual and textual data, leading to better alignment between the two modalities." Is this some idea that you came up with, or is there an actual paper that does this?**\
A4: This idea, to the best of our knowledge, has not appeared in any existing papers. It is an original concept that we plan to explore further as we expand this work.
Thank you once again for providing feedback on our paper and for your willingness to consider improving the score.

**C1: I still have doubts about using distance in HPCL since in many Re-ID solution there is significant effort also in metric learning showing that distance is not capturing all the characteristics of the feature space.**\
A1: We agree with your observation that distance is not a perfect method for capturing all the characteristics of the feature space. **However, the key idea of HPCL is to focus the network on divergence using distance-based hard prototype learning, rather than on measurement methods.** We have used the simple distance measure to verify that focusing on divergence is important. We also acknowledge that there are more complex ways to measure divergence, and we believe that using more advanced metrics could improve our approach.

**C2: Concerning the $E_{CPCL}$ the authors confirm the necessity of ablation on such a parameters since 40 is a better value. What is missing is a comment on why such a parameters has such an impact and with such characteristics.**\
A2: The parameter $E_{CPCL}$ is crucial in determining the timing for focusing on divergence and variety. When $E_{CPCL}$ is set to 20, the hard prototypes are primarily composed of noisy samples. However, when $E_{CPCL}$ is set to 40, these prototypes are more likely to consist of clear hard samples. For instance, at the 20th epoch, the ARI score for IR image pseudo-label accuracy is only 0.68, but it improves to 0.89 at the 40th epoch, which demonstrates how $E_{CPCL}$ affects the quality and composition of prototypes. When $E_{CPCL}$ is set beyond 40, the network has already somewhat converged, which reduces the impact of proposed method.
Thank you so much for acknowledging the strengths of our method again. We have carefully considered your constructive comments and here are the answers to your concerns.

**C1: Do I understand correctly that you do not plan to include a limitation section? If that is the case, I strongly urge you to consider adding such a section.**

A1: We greatly value the reviewers' feedback and will be adding a limitation section as shown in the general response to the manuscript. We believe this addition will provide a more complete understanding of our work.

**C2: In Eq. 19, Does $v_j$ refer to the query image? Could you please clarify this here? Besides, $v_q$ should be $v_j$ in line 180.**

A2: Thank you for your detailed feedback. in Eq. 19, $v_j$ denotes the query image. Besides, We will also correc $y_q$ to$ y_j$ in line 180.

**C3: Please note that there is a typo in line 175.**

A3: Thank you for pointing out the typo. We will thoroughly proofread the manuscript and correct typos to the best of our efforts.
Could you actually elaborate on W3? You plan to add additional related work, but maybe give a list of the papers you plan to discuss here? As is, I can only hope you will come up with a meaningful discussion and that doesn't feel very convincing.

Also, as another reviewer highlighted, will you be adding a limitations section to the paper?

You have promised to add quite some things to the final paper and I'm wondering how all of this will fit and which parts will be sacrificed and moved to the supplementary to make all of this fit.

Finally, in another comment you wrote "For instance, in image-text matching tasks, hard and dynamic prototypes help capture divergence and variety in visual and textual data, leading to better alignment between the two modalities." Is this some idea that you came up with, or is there an actual paper that does this?
Thank you very much for addressing most of my concerns. Considering the other reviews, I believe the revised manuscript will be nice, and I am willing to raise my score.

However, after re-reading the manuscript, I have few new questions and suggestions:

(1) Do I understand correctly that you do not plan to include a limitations section? If that is the case, I strongly urge you to consider adding such a section.

(2) In Eq. 19, Does $v_j$  refer to the query image? Could you please clarify this here? Besides, $y_q$ should be $y_j$ in line 180.

(3) Please note that there is a typo in line 175.
The authors have successfully replied to some of my concerns. I still have doubts about using distance in HPCL since in many Re-ID solution there is significant effort also in metric learning showing that distance is not capturing all the characteristics of the feature space. Anyhow the authors show that simple distance is better than variance. I appreciate the effort. 
Concerning the E_{CPCL} the authors confirm the necessity of ablation on such a parameters since 40 is a better value. What is missing is a comment on why such a parameters has such an impact and with such characteristics.
With the rebuttal I would like to raise my score to borderline accept.