Disentangling Identity Features from Interference Factors for Cloth-Changing Person Re-identification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-15 23:51:37

Abstract: Cloth-Changing Person Re-Identification (CC-ReID) aims to accurately identify a target person in the more realistic surveillance scenario where clothes of the pedestrian may change drastically, which is critical in public security systems for tracking down disguised criminal suspects. Existing methods mainly transform the CC-ReID problem into cross-modality feature alignment from the data-driven perspective, without modelling the interference factors such as clothes and camera view changes meticulously. This may lead to over-consideration or under-consideration of the influence of these factors on the extraction of robust and discriminative identity features. This paper proposes a novel algorithm for thoroughly disentangling identity features from interference factors brought by clothes and camera view changes while ensuring the robustness and discriminativeness. It adopts a dual-stream identity feature learning framework consisting of a raw image stream and a cloth-erasing stream, to explore discriminative and cloth-irrelevant identity feature representations. Specifically, an adaptive cloth-irrelevant contrastive objective is introduced to contrast features extracted by the two streams, aiming to suppress the fluctuation caused by clothes textures in the identity feature space. Moreover, we innovatively mitigate the influence of the interference factors through a generative adversarial interference factor decoupling network. This network is targeted at capturing identity-related information residing in the interference factors and disentangling the identity features from such information. Extensive experimental results demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art methods. Our source code is available in the supplementary materials.

Main paper: 
 related work 2.1 person re-identification: 
 Person ReID aims to retrieve images of the same individual from a gallery taken by non-overlapping cameras in the RGB image space. Existing deep learning-based person ReID methods generally fall into two main categories. Firstly, improvement of pedestrian feature extraction network structures [32, 36, 42] , involves extracting fine-grained local features to enhance the model's attention to detail features. For instance, [32] extracts local features by dividing the feature map into vertical parts. Secondly, development of metric learning loss functions [3, 11, 35] , involves introducing contrastive loss [35] and triplet loss [11] into person ReID.However, these traditional person ReID methods have encountered significant performance degradation under poor lighting conditions. Consequently, many researchers have proposed valuable methods in cross-modality person ReID. Unlike single-modality person ReID methods, the main challenge of cross-modality person ReID lies in the heterogeneous modality discrepancy [4] , with most methods achieving this by learning modality-shared information to suppress modality discrepancy. Some approaches [6, 23, 37] achieve image-level modality alignment by generating intermediate modalities, while others achieve feature-level modality alignment through metric learning methods [13, 41] . Cross-modality person ReID and Cloth-changing person ReID(CC-ReID) share similarities in that, both require exploring shared knowledge and suppressing the drastic variances within intra-pedestrian images. Unlike person ReID with RGB images, where clothing remains constant, CC-ReID presents a greater challenge by requiring the model to extract identity features that are not influenced by clothing changes.CC-ReID becomes increasingly critical for long-term person ReID in surveillance scenarios. To address the challenge of changing clothes over time, an increasing number of researchers focus on CC-ReID and have made significant contributions. Existing CC-ReID methods can be roughly categorized into single-modality and multi-modality methods.Single-modality methods [9, 12, 18, 40] usually aim to explore cloth-irrelevant features just from RGB images. For instance, [9] penalizes the model's ability to predict clothes by an additional clothing classifier, [40] proposes to reduce the information from clothing through causal intervention. Although single-modality methods reduce the workload during data preparation, the scarcity of clothes diversity in training images makes it difficult to achieve better results from a single modality alone. Multi-modality works [10, 26, 31] adopt human parsing maps to obtain the foreground cloth region, creating intermediate images through cloth erasure. Besides, gait information [21] and shape information [16] are also adopted to make the model focus more on the pedestrian's latent identity information. Some methods [20, 38] also utilize GANs to change clothes of images for enriching the dataset. These multimodality methods often rely too much on the quality of the generated images. When the quality of the generated images is low, the model may learn irrelevant features. Unlike these methods, we first design a Dual-stream Identity Feature Learning Framework (DSIFLF) to encourage the model to ignore the texture information of clothing. Furthermore, a specifically designed generative adversarial interference factor decoupling network (GAIFDN) is utilized to eliminate interference information, 𝑖.𝑒., clothes and viewpoint changes, thus enhancing the robustness of identity features.Generative Adversarial Networks (GANs) involve continuous adversarial training between a generator and a discriminator, ultimately producing data that the discriminator finds challenging to differentiate. GANs are originally proposed from the perspective of game theory [8] , and thereafter, they are widely applied in the field of image generation [1, 19, 28, 44] . Among these works, CycleGAN [44] achieves image generation through cycle consistency constraints.GANs have many applications in CC-ReID [12, 20, 38] . However, unlike these methods that utilize GANs to change the clothes in pedestrian images, we design GAIFDN, which extracts identityrelated information from the interference-factor-oriented features. Through disentanglement methods, the model is able to learn more robust and discriminative identity features. 
method 3.1 problem definition: 
 Let X = {(X 𝑖 , 𝑦 𝑖 , 𝑦 clo 𝑖 , 𝑦 cam 𝑖 )} 𝑁 𝑖=1represents the training dataset containing 𝑁 images from 𝑀 ped pedestrians captured with 𝑀 cam cameras. In practical scenarios, the same pedestrian may appear wearing different clothes, and we assume the total number of clothes in X is 𝑀 clo . Here, X 𝑖 ∈ R 𝐻 ×𝑊 ×𝐶 denotes the 𝑖-th pedestrian image, where 𝐻 , 𝑊 , and 𝐶 represent the height, width, and number of channels, respectively.𝑦 𝑖 ∈ [1, 𝑀 ped ], 𝑦 clo 𝑖 ∈ [1, 𝑀 clo ],and 𝑦 cam 𝑖 ∈ [1, 𝑀 cam ] denote the pedestrian identity, clothes number, and camera number of X 𝑖 , respectively. The target of this paper is to learn a feature extractor capable of discriminating whether two images belong to the same pedestrian ignoring clothes and camera differences.As illustrated in Fig. 1 , we use ResNet50 [14] pre-trained on Im-ageNet [5] as the backbone for extracting discriminative identity features from pedestrian images which are robust against interference factors including clothing and camera view changes. Based on the backbone, we first set up a Dual-stream Identity Feature Learning Framework (DSIFLF) which is constituted by a raw image stream and a cloth-erasing stream, for learning identity features which are independent of the clothing textures. To achieve this target, we introduce a novel adaptive cloth-irrelevant contrastive learning loss constraining the outputs of the two streams. Moreover, we devise a Generative Adversarial Interference Factor Decoupling Network to more thoroughly mitigate the influence of clothing and camera view changes on the robustness of identity features. Concretely, an interference factor recognition branch is first utilized to capture the information related to clothing and camera specifications. Then, we leverage the cyclic generative adversarial learning algorithm [44] to foster a feature disentanglement module which aims to squeeze identity-related information from the features captured by the interference factor recognition branch. Finally, we design an interference factor decoupling objective to guide the learning of identity features without depending on the information related to clothing and camera specifications. Technical details are introduced in the following subsections. •" represents the features of our original images, while "△" represents the features of images with clothes erased. It can be observed that, given an image in which the pedestrian wearing any clothes, the feature distance between it and its cloth-erased counterpart is usually to be smaller than the intra-pedestrian feature distance.Framework (DSIFLF)The change of clothes increases the intra-pedestrian appearance variance due to the texture differences between different clothes, which increases the variance in the identity feature space as illustrated by Fig. 2 . This increases the difficulty in the ReID of pedestrians wearing substantially different clothes and the discrimination between different pedestrians wearing similar clothes. To address this issue, we propose a novel dual-stream identity feature learning framework through binding a raw image and a cloth-erasing streams. Conventional pedestrian-discriminative feature learning, prototype-based cross-stream matching, and novel cloth-invariant contrastive learning objectives are adopted for optimization in this framework.3.3.1 Dual-stream Identity Feature Extraction. We split the ResNet50 backbone into two components: a shared image encoder (denoted as F sh (•)) composed of the first four layer blocks, and an identity feature extractor (denoted as F id (•)) consisting of the fifth layer block. A pedestrian classifier denoted as C ped (•) is attached after F id (•) to recognize the pedestrian identification of input images. The calculation process of the raw image stream is as follows. Given an input image X 𝑖 , a 𝐷-dimensional identity feature vector denoted as f 𝑖 can be generated via f 𝑖 = F id (F sh (X 𝑖 )). The pedestrian classifier predicts a 𝑀 ped -dimensional probability vector denoted as p 𝑖 from f 𝑖 , i.e., p 𝑖 = C ped (f 𝑖 ). The cloth-erasing stream is implemented as follows. First, we extract the clothes region using the SCHP model [24] , which derives a binary mask S 𝑖 . Then, an augmented counterpart (denoted as X𝑖 ) for X 𝑖 is obtained via X𝑖 = X 𝑖 • (1 -S 𝑖 ), where • denotes the element-wise product. Such a clothes masking operation is capable of removing the clothes textures. We feed X𝑖 through the shared image encoder and identity feature extractor, which derives the other feature vector f𝑖 , i.e., f𝑖 = F id (F sh ( X𝑖 )).Finally, we input f𝑖 into the pedestrian classifier, generating a probability vector p𝑖 , i.e., p𝑖 = C ped ( f𝑖 ).3.3.2 Pedestrian-Discriminative Feature Learning. The cross-entropy loss function is applied for constraining the predictions p 𝑖 and p𝑖 of the pedestrian classifier:EQUATIONwhere 𝑝 𝑖 [𝑚] and p𝑖 [𝑚] denote the 𝑚-th elements of p 𝑖 and p𝑖 , respectively. The above objective can make the backbone model focus on extracting features capable of discriminating pedestrian identities.Moreover, the triplet loss [15] denoted as 𝐿 tri is used to regularize the outputs of the feature extractor for improving the interpedestrian discriminability:EQUATIONwhere S + 𝑖 represents the indices of images having the same pedestrian identity with X 𝑖 and S - 𝑖 represents those of images having different pedestrian identities with X 𝑖 . dist(f 𝑖 , f 𝑗 ) calculates the cosine distance between f 𝑖 and f 𝑗 , i.e., dist(f 𝑖 , f 𝑗 ) = 1cos(f 𝑖 , f 𝑗 ). 𝜖 is a constant.Cross-stream Matching. The cloth-erased images have a distinct feature distribution compared to raw images. This impacts the representation learning of the shared image encoder and the identity feature extractor. To address this issue, we devise a prototype-based cross-stream matching objective for pulling close the feature distributions of raw and cloth-erased images. First, for estimating the feature distributions of the 𝑚-th pedestrian, we construct two prototypes denoted as o 𝑚 and ô𝑚 by accumulating features of raw and cloth-erased images, respectively. The exponential moving average is used to achieve the calculation of o 𝑚 and ô𝑚 , i.e.,EQUATIONô𝑚EQUATIONwhere 1(•) is the indicator function. The second term of Eq. ( 3)or (4) aims to calculate the average of each pedestrian's identity features extracted by the current model. The averaged identity features are used to update the pedestrian prototypes iteratively. 𝛾 is a hyper-parameter controlling the updating rate of pedestrian prototypes.Afterwards, we calculate the prototype-based cross-stream matching loss as follows,EQUATION'•' denotes the inner product operation. The above loss attracts the identity features of one stream to the prototypes estimated from identity features of the other stream, hence achieving the target of reducing the feature distribution gap between raw and cloth-erased images.Cloth-irrelevant Contrastive Learning Loss. For mitigating the variance caused by clothes textures, we propose a adaptive cloth-irrelevant contrastive learning loss objective which helps foster identity features with more compact intra-pedestrian relationship structures. First, we estimate the minimal intra-pedestrian distance of each raw image in the identity feature space. Subsequently, we use it to restrict the distance between identity features of the raw and cloth-erased images as follows,EQUATIONsg(•) denotes the stop-gradient operation. 𝜔 𝑖 is a weighting coefficient for measuring the identity discriminability of X𝑖 , which is calculated from the prediction of the pedestrian classifier namely p𝑖 . The calculation formulation is as below,EQUATION)where 𝜏 is a temperature coefficient. The target of the loss term in Eq. ( 6) is: given an image in which the pedestrian wearing any clothes, the feature distance between it and its cloth-erased counterpart is constrained to be smaller than the minimal intra-pedestrian feature distance. This means that the differences to the identity features brought by the cloth textures would not surpass the normal feature fluctuation caused by other factors such as pose changes, under the condition of the same pedestrian identification. Hence, this loss term is beneficial for fostering identity features which are irrelevant to the change of clothes textures. The introduction of the weighting coefficient is motivated by the phenomenon that, the segmented clothes regions may be inaccurate, leading to severe corruption on the identity discriminability for cloth-erased images.The importance of such images needs to be weakened in Eq. ( 6).The overall objective denoted as 𝐿 dual for the dual-stream identity feature learning framework is formed by combining Eq. ( 1), ( 2), ( 5), and:EQUATIONwhere 𝛽 is a constant. This framework ensures the learned identity features to be discriminative among different pedestrians while irrelevant to diversified clothes textures. In the subsequent subsection, we concentrate on decoupling identity features from interference factors more thoroughly.The pedestrian identification is moderately coupled with clothes and camera specifications, e.g., one person may frequently wear the same clothes or appear under the same camera view in a period of time. The pedestrian classifier may inadvertently rely on such factors for accomplishing the task of pedestrian recognition, which hinders the exploration of robust identity features. To effectively counteract the influence of interference factors including clothes and camera view changes on identity features, we propose a generative adversarial interference factor decoupling network which is composed of an interference factor recognition branch, a generative adversarial feature disentanglement module, and an interference factor decoupling objective.Factor Recognition Branch. We position a specific feature extractor denoted as F int (•) after the shared image encoder. This configuration facilitates the extraction of features specifically targeted at recognizing interference factors. We denote the features extracted for X 𝑖 as f int 𝑖 , i.e., f int 𝑖 = F int (F sh (X 𝑖 )). Subsequently, we employ two specialized classifiers to ascertain the clothing and camera numbers from f int 𝑖 , yielding outputs p clo 𝑖 and p cam 𝑖 , respectively.The training of these classifiers is governed by a cross-entropy loss function, designed to fine-tune their accuracy in predicting clothing and camera numbers:EQUATION3.4.2 Generative Adversarial Feature Disentanglement Module. To squeeze identity-related information from the above interferencefactor-oriented features, we introduce a generative adversarial feature disentanglement module, inspired from [44] . This involves the construction of a feature generator G id (•), which extracts out identity-related information from the interference-factor-oriented features; the other feature generator G int (•) is targeted at grasping interference-factor-related features from identity features. Two discriminators, denoted as D id (•) and D int (•), are used to assess the authenticity of the features within the domains of identity features and interference-factor-oriented feature, respectively. The discriminators are optimized through an objective function that enhances their ability to discern real identity features from those translated from interference-factor-oriented features, and conversely:EQUATION) In the above objective function, the first two terms constrain the learning of D id (•), while the last two terms constrain the learning of D int (•).The generators are subject to an objective (𝐿 gen ) comprising an adversarial loss (𝐿 adv 𝑖 ) and a cycle consistency regularization 𝐿 cyc 𝑖 :EQUATIONEQUATION)The adversarial loss in Eq. ( 12) effectively facilitates feature domain translation, while the consistency regularization in Eq. ( 13) maintains a cohesive relationship between identity and interferencefactor-oriented features.To remove the contribution of interference factors, we reallocate the output probabilities of the pedestrian classifier. Then, the following objective is used to constrain the reallocated output probabilities:EQUATION)wherep ′ 𝑖 = C ped (G id (f int 𝑖 )). This loss function ensures the classification decision-making process becomes independent of interference factors, thereby directing the feature extractor to disregard features associated with such factors.The overall training loss for optimizing network parameters excluding those of the generative adversarial feature disentanglement module is formed by combining equations ( 8), (9) , and ( 14):EQUATIONwhere 𝛼 and 𝜂 are both constant.In the inference phase of our method, we are tasked with identifying the correct matches between a set of gallery images, denoted asG = {G 𝑖 } 𝑁 gal 𝑖=1, and a set of query images, denoted asQ = {Q 𝑗 } 𝑁 que𝑗=1 . We accomplish this by calculating the distances between all pairs of gallery and query images. We feed G 𝑖 and Q 𝑗 into the feature extractor, generating feature vectors g 𝑖 and q 𝑗 , respectively. Subsequently, we utilize the camera classifier, trained as part of the interference factor disentanglement module, to assign camera numbers denoted as 𝑐 gal 𝑖 and 𝑐 que 𝑗 to G 𝑖 and Q 𝑗 , respectively. Considering images captured by the same camera usually share relatively higher visual similarities than those captured by different cameras, we use these predicted camera labels to adjust the cosine distance between g 𝑖 and q 𝑗 . The formulation for calculating this distance denoted as 𝑑 𝑖 𝑗 is as follows:EQUATION)Here, we decrease the cosine distance between g 𝑖 and q 𝑗 by a constant threshold denoted as 𝛿 if the inferred camera numbers 𝑐 gal 𝑖 and 𝑐 que 𝑗 are consistent. With help of the above distance measurement, we can easily find the gallery images which are most similar to each query image, hence accomplishing the pedestrian re-identification task. 
experiments 4.1 datasets and metrics: 
 We adopt the following three commonly used CC-ReID datasets (𝑖.𝑒., PRCC, VC-Clothes and LTCC) in our experiments. PRCC [39] dataset consists of 33,689 images from 221 identities. Each person in Camera A and B wears the same clothes. While for Camera C, the persons wear different clothes, and the images are taken at different times. VC-Clothes [34] is a synthetic dataset, containing 19,060 images rendered by the GTA5 game engine. These images are captured from 512 pedestrian identities under 4 cameras. Each person has 1 to 3 outfits. LTCC [30] is an indoor CC-ReID dataset, which has 17,138 images of 152 identities, with 478 different outfits captured from 12 camera views.We adopt the standard rank-1 accuracy and mean average precision (mAP) for evaluation. In line with prior studies [9, 40] , we verify our model through the following three settings: 1) General Setting: The gallery set includes both cloth-changing and clothconsistent samples; 2) Cloth-Changing Setting: The gallery set only consists of cloth-changing samples; 3) Cloth-Consistent Setting: The gallery set only consists of cloth-consistent samples; For the LTCC and VC-Clothes datasets, we report Rank-1 accuracy and mAP within the general and cloth-changing settings. Regarding the PRCC dataset, we report the Rank-1 accuracy and mAP under the cloth-consistent and cloth-changing settings.We adopt ResNet50 [14] pre-trained on ImageNet [5] to form our backbone network. The traditional image parsing method SCHP [24] is utilized to implement the cloth erasing, where we directly perform inference on the training set with the provided pre-trained model. The input images are first resized to 384 × 192, and then subjected to random horizontal flipping, random cropping, and random erasing [43] . The training batch size is 32, with 4 person identities and 8 instances for each. The Adam [22] optimizer is used with initial learning rate of 3.5 × 10 -4 , which is divided by 10 every 20 epochs, for a total of 80 epochs. The discriminator is trained for 2 epochs, and the generator for 5 epochs, with training proceeding in an alternating fashion. The triplet margin 𝜖 in Eq. ( 2) and ( 6) is set to 0.3, the parameter for momentum update 𝛾 in Eq. (3), and ( 4) is set to 0.01, and the temperature factor 𝜏 in Eq. ( 7) is set to 0.05 across all datasets. In Eq. ( 15), the parameters 𝛼 and 𝛽 are set as 1.0, 0.05 for the LTCC dataset, 0.1 and 1.0 for PRCC and VC-Clothes, respectively. The constant threshold 𝛿 in Eq. ( 16) is set to 0.3 for the LTCC dataset, and 0.05 for both the PRCC and VC-Clothes datasets.In the experiments, we compare our approach with existing stateof-the-art (SOTA) methods on the LTCC, PRCC and VC-Clothes datasets. Here, we roughly divide these methods based on whether some auxiliary label information (𝑖.𝑒., Clothes ID, Camera ID) is used or not. Specifically, as shown in Table 1 , we compare our method against four methods that use auxiliary labels (i.e. CAL [9] , AIM [40] , CCFA [12] , CVSL [29] ) and ten methods that do not use auxiliary labels. (i.e. HACNN [25] , PCB [32] , IANet [17] , ISP [45] , RCSANet [18] , FSAM [16] , 3DSL [2] , GI-ReID [21] , AFL [27] , ). To reveal the overall performances with/without using auxiliary labels in detail, we have implemented the following four different variants of our method: Variant 1 (denoted as without Auxiliary Label): We completely abandon the Generative Adversarial Interference Factor Decoupling Network module proposed in Section 3.4 and the Camera-Adaptive Inference Procedure introduced in Section 3.5.Variant 2 (denoted as without Camera ID): We remove the use of Camera ID from the method described in Section 3.4 and discard the Camera-Adaptive Inference Procedure mentioned in Section 3.5.Variant 3 (denoted as without CAIP): We only remove the Camera-Adaptive Inference Procedure used for inference, as described in Section 3.5.Variant 4: represents the full assembly of our method. The overall experimental results are reported in Table 1 . We can clearly see that our proposed method surpasses almost all the methods, which outperforms the second-best method by a large margin of 4.74% Rank-1 accuracy and 3.55% mAP on average, over the three datasets under six experimental settings. Compared to other variants of our method without using auxiliary labels, we can conclude that, the designed strategy which properly utilizes the auxiliary cloth and camera IDs, helps a lot to improve the recognition performances. Even without using these auxiliary labels, our method can also outperform existing state-of-the-art method, by a margin of 3.0% Rank-1 accuracy and 1.2% mAP under the same evaluation condition.Effectiveness of the Three Novel Components. There are three novel components in our proposed method, 𝑖.𝑒., 1) the discriminative cloth-irrelevant identity feature learning component (DSIFLF), which optimize the backbone model using the proposed prototypebased cross-stream matching loss 𝐿 mm and the adaptive clothirrelevant contrastive learning loss 𝐿 cicl ; 2) the GAIFDN module cross-stream matching 𝐿 mm in Table 3 . It can be observed that our 𝐿 * cicl improved the rank1/mAP by (0.7%/1.5%, 4.4%/3.3%) on both datasets. By introducing the weighting coefficient sg(𝜔 𝑖 ), our model shows an increase by (0.9%/0.7%, 0.7%/0.2%), which indicates that the incorporation of sg(𝜔 𝑖 ) can alleviate the issues of model training precision degradation due to noise factor from human parsing images. Furthermore, with the introduction of 𝐿 mm , our model achieves an improvement relative to Index 3 by 2.9%/1.7% in PRCC, demonstrating that our 𝐿 cicl has made substantial gains by the proposed prototype-based cross-stream matching loss 𝐿 mm .Parameter Sensitivity Analysis. We analyze the sensitivity of two key parameters in our method, 𝑖.𝑒., the weight 𝛼 in Eq. ( 15) and 𝛽 in Eq. ( 8). We tune the values of each parameter while keeping the others fixed on the LTCC and PRCC datasets. The results are shown in Fig. 3 . The results indicate that if the model excessively focuses on the extraction of irrelevant features, or pays too much attention to the matching between pedestrian images and clotheserased images, then the precision of the model will decrease. Based on these experiments, we set the parameters (𝛼, 𝛽) to (1.0, 0.1) for the LTCC dataset, and to (0.05, 1.0) for PRCC and VC-Clothes, respectively.T-SNE Visualization. We visualize the t-SNE map of six randomly selected identities, as shown in Fig. 4 . The t-SNE map provides a comparative analysis between Ours and the Baseline methods. As shown in the baseline method, some different identities have features that are very close to each other, while the same identity with different clothes has features that are farther apart, leading to erroneous matches during the query (as in the left image with the blue and green identities). Our method increases the distance between features of different identities and enhances the compactness of features within the same identity (as in the right image with the brown and blue).Visualization of Sample Distance Distribution. We visualize the cosine distance distribution of randomly selected 200, 000 negative sample pairs (a) and 200, 000 positive sample pairs, as shown in Fig. 5 . By incrementally integrating the following modules (i.e., GAIFDN, DSIFLF) into the training framework, the peak of the negative sample distance distribution gradually shifts to the right (indicating distance becomes larger). At the same time, the distance distribution of positive samples gradually shifts to the left (indicating a decrease in distance). By comparing the magnitude of this shift, our proposed method primarily acts on increasing the distance of negative sample pairs. This greatly demonstrate the effectiveness of each component in our framework to reduce the interference effects for CC-ReID. 
conclusion: 
 This paper proposes a dual-stream identity feature learning framework to explore discriminative cloth-irrelevant identity feature representations for CC-ReID. Specifically in the cloth-erasure stream, the cloth-irrelevant features can be learned through the proposed adaptive cloth-irrelevant contrastive learning objective, by capturing compact intra-pedestrian structure relationships to mitigate drastic variances caused by clothes characteristics. Meanwhile, to further enhance the discriminability of the internal-identity features, we are the first to propose to address this issue from a reverse perspective by disentangling interference-factor-related identity features for discriminative learning. Specially, we achieve identity feature disentanglement through the proposed adversarial generative noise decoupling network. In the future, we will extend our disentanglement-based method to other cross-modality ReID tasks, as well as some domain generation problems.  
: 
 ReID methods primarily focus on identifying the same pedestrian only on the short-term scenarios with limited time and space span, under the assumption that each individual maintains consistent clothes and accessories. On the contrary, the cloth-changing person re-identification (CC-ReID) concentrates on the more challenging and realistic long-term surveillance scenario where the clothes of the pedestrian may change drastically, which is critical in public security systems for tracking down disguised criminal suspects. Obviously, apart from the general challenges, such as moderate view/pose changes, occlusion, low resolution etc., the CC-ReID faces more severer challenges, 𝑖.𝑒., clothes changes and drastic view changes caused by the long-term applications.Considering that the pedestrian clothes and camera views are often coupled with the identity information, they interfere with the learning of robust identity features. Disentangling identity features from those interference factors is the key to address CC-ReID. Such disentanglement is not a trivial problem. For example, roughly erasing all information from the clothing areas in the image space not only destroys the integrity of the features but also causes the model to lose critical information from the clothing areas.Existing CC-ReID methods mainly focus on learning identity feature representations which are relevant to stable identity-related factors rather than the interference factors, like cloth and environmental characteristics. They can be roughly divided into single-modality and multi-modality based methods. Specifically, the single-modality methods [9, 12, 18, 40] learn cloth-irrelevant identity features solely from the RGB images, which are insufficient for removing the influence of interference factors. The multi-modality methods improve the performance by exploring other synthesized data modalities such as shapes [16] , gait motions [21] , cloth-erased images [31] , or cloth-exchanged images [10] to capture cloth-irrelevant identity features. They mainly transform the interference factor disentanglement problem into cross-modality feature alignment from the datadriven perspective. The auxiliary data modalities used in [16, 21] only preserve limited identity-related information, leading to the under-exploration of identity features. Those methods [10, 31] using cloth-erased images or cloth-exchanged images lack intrinsic consideration of the influence of clothes characteristics and neglect the influence of other underlying interference factors. These issues make them very hard to ensure the robustness of the learned identity features.In this paper, we advocate for disentangling identity features from interference factors in a novel reverse perspective. Namely, we propose to enhance the robustness of identity features through suppressing the intra-pedestrian variances caused by clothes textures and relieving the dependence on interference factors during the discriminative learning of identity features. First, we propose a dualstream identity feature learning framework, consisting of a raw image stream and a cloth-erasing stream. The cloth-erasing stream is implemented based on the image parsing method SCHP [24] , achieving the extraction of identity features unrelated to clothes textures. This dual-stream framework aims to suppress the fluctuations caused by diversified cloth textures in the identity feature space, hence deriving cloth-irrelevant identity feature representations. A cloth-irrelevant contrastive learning objective is introduced to constrain the distance between features extracted by the raw image and cloth-erasing streams within the minimal intra-pedestrian distance. It contributes to building more compact intra-pedestrian relationship structures among the identity features of pedestrian images with diverse clothes. Such intra-pedestrian relationship structures greatly help to obtain cloth-irrelevant identity features, which could mitigate the variance caused by clothes textures. Furthermore, to reduce the negative effects caused by the inaccurate image parsing result for cloth erasure, we further propose to adaptively reweight the cloth-irrelevant contrastive learning objective according to the discriminability of the cloth-erased pedestrian image.To mitigate the influence of interference factors more thoroughly, we also propose a generative adversarial interference factor decoupling network. Specifically, we first design an interference factor recognition branch to capture the interference-factororiented features. Simultaneously, we design a feature disentanglement module for extracting identity-related information from the interference-factor-oriented features. This module is learned in the cycle-consistent adversarial manner [44] , guaranteeing the homogeneity between the generated features and the identity features. Finally, we make the discriminative learning process of identity features independent of identity-related information.In summary, our main contributions are as follows:• We propose a dual-stream identity feature learning framework to obtain discriminative cloth-irrelevant identity feature representations. Specifically, an adaptive cloth-irrelevant contrastive learning objective is introduced to suppress the fluctuations caused by clothes textures in the identity feature space. • To further enhance the robustness of identity features, we introduce a generative adversarial interference factor decoupling network, which can relieve the dependence on interference-factor-related identity information during the discriminative learning of identity features. • Extensive experimental results greatly demonstrate the effectiveness of the proposed method in learning discriminative identity features for CC-ReID, which achieves superior performance to state-of-the-art methods. 
 
 Appendix:  


Decision: The paper initially got accept, two borderline accept and one weak reject. The authors have provided a rebuttal. After checking the rebuttal and comments of the other reviewers, three reviewers are satisfied with the rebuttal and gave two accept and one weak accept. However, the Reviewer HQtq who gave weak reject is still not convinced with the rebuttal due to four reasons. 1) The authors deliberately do not compare with DLAW [a] on the LTCC dataset, which largely outperforms the proposed method. However, in the rebuttal, the authors claimed that the proposed method consistently outperforms DLAW; 2) inconsistent between code and implementation details; 3) missing ablation study; and 4) manipulate the ReID protocol. 

The AC has carefully checked the rebuttal and found that (2) and (4) have been solved and (3) has been partially solved. However, the (1) issue is indeed existed: the authors deliberately do not report the results of DLAW on LTCC dataset and claimed that the proposed method consistently achieves higher results than DLAW on all datasets. Indeed: DLAW (Table I)=35.56%mAP>>this paper (Table I) = 25.1%mAP). Taking the above consideration, the AC thinks this paper is not yet ready for publication in MM and should fairly reconsider the comparison with state-of-the-art methods in next version.

***TPC Addendum ***
The paper received a split in recommendations. Given the importance of the topic, high average scores, and trend toward increase after rebuttal, the TPC suggests that the debate/discussion continue at the conference with a poster presentation.

Reviews: review: paper_summary: paper_summary: This paper proposes a disentangle-based method for Cloth-Changing Person ReID by disentangling identity features from interference factors brought by clothes and camera view changes while ensuring robustness and discriminability. Specifically, It adopts a dual-stream identity feature learning framework consisting of a raw image stream and a cloth-erasing stream, to explore discriminative and cloth-irrelevant identity feature representations. The proposed network targets capturing identity-related information residing in the interference factors and disentangling the identity features from such information.
strength_weakness: strength_weakness: 1)	The proposed method is novel and proposes disentangling identity features from interference factors for CC-ReID.
2)	The experimental results of this paper are superior to those of SOTA methods, and the ablation study is sufficient, which carefully illustrates the effectiveness of all the modules in the proposed method.
3)	This paper is well written. Each component of the proposed method is clearly defined and described in detail, making it easy for readers to follow.
limitations: limitations: 1)	In the adaptative cloth-irrelevant contrastive loss, as shown in Eq.(6), what is the difference between Eq.(6) and the traditional triplet loss? And why the weighting parameter w in Eq.(6) is not trainable, which defined as “sg(w)”?
2)	What motivates Eq.(14)? Why does it use the difference value as the input for cross-entropy loss?
3)	The training procedure for the Generative adversarial feature disentanglement module is unclear, especially for Eq(11-13).
4)	In Eq.(16), how to set the hyper-parameter \sigma?
5)	In Eq.(15), how to set the hyper-parameter \eta? The parameter sensitivity analysis is required.
paper_summary: paper_summary: The paper proposes a novel method for cloth-changing person re-identification. They design a dual-stream identity feature learning framework to explore discriminative and cloth-irrelevant identity feature representations.
strength_weakness: strength_weakness: The motivation is clear. The performance is significantly improved and experiments are abundant.
limitations: limitations: 1、	The model is relatively complex. So, the time complexity and parameters should be analyzed.
2、	The authors employ the cloth-erasure stream. How much impact does this operation have? What color will this operation erase the clothes into? Do different colors have different effects?
3、	There are many new studies about cloth-changing person re-identification. So, you should compare several new papers instead of unimportant methods.
paper_summary: paper_summary: This paper proposes to disentangle identity features from interference factors brought by clothes and camera view changes in cloth-changing person ReID. It adopts a dual-stream identity feature learning framework and introduces a generative adversarial interference factor decoupling network, which can relieve the dependence on interference-factor-related identity information. Experimental results show its effectiveness.
strength_weakness: strength_weakness: -	The paper addresses CC-ReID from an interesting perspective of disentangling interference-factor-related identity features for discriminative learning.
-	Results on three datasets show its effectiveness.
limitations: limitations: -	Many recent SOTA methods [20, a, b, c, d] are not compared in Tab. 1. Some of them achieve much better results without using camera labels or clothing labels than the proposed method, but they are not discussed adequately.

[a] Dual Level Adaptive Weighting for Cloth-Changing Person Re-Identification

[b] Semantic-aware Consistency Network for Cloth-changing Person Re-Identification

[c] Exploring Fine-Grained Representation and Recomposition for Cloth-Changing Person Re-Identification

[d] Exploring Shape Embedding for Cloth-Changing Person Re-Identification via 2D-3D Correspondences

-	Eq. 16 in the paper is inconsistent with the implementation in the code. In Line 77 of “test.py”, “distmat” is multiplied rather than added. There are also other inconsistencies in the code, e.g., Lines 76-77 of “train.py”. Therefore, the results of the paper are doubtful.

-	“We decrease the cosine distance” as mentioned in Line 632, but why it is an addition operation in Eq. 16. Moreover, the testing protocol of ReID would exclude the samples with the same camera labels for each person in the query and gallery sets, so is changing the distance between samples with the same inferred camera numbers and different identities like a trick in testing?

-	Ablation studies of each component in GAIFDN are absent, making it hard to understand the contributions of each design. 

-	The value of $\eta$ in Eq. 15 is not discussed.

-	The proposed method uses different values for different hyperparameters ($\alpha$, $\beta$, $\delta$) on different datasets. As shown in the ablation studies, different values have a great impact on performance, which makes the method unstable and hard to tune for the best hyperparameters in application.

-	There are some inconsistent results in the paper. For example, different cloth-changing results of LTCC in Tabs. 1 (Ours(without Auxiliary label)) and 3 (Index 4); different optimal values of parameters between the text description and Fig. 3 in Parameter Sensitivity Analysis.
paper_summary: paper_summary: The paper propose a method for Cloth-Changing Person Re-ID. It aims at disentangling identity-related features from interference factors like clothing and camera.
strength_weakness: strength_weakness: - Writing is clear and easy to follow
- The idea of disentangling ID-related features from clothing or camera is not novel, but the paper presents some novelty in the method of doing so, e.g. using the idea of CycleGAN to generate interference-factor-oriented features then suppress it from global features
- Experimental results are decent, outperforming previous methods by around 4% in R-1 acc. 
- Ablation studies on the proposed components are quite adequate. 
- Analysis is solid with deep insights into the influence of several details in the framework.
limitations: limitations: - Why using Prototype-based Cross-stream Matching (but not existing techniques, even simple ones like KL divergence) to pull two feature distributions closer is not clarified, making it unconvincing. 

- For all cross-entropy losses (eq. 1 and 9), in the equations, no ground truth labels is input, which is not correct. Normally, both the predicted class and the ground truth class are passed into the CE loss, however, in eq.1 for example, only predictions $p_i$ and $\hat{p}_i$ of raw image and cloth_erased image are input.

- How cloth labels are constructed is not clarified. Is the number of cloth labels equal to the total number of cloth variations in the entire training set (as similar to CAL paper)?

- For feature extractor $F_{int}$, I am unconvinced how it can be trained to specifically recognize interference factors. No rationale for this is provided.

- The inference process is not technically correct. 

       + First, for Re-ID task, it is required that for the same ID, the camera label of the query must be different from that of gallery images. The proposed distance conflicts with the requirement of the task. 
       + Second, the claim "images captured by the same camera usually share relatively higher visual similarities" does not make sense for images of the different ID, making the proposed equation 16 of distance unconvincing.

- The used datasets are quite small-scale. Evaluation on very large-scale datasets with diverse distributions in data like DeepChange [1] would be more convincing. This also validates generalizability of the method.

- Unclear implementation details:

       + Architectures of the generators and discriminators are not provided
       + The authors say: "The discriminator is trained for 2 epochs, and the generator for 5 epochs with training proceeding in an alternating fashion". This is pretty unclear: 
                 (1) whether the gens and discs are trained along with the entire framework (if so, in which stage they are trained? For the remaining epochs, they are frozen?) or separately (if so, before or after?)
                 (2) how could it be alternating while the number of epochs is not equal (2 vs 5)?
       + Why set the constant threshold in eq. 16 to 0.3 for LTCC and 0.5 for the rest two datasets? Is that after trials and errors?

[1] Xu, Peng, and Xiatian Zhu. "Deepchange: A long-term person re-identification benchmark with clothes change." ICCV. 2023.