DenoiseRep: Denoising Model for Representation Learning

Authors: zhengrui  xu
xiaowen  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:55:51

Abstract: The denoising model has been proven a powerful generative model but has little exploration of discriminative tasks. Representation learning is important in discriminative tasks, which is defined as *"learning representations (or features) of the data that make it easier to extract useful information when building classifiers or other predictors"*. In this paper, we propose a novel Denoising Model for Representation Learning (*DenoiseRep*) to improve feature discrimination with joint feature extraction and denoising. *DenoiseRep* views each embedding layer in a backbone as a denoising layer, processing the cascaded embedding layers as if we are recursively denoise features step-by-step. This unifies the frameworks of feature extraction and denoising, where the former progressively embeds features from low-level to high-level, and the latter recursively denoises features step-by-step. After that, *DenoiseRep* fuses the parameters of feature extraction and denoising layers, and *theoretically demonstrates* its equivalence before and after the fusion, thus making feature denoising computation-free. *DenoiseRep* is a label-free algorithm that incrementally improves features but also complementary to the label if available. Experimental results on various discriminative vision tasks, including re-identification (Market-1501, DukeMTMC-reID, MSMT17, CUHK-03, vehicleID), image classification (ImageNet, UB200, Oxford-Pet, Flowers), object detection (COCO), image segmentation (ADE20K) show stability and impressive improvements. We also validate its effectiveness on the CNN (ResNet) and Transformer (ViT, Swin, Vmamda) architectures.

Main paper: 
 introduction: 
 Denoising Diffusion Probabilistic Models (DDPM) [21] or Diffusion Model for short have been proven to be a powerful generative model [5] . Generative models can generate vivid samples (such as images, audio and video) by modeling the joint distribution of the data P (X, Y ), where X is the sample and Y is the condition. Diffusion models achieve this goal by adding Gaussian noise to the data and training a denoising model of inversion to predict the noise. Diffusion models can generate multi-formity and rich samples, such as Stable diffusion [50] , DALL [47] series and Midjourney, these powerful image generation models, which are essentially diffusion models. from a low level to a high level. It means that the feature space changes from layer to layer, which contradicts the requirement of the denoising layer.To resolve all the difficulties above and efficiently apply the denoising process to improve discriminative tasks, our proposed Denoising Model for Representation Learning (DenoiseRep) is as below: Firstly, we utilize a well-trained backbone and keep it fixed throughout all subsequent procedures. This step is a free launch as we can easily use any publicly available backbone without requiring additional training time. This approach allows us to preserve the backbone's inherent characteristics of semantic feature extraction. Given the backbone and an image, we can get a list of features. Next, we train denoising layers on those features. The weights of denoising layers are randomly initialized and their weights are not shared. The training process is the same as that in DDPM [21] , where the only difference is that the denoising layer in DDPM takes a dynamic t ∈ [1, T ], and our denoising layers take fixed n ∈ [1, N ], where n is the layer index, T is denoise times and N is backbone layer number as shown in Fig. 1(c ). Finally, considering that the N denoising layers consume additional execution latency, we propose a novel feature extraction and feature denoising fusion algorithm. As shown in Fig. 1(d) , the algorithm merges parameters of extra denoising layers into weights of the existing embedding layers, thus enabling joint feature extraction and denoising without any extra computation cost. We also theoretically demonstrate the total equivalence before and after parameter fusion. Please see Section 3.3 and Eq (7) for more details.Our contributions can be summarized as follows:(1) We propose a novel Denoising Model for Representation Learning (DenoiseRep), which innovatively integrates the denoising process, originating from generative tasks, into the discriminative tasks. It treats N cascaded embedding layers of a backbone as T times recursively proceeded denoising layers. This idea enables joint feature extraction and denoising is a backbone, thus making features more discriminative.(2) The proposed DenoiseRep fuses the parameters of the denoising layers into the parameters of the corresponding embedding layers and theoretically demonstrates their equivalence. This contributes to a computation-efficient algorithm, which takes no extra latency.(3) Extensive experiments on 4 ReID datasets verified that our proposed DenoiseRep can effectively improve feature performance in a label-free manner and performs better in the case of labelargumented supervised training or introduction of additional training data. We also extend DenoiseRep to large-scale (ImageNet), fine-grained (CUB200, Oxford-Pet, Flowers) image classifications, object detection (COCO) and image segmentation (ADE20K), showing its scalability.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: In the abstract and introduction, we have elaborated on the starting point and research direction of the paper, and summarized the contributions of the paper in detail.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Generative models learn the distribution of inputs before estimating class probabilities. A generative model learns the data generation process by learning the probability distribution of the input data and generating new data samples. The generative models first estimate the conditional density of categories P (x|y = k) and prior category probabilities P (y = k) from the training data. The P (x) is obtained by the full probability formula. So as to model the probability distribution of each type of data. Generative models can generate new samples by modelling data distribution. For example, Generative Adversarial Networks (GANs) [17, 43, 23] and Variational Autoencoders (VAEs) [24, 53, 69, 64] are both classic generative models that generate real samples by learning potential representations of data distributions, demonstrating excellent performance in data distribution modeling. Recent research has focused on using diffusion models for generative tasks. The diffusion model was first proposed by the article [51] in 2015, with the aim of eliminating Gaussian noise from continuous application to training images. The DDPM [21] proposed in 2020 have made the use of diffusion models for image generation mainstream. In addition to its powerful generation ability, the diffusion model also has good denoising ability through noise sampling, which can denoise noisy data and restore its original data distribution.Discriminative models learn condition distribution, i.e. P (y|x), where x is data and y is task-specific features. For example, classification tasks [1, 2, 13] map data to tags, retrieval tasks [36, 62] map data to a feature space where similar data should be near otherwise faraway, detection task [49, 27] map data to space position and size. Person Re-Identification (ReID) is a fine-grained retrieval task which identifies individuals among disjoint camera views. Considering its challenge to feature discrimination, we take ReID as the major benchmark task and the others as auxiliary benchmarks. Existing ReID methods can be grouped into hand-crafted descriptors [35, 42, 65] incorporated with metric learning [25, 34, 71] and deep learning algorithms [58, 57, 56, 44, 16, 10] . State-of-theart ReID models often leverage convolutional neural networks (CNNs) [28] to capture intricate spatial relationships and hierarchical features within person images. Attention mechanisms [54, 14] , spatial-temporal modeling [31, 30] , and domain adaptation techniques [9] have further enhanced the adaptability of ReID models to diverse and challenging real-world scenarios.3 DenoiseRep: Denoising Model for Representation LearningQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have explained the limitations of our method and future improvement directions in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
review representation learning: 
 Representation learning plays a pivotal role in discriminative tasks, which is defined as "learning representations (or features) of the data that make it easier to extract useful information when building classifiers or other predictors" [4] . A common architecture of discriminative tasks consists of a vision backbone to extract discriminative features (e.g., ResNet [18] , ViT [14] ) and a task-specific head that operates on these features (e.g. MLP [26] for classification, RCNN [49] for object detection, FCN [40] for segmentation). It is evident that the vision backbone is central to representation learning. In this paper, we introduce a novel Denoising Model for Representation Learning (DenoiseRep), which integrates feature extraction and feature denoising within a single vision backbone. This approach aims to enhance the discriminative power of the features extracted.We refer to the diffusion modeling approach to denoise the noisy features through T-steps to obtain clean features. At the beginning, we use the features output from the backbone network as data samples for diffusion training, and get the noisy samples by continuously adding noise and learning through the network in order to simulate the data distribution of its features.EQUATIONq(x t |x t-1 ) := N (x t ; 1 -EQUATIONwhere X 0 represents the feature vector output by the backbone, t represents the diffusion step size, β t is a set of pre-set parameters, and X t represents the noise sample obtained through diffusion process.In the inference stage, as shown in Fig. 1 (b), we perform T-step denoising on the output features, to obtain cleaner features and improve the expressiveness of the features.EQUATIONEQUATIONwhere X t represents the feature vector output by the backbone in the inference stage. T is the denoising step size, representing the magnitude of the noise. We adjust t appropriately based on different datasets and backbones to obtain the optimal denoising amplitude. According to p θ (x t-1 |x t ) denoise it step by step, and finally obtains X 0 , which represents the clean feature after denoising.As described in Section 3.2, the proposed method above could effectively improve the discriminability of features. Still, extra inference latency is introduced caused by recursive calling of the denoising layers. To solve the problem, we propose to fuse parameters of feature denoising layers into parameters of existing embedding layers of the backbone. The core idea is to expand the linear layer each transformer encoder block into two branches, one for its original embedding layer and the other for extra denoising layer. As shown in Fig. 2 , during the training phase, we freeze the original embedding layers and only train the denoising layers. The training method is consistent with section 3.2, and the features are diffused and fed into the denoising layers. Please refer to Algorithm 1 for more details. In the inference stage, we fuse the pre-trained parameters of embedding and denoising layers, merging the two branches into a single branch without additional inference time. Please note that, here we take the transformer architecture as an example, but DenoiseRep is suitable for CNN architecture. We demonstrate its scalibity on CNN in experiments. The derivation of parameter merging is as follows:EQUATIONwhere a t = 1 -β t , D θ are the parameters of the prediction noise network.EQUATIONWe make a simple transformation of Eq. ( 5) and multiply both sides simultaneously by W . The simplified equation can be obtained by bringing Y t in terms of W X t + b:EQUATIONwhere W D denotes the parameters of D θ (X t , t), X t denotes the input of this linear layer, Y t denotes the output of this linear layer, and Y t-1 denotes the result after denoising in one step of Y t . Due to the cascading relationship of blocks, as detailed in Algorithm. 2, different t values are set according to the order between levels, and the one-step denoising of one layer is combined to achieve the denoising process of Y t → Y 0 , ensuring the continuity of denoising and ultimately obtaining clean features. We split the original single branch into a dual branch structure. During the training phase, the backbone maintains its original parameters and needs to train the denoising module parameters. In the inference stage, as shown on the left side of Fig. 2 , we use the method of reparameterization, to replace the original W parameter with W ′ , whereW ′ = [W -C 1 (t)W W D ]in Eq. ( 7), which has the sameInput: The number of feature layers in the backbone N, features extracted from each layer {F i } N i=1 , the denoising module that needs to be trained{D i (•)} N i=1 . 1: repeat 2:for each i ∈ [N, 1] do 3: t = i: Specify the diffusion step t for the current layer based on the order of layers. EQUATIONTake gradient descent step on ∇ θ ∥ϵ -D i (X t , t)∥ end for 8: until converged number of parameters as W , thus achieving the combination of F C operation and denoising without additional time cost. It is a Computation-free method.In Eq. ( 7), we achieve one-step denoising Y t → Y t-1 . If we need to increase the denoising amplitude, we can extend it to two-step or multi-step denoising. The following is the derivation formula for two-step denoising:EQUATIONWe can obtain this by eliminating Y t-1 from Eq.( 8) and Eq.( 9) and replacing Y t with W X t + b:EQUATIONNote that a single module completes two steps of denoising. To ensure the continuity of denoising, the t value should be sequentially reduced by 2.Our proposed DenoiseRep is based on feature-level denoising and can be migrated to various downstream tasks. It denoises the features on each layer for better removal of noise at each stage, as the noise in the inference stage comes from multiple sources, which could be noise in the input image or noise generated while passing through the network. Denoising each layer avoids noise accumulation and gives better quality output. And according to the noise challenges brought by data in different scenarios, the denoising intensity can be adjusted by controlling t, β t , and the number of denoising times, which has good generalization ability.Input: The number of feature layers in the backbone N, features extracted from each layer {F i } N i=1 , trained denoising module parameters {W Di } N i=1 in Algorithm(1), after obtaining the initial feature F N through patch_embed, it is necessary to remove N-step noise from it, the pre-trained parameters {W i } N i=1 and {b i } N i=1 for the backbone. Output: Feature F 0 after denoising.1:for each i ∈ [N, 1] do 2:t = i: Set the denoising amplitude based on the depth of the current layer.3:W ′ = [W i -C 1 (t)W i W Di ], b ′ = W i C 2 (t)C 3 + b i :Parameter fusion according to Eq.( 7).4:F t-1 = W ′ F t + b ′ :Fuse feature extraction and feature denoising. 5: end for 6: return F 0Our proposed DenoiseRep is label-free because its essence is a generative model that models data by learning its distribution. Thus the training loss contains only the Loss p of denoising layers:EQUATIONwhere ϵ denotes the sampled noise, N denotes the number of denoising layers, X t denotes the noise sample, t denotes the diffusion step, and D θ (X t , t) denotes the noise predicted by the denoising layer.However, it is worth noting that our method is complementary to label if the label is available. Loss l is the task-specific supervised loss with label, λ is the trade-off parameter between two losses. The label-argumented learning is defined as:EQUATIONResults in experiments Section 4.1 shows the improvement from label. Our proposed DenoiseRep is a versatile method that can be incrementally applied to various discriminative tasks. Table 1 demonstrates that DenoiseRep yields stable and substantial improvements across image classification, object detection, image segmentation, and person re-identification. Given that person re-identification is a nuanced image retrieval task that poses a greater challenge to feature discriminability, we take it as our benchmark for model analysis. Details of the experimental settings are provided in Appendix A. Additional experimental results on various tasks are presented in Appendices B, C, D, and E.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
analysis of label informations: 
 Table 2 : DenoiseRep is a label-free method that can also be effectively complemented with labels when they are available. The table below analyzes the effectiveness of using labels. The baseline method, TransReID-SSL, is based on a ViT-small backbone. "Label-free" indicates training without labels, "label-augmented" refers to the use of labels, and "merged dataset" denotes the use of combined datasets without labels. As mentioned in Section 3.3, DenoiseRep is an unsupervised denoising module, and its training does not require the assistance of label information. We conducte the following experiments to identify three key issues.(1) Is this label-free and unsupervised training denoising plugin effective? As shown in Table 2 (line2), compared with baseline method (line1), the baseline method performs better after adding our label-free plugin, which shows that our method does have denoising capability for features.(2) Could introducing label information for supervised training further improve performance? Introducing label information is actually adding Loss l as mentioned in Section 3.4 as a supervised signal. As shown in Table 2 (line3), baseline method with label-argumented DenoiseRep achieve improvements of 0.32% -0.70% on the mAP metric, indicating that our denoising plugin has label compatibility, in other words, the plug-in is effective for feature denoising regardless of labelargumented supervised or lable-free unsupervised training.(3) Since our plugin can perform unsupervised denoising of features, it is natural to think about whether adding more data for training the plugin could further improve its performance? We merge four datasets for training and then test on each dataset using mAP to evaluate. Comparing the results of training on sigle dataset (line2) with on merged datasets (line4), we found that adopting other datasets for unsupervised learning can further improve the performance of DenoiseRep, which also proves that DenoiseRep has good generalization ability.To demonstrate that our method can perform unsupervised learning and has good generalization, we merged four datasets and rearranged the sequence IDs to ensure the reliability of the experiment. The model is tested on the entire dataset. During the training process, we freeze the baseline parameters and only train the DenoiseRep module, without the need for labels, for unsupervised learning. Then test on a single dataset and compare the results of training on a single dataset. As shown in Table 2 , it can be observed that adding unlabeled training data from different datasets can improve the model's performance on a single dataset, proving that this module has a certain degree of generalization.We compare several state-of-the-art ReID methods on four datasets. One of the best performing comparison methods is TransReID-SSL, which is a series of ReID methods based on the ViT backbones. Other methods are based on structures such as CNNs. We add our method to TransReID-SSL series and observe their performance. As shown in Table 3 , we have the following findings: (1) Our method stands out on four datasets on ViT-base backbone with a large number of parameters, achieving almost the best performance on two evaluation metrics.(2) The methods using our plugin outperforms the original methods with the same backbone on all datasets. In addition, the performance improvement of small-scale backbones with the addition of DenoiseRep is more significant than the large-scale backbones approach due to the fact that DenoiseRep is essentially a denoising module that removes the noise contained in the features during the inference stage. For large-scale backbones, the extracted features already have good performance, so the denoising amplitude is limited. It has already fitted the dataset well. For small-scale backbones with poor performance, due to their limited fitting ability, there is a certain amount of noise in the extracted features during the inference stage. Denoising them can obtain better feedback.(3) In fact, our method can be applied to any other backbone, just add it to each layer. In particular, the performance improvement of adding the denoising plugin to a poorly performing backbone might be more significant. This needs to be further verified in subsequent work. However, it is undeniable that we have verified the denoising ability of the DenoiseRep in the currently optimal ReID method.In this section, a comparative analysis was conducted on four datasets to assess various existing ReID methods. These methods represent current mainstream ReID approaches, employing ResNet101, ViT-S, ViT-B, and ResNet50 as backbone architectures for feature extraction, respectively. Experimental results indicate that our proposed method outperforms other approaches in terms of both mAP and Rank-1 metrics.The proposed DenoiseRep is computation-free. In section 3.3, we proved by theoretical derivation that inserting our denoising layer into each feature layer and fusion it does not introduce additional computation. In this section, we also conduct related validation experiments, the results of which are shown in Table 4 . Compare to the baseline method TransReID-SSL, adding DenoiseRep -is able to improve the the performance, proving that feature based denoising is effective. However, it also brings extra inference latency (about 15%) because it is adding an extra parameter-independent denoising module at the end of the model.Adopting DenoiseRep achieves a greater increase, it denoise the features on each layer, which can better remove noise at each stage because the noise in the inference stage comes from multiple aspects, which may be the noise in the input image or generated when passing through the network. Denoising each layer avoids noise accumulation and obtains a better quality output. Most importantly, since the operation of fusion can merge the parameters of the denoising module with the original parameters, the adoption of DenoiseRep does not take extra inference latency cost, which is a computation-free efficient approach.The DenoiseRep is based on denoising at the feature level and demonstrates strong generalization capabilities. To validate this generalization ability, we conduct experiments on other vision tasks to test the effectiveness of the DenoiseRep. We validate the generalization ability of DenoiseRep in image classification tasks on ImageNet-1k [12] datasets and three fine-grained image classification datasets (CUB200 [55] , Oxford-Pet [46] , and Flowers [45] ). The accuracy index is chosen as the evaluation metric to assess model performance.As shown in Table 5 , we compare multiple classic backbones for representation learning on ImageNet-1k, and after adding the DenoiseRep, the accuracy of both top-1 and top-5 metrics improve without adding model parameters. Our method shows significant improvement in accuracy metrics compared to baseline on three fine-grained classification datasets. Prove that the DenoiseRep can improve the model's ability in image classification for different classification tasks. Additionally, our method proves to enhance the model's representation learning ability and extract more effective features  
conclusion: 
 In this work, we demonstrate that the diffusion model paradigm is effective for feature level denoising in discriminative model, and propose a computation-free and label-free method: DenoiseRep. It utilizes the denoising ability of diffusion models to denoise the features in the feature extraction layer, and fuses the parameters of the denoising layer and the feature extraction layer, further improving retrieval accuracy without incurring additional computational costs. We validate the effectiveness of the DenoiseRep method on multiple common image discrimination task datasets. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have added appropriate citation explanations to the existing papers, models, and datasets mentioned in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Datasets and Evaluation metrics. We conduct training and evaluation on four datasets: DukeMTMC-reID [72] , Market-1501 [70] , MSMT17 [61] , and CUHK-03 [33] . These datasets encompass a wide range of scenarios for person re-identification. For accuracy, we use standard metrics including Rank-1 curves (The probability that the image with the highest confidence in the search results is the correct result.) and mean average precision (MAP). All the results are from a single query setting.Implementation Details. We implement our method using Python on a server equipped with a 2.10GHz Intel Core Xeon (R) Gold 5218R processor and two NVIDIA RTX 3090 GPUs. The epochs we trained are set to 120, the learning rate is set to 0.0004, the batch size during training is 64, the inference stage is 256, and the diffusion step size t is set to 1000.Training and evaluation. To better constrain the performance of the denoised features of the DenoiseRep for downstream tasks, we employ alternating fine-tune methods. The parameters of the DenoiseRep and baseline are trained alternately, and when training a part of the parameters, the rest of the parameters are frozen and fine-tuned for 10 epochs at a time, with a total number of epochs of 120. When evaluating, we average the results of the experiments under the same settings for 5 times, thus ensuring the reliability of the data.In the image retrieval task, we also conduct experiments to verify the effectiveness of our method on the vehicle recognition task. Vehicle recognition in practical scenarios often results in images containing a large amount of noise due to environmental factors such as lighting or occlusion, which increases the difficulty of detection. Our method is based on denoising to obtain features with better representation ability. Therefore, we want to experimentally verify whether the DenoiseRep plays a role in vehicle recognition tasks with higher noise levels. We select vehicleID [37] as the dataset, vehicle-ReID [11] as the baseline, and ResNet-50 as the feature extractor for the experiment. In this section, we aim to test the generalization ability of DenoiseRep in other tasks. We conduct experiments on two image classification datasets, ImageNet-1k and Cifar-10. These two datasets are both classic image classification datasets, rich in common images in daily life, and belong to large-scale image databases. ImageNet-1k is a subset of the ImageNet dataset, containing images from 1000 categories. Each category typically has hundreds to thousands of images, totaling over one million images. The Cfiar-10 contains 60000 32x32 pixel color images, divided into 10 categories. Each category contains 6000 images. To evaluate the effectiveness of our method, we use standard metrics, including Top-1 accuracy and Top-5 accuracy, which are commonly used to evaluate model performance in image classification tasks and are widely used on various image datasets, and we conduct detailed comparative experiments on multiple backbones and models with different parameter versions to verify the reliability of our method.As shown in Table 7 , we compare multiple classic backbones for representation learning on two datasets, and after adding the DenoiseRep, the accuracy metrics improves without adding model parameters. Our method demonstrates the capability to enhance the model's representation learning In this section, we aim to test the generalization ability of DenoiseRep in image detection tasks. We conduct experiments on the COCO [22] dataset. The COCO (Common Objects in Context) dataset is a widely used dataset for large-scale image recognition, object detection, and image segmentation, particularly in computer vision tasks. It contains 80 types of objects, such as people, animals, daily necessities, etc., covering various common items in daily life. To verify that our method is model independent, we conduct experiments using different models including Mask-RCNN [19] , Faster-RCNN [49] , ATSS [67] , YOLO [48] , DETR [6] and CenterNet [75] , as well as diverse backbones.To evaluate the effectiveness of our method, we used standard metrics including AP, AP 50 , and AP 75 , which are commonly used metrics in object detection tasks to evaluate model performance, particularly for evaluating the effectiveness of bounding box detection. It is also an important part of the COCO dataset evaluation criteria, which can measure the detection ability of the model in multiple categories and scales. As shown in Table 8 , we compare multiple classic backbone networks across different methods. After adding DenoiseRep, the accuracy index shows improvement without the need for additional model parameters. This indicates that our method enhances the representation learning capability of the model and extracts more effective features through denoising, all while maintaining the same time costs. In addition, DenoiseRep well to generalized to image detection tasks.In this section, our objective is to assess the generalization capability of DenoiseRep in image segmentation tasks. The image segmentation task aims to divide an image into multiple regions in order to identify and understand objects or areas within them. The main challenges it faces include: complex and varied backgrounds that can easily interfere with segmentation results, objects obstruct each other, making segmentation difficult, objects have diverse shapes and may undergo deformations, etc. We conduct experiments on the ADE20K [73] dataset using the current mainstream image segmentation models. ADE20K is a widely used scene segmentation dataset, mainly used for image segmentation tasks. It contains approximately 20000 images and over 150 different object and region categories. We choose mIoU and B-IOU as evaluation metrics to comprehensively evaluate the performance of image segmentation models. MIoU is the average of IoUs for all categories, which can effectively reflect the segmentation ability of the model on different categories. It provides a quantitative model for the accuracy of handling complex scenes by calculating the degree of overlap between the predicted area and the real area. A higher mIoU value means that the model can better identify and segment the target object. B-IoU focuses on evaluating the accuracy of segmentation boundaries and is particularly suitable for object edge segmentation tasks. It provides sensitivity to boundary details by measuring the degree of overlap between predicted boundaries and real boundaries. As shown in Table 9 , we compare two classical backbone networks with different methods. After adding the DenoiseRep, both IoU metrics improve without adding model parameters. Practice proves that our method can improve the representation learning ability of the model and obtain more effective features through denoising without increasing additional time costs. In addition, DenoiseRep well to generalized to image segmentation tasks.Our proposed method DenoiseRep improves the accuracy of current mainstream backbones while ensuring label-free and no additional computational costs, and it has been experimentally verified to be generalizable in multiple image tasks. However, from the experimental results, it can be found that our method has limited improvement in model accuracy when generalized to general tasks, and in order to fuse the parameters of the denoising layer and the feature extraction layer, only one or two steps of denoise for each denoising layer, and the number of denoising layers is not more than that of the feature extraction layer, which limits the denoising intensity. We will continue to explore how to further improve the accuracy of the model without adding additional inference time costs or only adding a small number of additional parameters.NeurIPS Paper Checklist 


Decision: The paper received all accept recommendations. Reviewers found the proposed method novel and interesting. AC agrees with reviewers and is happy to accept this paper for publication.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel denoising model called DenoiseReID, designed to enhance representation learning in person re-identification (ReID) tasks. This approach combines traditional denoising processes with feature extraction through a feature extraction and denoising fusion algorithm (FEFDFA) that incurs no additional computational cost. It incrementally improves the discriminability of features without the need for labels.
strength_weakness: strength_weakness: 1. In the comparative experiments, the authors missed an opportunity to benchmark their method against the latest advancements like CLIP-ReID, restricting comparisons solely to TransReID.
   
2. When contrasted with TransReID, this approach shows a moderate enhancement in performance.
questions: questions: I am particularly concerned about the performance comparison in this paper. Is it necessary to use diffusion models for conducting ReID tasks, and what advantages do they offer compared to CNNs or ViTs? Currently, it seems that their performance improvement is rather modest.
limitations: limitations: yes, the authors adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes a new method, Feature Extraction and Feature Denoising Fusion Algorithm (FEFDFA), which utilizes the denoising ability of diffusion models to denoise the features in the feature extraction layer, and fuses the parameters of the denoising layer with those of the feature extraction layer through parameter fusion, further improving retrieval accuracy without incurring additional computational costs. The effectiveness of the FEFDFA method has been validated on multiple common image discrimination task datasets.
strength_weakness: strength_weakness: 1.In fact, the intermediate layer features of the pre-trained diffusion model can be used directly for the downstream task such as the discrimination task [1][2][3].
The authors need to enrich the Related Work.

2.Line144-146, no evidence provided to support the proposed hypothesis.

3.The authors treat the backbone as a series of denoising layers, so the training loss in Eq. (11) should be the sum of the MSE losses of each denoising layer.

4.The statement is inconsistent.

Line172-173，"we freeze the original parameters and only trained the FEFDFA."

Line509-511，"the parameters of the FEFDFA and baseline were trained alternately."

In addition, unless the baseline and FEFDFA are trained together, Eq. (12) is incorrect.


[1] Mukhopadhyay, Soumik, et al. "Diffusion models beat gans on image classification." arXiv preprint arXiv:2307.08702 (2023).

[2] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." ICCV 2023.

[3] Baranchuk, Dmitry, et al. "Label-Efficient Semantic Segmentation with Diffusion Models." ICLR 2022.
questions: questions: same to the weaknesses.
limitations: limitations: The authors have addressed the limitations.
paper_summary: paper_summary: This mauscript proposes a novel denosing model for representaetion learning and take person re-identification as a benchmark. It unifies the frameworks of feature extraction and feature denoising, where the former progressively embeds features from lowlevel to high-level, and the latter recursively denoises features step-by-step. Besides, a FEFDA is proposed to fuse feature extraction and denoising in a single backbone without changing its structure and taking extra runtime latency. Experiments on ReID, large-scale  and fine-grained classification tasks show its effectivenss.
strength_weakness: strength_weakness: 1. the proposed "Feature Extraction and Feature Denoising Fusion Algorithm" is little similar to reparameterization, please clarify their difference.
2. its application to transformer series are well analyzed, it will be better if peformance on CNN series are shown.
3. representation learning is a wide and fundational conception, its applications on more downstream tasks, such as detection, segmentation even generation, could be analyzed,  in future.
4. The hyper-parameter analysis is missed.
5. This method seems need extra training steps, what if the baseline methods are trained under the same steps?
questions: questions: For questions, please see the weakness in the above section.
limitations: limitations: The authors have discussed the limitations of the proposed method.
paper_summary: paper_summary: This paper proposes DenoiseReID to improve feature discriminative with joint feature extraction and denoising, in which FEFDFA is developed to merge parameters of the denoising layers into embedding layers. Experimental results show the proposed DenoiseReID improves performance.
strength_weakness: strength_weakness: The author solves all my quentions.
questions: questions: N/A
limitations: limitations: Yes

Comments: title: Response by Authors
Responses by Authors
Comment by Authors
Response to reviewer's comment
Reponse to reviewer's comment
Response to Comment by Reviewer 4MBF
Open Discussion
comment: Thank you for your kind update. Should you have any additional comments or questions, please feel free to share them with us. We are more than happy to respond to them.
Thank you for your the positive response. The authors have addressed some of my concerns. I will revise my initial rating, combined with the comments of other reviewers.
We thank the reviewer kind comment. We will keep improving the proposed method and apply it to more downstream tasks in future.
The authors slove all my quentions. Overall, the motivation of this paper is clear, the idea is novel, the proposed is simple yet effective, and the writting is satifactory.
Hi, dear reviewer, we hope our responses have solved your concerns. If you still have some concerns, please feel free to raise them here and we will respond as soon as possible.
We thank the reviewer kind comment. We will keep improving the proposed method and apply it to more downstream tasks in future.
We thank the reviewer kind comment. We will keep improving the proposed method and apply it to more downstream tasks in future.
We thank the reviewer's detailed comment. It seems that there are still some misunderstandings. Please allow us to explain them again. The responses are listed below and free feel to deeply discuss.

**Q: "For Q2"**

A:  As we have clarified in the rebuttal, we DO NOT use any label (i.e. ReID loss is NOT used) in this experiment. The improvement should come from denoising loss. Based on the observation, we "suppose that in the inference stage, the features obtained by backbone extraction are noisy".

**Q: "For Q4"**

A: As far as I know, "unifying feature extraction and feature denoising" is pioneeringly proposed in this manuscript. Thus, we can't provide previous works to prove "why denoising loss and task losses can be used together". However, we show the fake code of our proposed method, hoping this can solve your misunderstanding. Please pay attention to the line "denoised_feats = feats - self.denoise_layer(x)", which we think may cause your misunderstanding. 

```
// Fake Code of the Proposed Algorithm
// Plz note that, this code targets understanding "why reid_loss and denosing_loss can be used together". Thus, we only show some critical logic and details may be missed and inaccurate.

class DenoiseLinear:

	def __init__(self):
		self.linear = nn.Linear()
		self.denoise_layer = nn.Linear()
		
		set_require_grad_false(self.linear)
		
	def forward_train(self, x):
		// basic branch
		feats = self.linear(x)
		
		// denoising branch, this process is the same with DDPM
		gt_noise = sample_from_gaussian()
		pred_noise = self.denoise_layer(x + gt_noise)
		denoise_loss = l1_or_l2_loss(gt_noise,  pred_noise)
		
		// tip to optimzie denoise layer with reid_loss is returning denoised_feats NOT feats
		denoised_feats = feats - self.denoise_layer(x)
		return denoised_feats, denoise_loss
		
	def forward_test(self, x);
		w, b = fuse_weight(self.linear, self.denoise_layer) // compute offline
		denosied_feats = w * x + b
		return denosied_feats


// A Toy ReID Model with 2 linear layers
class ReIDModel: 
	def __init__(self):
		self.linear1 = DenoiseLinear()
		self.linear2 = DenoiseLienar()
	
  // unsupervised manner
	def train_without_reid_loss(self, x):
		feat1, denoise_loss1 = self.linear1.forward_train(x)
		feat2, denoise_loss2 = self.lienar2.forward_train(x)
		return feat2, denoise_loss1 + denoise_loss2

	// supervised manner
	def train_with_reid_loss(self, x, y):
		feat1, denoise_loss1 = self.linear1.forward_train(x)
		feat2, denoise_loss2 = self.lienar2.forward_train(x)
		reid_loss = compute_reid_loss(feat2, y)
		return feat2, denoise_loss1 + denoise_loss2 + reid_loss
	
	def forward_test(self, x):
		feat1 = self.linear1.forward_test(x)
		feat2 = self.linear2.forward_test(feat1)
		return feat2

```

**Q: "For Q1"** 

A: Our proposed method is very different from the related work [1-3]. We summarize them in the table below:

|                     | Usage                                                        | Scalibity                                                    | Improvement and Latency                                      |
| :-----------------: | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
|        Ours         | Improve existing vision models, which could be from many vision tasks. | [better] ONE method for MANY vision taks without customizing implementation and hyper-parameters. | [better] Given a model of specific vision task (e.g. CLIP-REID), achieve STABLE improvement with NO extra latency compared to the given model. |
| Related Works [1-3] | Customize a model for a specific vision task based on a well-trained strong diffusion model. | ONE method for ONE vision task. Need customize Implementation and hyper-parameters for specific tasks. | Given a diffusion model (e.g. StableDiffusionXL),  improvement and latency depend on how strong the diffusion model is and the details of the customized implementation for specific tasks. |
Thanks for the authors' feedback. I am still concerned about the following issues.

For Q1, I think the motivation of this paper is the same as that of [1, 2, 3], both of which apply denoising features to downstream tasks. I think it makes more reasonable to directly use the features of the pre-trained diffusion model.

For Q2, the results in Table 3 cannot provide strong evidence for the hypothesis stated by the authors in Line144-146 of the paper. Since the ReID loss is used to optimize the parameters of FEFDFA and the parameters of the FEFDFA are fused with the backbone in the final, I suspect that the performance improvement in the Table 3 is more like the data augmentation effect brought about by adding noise, rather than denoising. 

For Q4, unless the authors provide relevant theory or previous works to prove that denoising loss can be used with other task losses to optimize diffusion models or visual task models, I believe that the training loss Eq.(12) is questionable. Denoising in diffusion is fundamentally different from other downstream visual tasks as the optimization objective of them are inconsistent.
The relevant paper listed by the author cannot provide evidence to the rebuttal.
DiffDet[4] applies the idea of denoising to the bbox regression in object detection.
DiffSeg[5] directly uses the intermediate attention layer features of the stable diffusion for the segmentation task, which is similar to the method [1,2,3] mentioned in Q1 that directly uses the intermediate layer features of the pre-trained diffusion model.

[1] Mukhopadhyay, Soumik, et al. "Diffusion models beat gans on image classification." arXiv preprint arXiv:2307.08702 (2023).

[2] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." ICCV 2023.

[3] Baranchuk, Dmitry, et al. "Label-Efficient Semantic Segmentation with Diffusion Models." ICLR 2022.

[4] Chen, Shoufa, et al. "Diffusiondet: Diffusion model for object detection." ICCV 2023.

[5] Tian, Junjiao, et al. "Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion." CVPR 2024.
After carefully reviewing the author's rebuttal, I believe that the concerns I previously raised have been thoroughly addressed. I have decided to accept this paper for the following main reasons:

This paper presents a novel representation learning denoising model for person re-identification, named DenoiseRelD. By integrating feature extraction and denoising techniques, the model significantly enhances feature discriminability. The approach is both theoretically elegant and practically feasible. The author has clearly articulated the motivation and design details of the DenoiseRelD method in both the paper and the rebuttal. Compared to existing methods, DenoiseRelD demonstrates outstanding performance across four benchmark datasets, showing significant advantages.

Given these reasons, along with the author's effective response to the concerns raised by other reviewers, I have decided to revise my initial rating to  'Strong Accept'.
Thanks for authors's feedback. After reading it, all of my questions have been addressed.

I also read the comments from other reviewers and notice an extra point "minimal gains on TransReID".  The authors claim that 1.1-1.6% extra gain based on MaskRCNN and 0.6-0.8% extra gain based on latest state-of-the-art CLIP-ReID. I thought this is a satisfying improvement.

I prefer its novelty, concision, and its advantages of no-extra-lancy, stable improvements and scalibity to many downstream tasks. In summary, I keep my original rating (strong accept). Looking forward to seeing its application to more tasks.
Hi dear reviewers, we thank all your professional and kind comments. We have carefully read and responded to all the comments. Please allow me to clarify some misunderstandings and highlight our contributions.

Our proposed method is test-time computation-free (FREE LUNCH!) and scalable to various baseline and downstream tasks, which has been proven in CNN series (e.g. ResNet50), Transformer series (e.g. ViT, Vmanba), Person Re-ID (MSMT, Duke), Vehicle Re-ID (vehicReID), Large-Scale Image Classification (ImageNet), Fine-Grained Image Classification (CUB) and Object Detection (COCO). For its scalability, we DO NOT hack it for backbones or tasks. With the same hyper-parameters, ours achieve consistent and stable improvement on ALL backbones and tasks above. 

For example, based on the famous object detection method Mask-RCNN, our proposed method achieves extra improvement from 1.6%-1.1% on COCO. Based on previous state-of-the-art CLIP-ReID, our proposed method achieves extra improvements from 0.8%-0.6% on Duke, MSMT and Market.  All those improvements take NO extra test-time computation and NO specific hyper-parameters customized by tasks.

Considering the advantages of **test-time computation-free** and the **consistent and stable improvement on the MANY backbones and tasks** (CNN series, Transformer series, Person ReID, Vehicle ReID, Occluded ReID, Large-Scale Image Classification, Fine-Grained Image Classification, Object Detection), we believe the proposed method will make significant contributions to the Representation Learning community.

Code will be released in future.