Title: Diffusion-Driven Progressive Target Manipulation for Source-Free Domain Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Source-free Domain Adaptation

Diffusion Models

Controllable Generation
Abstract: Source-free domain adaptation (SFDA) is a challenging task that tackles domain shifts using only a pre-trained source model and unlabeled target data. Existing SFDA methods are restricted by the fundamental limitation of source-target domain discrepancy. Non-generation SFDA methods suffer from unreliable pseudo-labels in challenging scenarios with large domain discrepancies, while generation-based SFDA methods are evidently degraded due to enlarged domain discrepancies in creating pseudo-source data. To address this limitation, we propose a novel generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) that leverages unlabeled target data as references to reliably generate and progressively refine a pseudo-target domain for SFDA. Specifically, we divide the target samples into a trust set and a non-trust set based on the reliability of pseudo-labels to sufficiently and reliably exploit their information. For samples from the non-trust set, we develop a manipulation strategy to semantically transform them into the newly assigned categories, while simultaneously maintaining them in the target distribution via a latent diffusion model. Furthermore, we design a progressive refinement mechanism that progressively reduces the domain discrepancy between the pseudo-target domain and the real target domain via iterative refinement. Experimental results demonstrate that DPTM outperforms existing methods by a large margin and achieves state-of-the-art performance on four prevailing SFDA benchmark datasets with different scales. Remarkably, DPTM can significantly enhance the performance by up to 18.6\% in scenarios with large source-target gaps.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to the unsupervised source-free domain adaptation problem by leveraging diffusion models to transform unlabeled target images into images of specified categories. Specifically, the method first separates given target samples into a trust set and a non-trust set based on predictive uncertainty. For the samples in the non-trust set, a diffusion model is used to transform them into labeled images while preserving the characteristics of the target domain. By iteratively repeating this process, the authors demonstrate that their method achieves strong domain adaptation performance even in scenarios with a large source-target domain gap.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n  * This work explores a novel approach that leverages the powerful generative capability of pretrained diffusion models to convert given unlabeled target samples into labeled target samples. To preserve target-domain-specific features during this transformation, the authors propose several FFT-based techniques.\n  * The paper evaluates the proposed method across a range of benchmark datasets and compares it with various existing unsupervised SFDA approaches, focusing on scenarios with significant domain shifts.\n\n* Weaknesses\n  * The most significant concern with this work lies in its practical applicability to real-world scenarios. The proposed method involves several hyperparameters, such as the uncertainty threshold $E$, FFT-related parameters, and guidance scales $\\gamma_1$ and $\\gamma_2$, but the paper does not discuss how these values are determined. By definition, in unsupervised SFDA, these hyperparameters must be chosen using only the unlabeled target samples. However, the paper omits any discussion of this critical issue. While prior works have often reported results using hyperparameters tuned based on target-domain performance, I believe this convention should not be followed, as it entirely neglects the practical usability and generalizability of the method. There is a growing body of literature addressing hyperparameter selection in the context of SFDA, and at the very least, the authors should evaluate whether their method remains effective when such methods are employed. Relevant references include:\n    * Saito et al., “Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density,” ICCV 2021\n    * Hu et al., “Mixed samples as probes for unsupervised model selection in domain adaptation,” NeurIPS 2023\n    * Lee et al., “Few-shot fine-tuning is all you need for source-free domain adaptation,” arXiv:2304.00792\n  * I also have concerns regarding the reported effectiveness of the proposed method. Related to the above point, the ablation study on the uncertainty threshold $E$ clearly shows how sensitive the performance is to hyperparameter choices. For instance, Table 6 shows that when $E=0.001$, the average performance drops to 80.7, at which point the method no longer shows meaningful improvements over existing approaches. Additionally, in the performance comparisons, results from baselines such as ProDe appear selectively chosen. ProDe, for example, reports significantly higher performance when using a ViT-CLIP backbone, which is not shown in this paper. This undermines the objectivity of the evaluation and should be avoided.\n  * The proposed method involves several complex components that serve similar functions, yet the effectiveness of this design is only qualitatively illustrated (e.g., in Figure 3). A more thorough, quantitative component-wise analysis is necessary to provide a clearer understanding of the contribution of each part of the method.\n\nQUESTIONS:\n* The diffusion process relies on the assumption that for a sufficiently large $T$, the latent variable $z_T$ follows a Gaussian distribution. However, Eq. (5) assumes that low-frequency information remains in $z_T$. I am curious whether there are specific conditions under which this assumption holds.\n* In the progressive refinement mechanism described in Section 3.4, the manipulation is repeatedly applied to all samples in $U^{r+1}$. However, this manipulation process appears to be independent of the target model $\\phi^r_{trg}$, then already manipulated samples can be reused. Would it not be more reasonable to apply the manipulation only to the newly added samples in $U^{r+1}$? Does this design choice have any impact on the overall performance?\n* The proposed method is based on the assumption that domain-specific and semantic information can be separated via FFT. While this may reasonably hold for the benchmark datasets used in the paper, it is unclear whether the same assumption would remain valid for real-image benchmarks that are commonly used in SFDA, such as TerraIncognita or VLCS.\n\nLIMITATIONS:\n* The authors appropriately acknowledge the limitation that their method relies on a pretrained diffusion model. However, as noted in the weaknesses section, it is also important to discuss the limitations concerning the practical applicability of the proposed approach. In particular, I am concerned that without a clearly defined model selection procedure under the unsupervised SFDA setting, the method may yield results that do not generalize beyond the specific benchmarks used.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called Diffusion-Driven Progressive Target Manipulation (**DPTM**) to tackle the challenge of source-free domain adaptation (SFDA). Specifically, DPTM partitions the target data into trusted and non-trusted sets based on the uncertainty of pseudo-label predictions, quantified by the entropy of the model outputs. A progressive training strategy is then introduced to iteratively refine the model and enhance adaptation performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrenths:\n\n1. **Good Performance**. The proposed DPTM achieves state-of-the-art performance across several widely used SFDA benchmarks, including Office-31, Office-Home, VisDA, and DomainNet-126. In particular, DPTM achieves an avearge classification accuracy of 91.2% on Office-Home, largely surpassing previous approches.\n\n2. **Clear Presentation**. The presentation of the ideas, framework, and experimental results is clear, making the paper easy to read and follow.\n\nWeaknesses:\n\n1. **Limited Contribution to the Community**. \nOne of my main concerns is that in the current era of large multimodal language models (MLLMs), the challenge of domain adaptation has been significantly mitigated by their powerful generalization capabilities. While I acknowledge that domain adaptation still holds practical value in specific application areas with insufficient data, such as the medical domain. Furthermore, the proposed method appears to be a straightforward combination of existing techniques, such as leveraging uncertainty from model outputs, employing a strong image generation model, applying FFT, and iteratvie refinement in , to improve performance. While I appreciate the impressive performance demonstrated on the Office-Home dataset, I believe the paper lacks sufficient novelty and theoretical contribution to meet the standards for acceptance at a top-tier conference like NeurIPS.\n\n2. **Similarity to Existing Work**. \nUsing a powerful diffusion model to generate pseudo-labeled samples and incorporating them as external data for training has already been explored, such as [1] and [2].\n\n3. **Lack of Visualization Results**.\nIt is recommended to include some visualization results illustrating the domain discrepancy. These visualizations would help support the claim that the proposed method effectively reduces domain discrepancy, as performance improvement alone provides only a limited perspective.\n\n[1] Chopra, Shivang, et al. \"Source-free domain adaptation with diffusion-guided source data generation.\" arXiv preprint arXiv:2402.04929 (2024).\n\n[2] Peng, Duo, et al. \"Unsupervised domain adaptation via domain-adaptive diffusion.\" IEEE Transactions on Image Processing (2024).\n\nQUESTIONS:\n1. While it is common to use Fast Fourier Transform (FFT) to extract domain-specific features, I am also curious about the effectiveness of using an image encoder for this purpose. It would be helpful if the authors could provide some insight into how the results compare between these two approaches.\n\n2. I am wondering a more detailed comparison between this work and generation-based SFDA methods. Specifically, generating features and generating samples are fundamentally different, particularly in terms of computational cost. As far as I am aware, the generation-based methods listed in this paper do not generate samples. \n\n3. It is unclear why similar colors are used for target data and target-guided initialization, as well as for target model and semantic feature injection in Figure 1. This similarity may cause confusion and reduce the overall clarity of the figure. Additionally, it could benefit from a clearer and more informative layout. For example, explicitly highlighting the interaction relationships among different color blocks would enhance the reader's understanding.\n\nLIMITATIONS:\nThe major limitation is similar to the first weakness mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n-\tIn the source free domain adaptation (SFDA) task that uses only unlabeled target data and a source model pretrained with labeled source data, the authors propose a generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) to effectively utilize the non-trust set with high uncertainty among the target model's prediction results in the target domain by supplementing it with a diffusion model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tThe proposed method of combining the diffusion model with SFDA significantly improves the performance of SFDA on existing datasets.\n-\tIt is an ingenious approach to use the existing widely used DA technique based on the idea that low-frequency of an image represents domain information and high-frequency represents semantic information, to the starting point data of the diffusion model.\n\nWeaknesses\n\n-\tThe proposed method aims at SFDA, but it seems unfair to compare the performance with existing SFDA methods because it additionally uses a diffusion model with high computational cost in addition to the source model.\n-\tThe computational complexity of the proposed method is expected to be considerably high, and a detailed analysis is required, including comparison with benchmark methods.\n-\tThe reliability of the proposed method's entropy-based Trust and Non-trust Partition is not high. Therefore, the evaluation of the reliability of the entropy-based Trust and Non-trust Partition is required. In fact, the pseudo labeling accuracy of the two sets of needs to be evaluated, and the SFDA performance of the proposed method needs to be evaluated depending on the partition accuracy.\n-\tThe performance improvement is unrealistically high, so it is necessary to present and compare the performance when the model is trained with all samples from the target domain. In addition, considering the current performance improvement of the proposed method, it is expected that the performance of the proposed method should be higher when all samples from the target domain are set as non-trust set.\n-\tThe supplementary material contains repeated experimental results from the manuscript and does not contain the necessary new experimental results mentioned above. In particular, there should be a wider range of examples of samples generated via the Diffusion model in the target domain\n\nQUESTIONS:\n-\tWhen generating samples of the target domain and applying them to train the target SFDA model, it seems that the size of the target SFDA model will have a great impact on the performance. Is there any analysis related to this?\n-\tA description and classification of the types of benchmark methods is needed.\n-\tThe performance improvement is quite extraordinary, and theoretical analysis is essential. The proposed method should quantitatively measure how well the samples generated by the diffusion model represent the target domain.\n-\tHow specifically was the grid search performed to find threshold 𝐸?\n-\tAt line 255, it says ‘we do not specifically optimize the search space’. What does this mean?\n\nLIMITATIONS:\n-\tThe limitations of the proposed technique are stated in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents DPTM, a generation-based method for source-free domain adaptation that leverages diffusion models to transform images from non-trust sets of the target domain into new images assigned to different categories while preserving target domain characteristics. The approach incorporates three core techniques: Target-guided Initialization, Semantic Feature Injection, and Domain-specific Feature Preservation. The method demonstrates strong performance across multiple benchmarks, outperforming state-of-the-art SFDA approaches in image classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- **Writing Quality:** The paper is exceptionally well-written with clear, easily followable sections and well-motivated technical components.\n- **Comprehensive Evaluation:** The experimental comparison against existing methods is extensive and thorough, providing strong empirical validation.\n- **Technical Innovation:** The proposed techniques are ingenious in their dual approach of ensuring accurate generation of newly assigned classes while preserving essential domain-specific information within the images. The progressive refinement strategy represents a particularly compelling approach to the generation process.\n\n**Weaknesses:**\n\n- **Missing Component Analysis:** The method lacks quantitative ablation studies showing the individual impact of each technique. The paper should evaluate DPTM performance without Domain-specific Feature Preservation, Target-guided Initialization, and Semantic Feature Injection separately to understand each component's contribution.\n- **Diversity vs. Target Alignment Trade-off:** While generating target domain-aligned images is beneficial, the paper doesn't address whether increased image diversity might provide better generalization to the target domain. The authors need to discuss this trade-off and demonstrate experimentally that naive targeted data generation performs worse than their proposed approach.\n- **Incomplete Baseline Comparisons:** The paper omits comparison with existing generation methods that explicitly use diffusion models for pseudo-target domain population, such as DATUM [1]. Including these comparisons would strengthen the proposed technique and better justify the multiple components employed.\n- **Limited Qualitative Analysis:** The authors provide insufficient qualitative results to fully demonstrate the effectiveness and visual quality of their generation techniques.\n\n[1] Benigmim, Yasser, et al. \"One-shot unsupervised domain adaptation with personalized diffusion models.\" CVPR 2023.\n\nQUESTIONS:\n1. **Can you provide comprehensive ablation studies showing DPTM's performance when each component (Domain-specific Feature Preservation, Target-guided Initialization, Semantic Feature Injection) is removed individually?**\n2. **How does your targeted generation approach compare against methods that prioritize image diversity for target domain generalization, and can you provide experimental evidence supporting your design choice?**\n3. **How does DPTM compare against other diffusion-based domain adaptation methods like DATUM for instance or other similar approaches, and what specific advantages do your multiple techniques provide over simpler generation approaches?**\n4. **Can you provide more extensive qualitative results demonstrating the visual quality and domain adaptation effectiveness of your generated images?**\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for Source-Free Domain Adaptation (SFDA) called Diffusion-Driven Progressive Target Manipulation (DPTM). The key idea is to leverage a pre-trained latent diffusion model (stable diffusion) to generate a pseudo-target domain from unlabeled target data. The target data are split into a “trust set” (high-confidence samples with reliable pseudo-labels) and a “non-trust set” (low-confidence samples) based on prediction entropy. The trust set samples are used directly with their pseudo-labels for supervised fine-tuning, while the non-trust samples are assigned new labels uniformly at random (to avoid class imbalance) and semantically manipulated into those categories using the diffusion model.\n\nThe manipulation through the diffusion model is carefully designed to preserve the target-domain characteristics of each image while changing its semantic class. DPTM employs a three-key strategy: \n1.\tTarget-guided initialization: initialize the diffusion sampling with a latent that retains the original image’s structural features,\n2.\tSemantic feature injection: iteratively guide the diffusion process toward the assigned label,\n3.\tDomain-specific feature preservation: inject features from the original image into the diffusion latent to maintain the target domain style.\n\nAfter this manipulation, the new generated images (with their assigned labels) together with the trust-set images form the “pseudo-target domain” used to train the target model. Importantly, DPTM is applied in progressive refinement iterations, i.e., after one round of adaptation, the target model is improved and can produce more confident pseudo-labels, so the trust set grows and the non-trust set shrinks. \n\nEmpirically, DPTM demonstrates state-of-the-art performances on 4 standard SFDA benchmarks of varying scales: Office-31, Office-Home, VisDA-2017, and DomainNet-126. It outperforms existing methods by substantial margins, especially on challenging transfers with large domain shifts.\n\nSTRENGTHS AND WEAKNESSES:\n**STRENGTHS**\n\n**Originality**: The paper offers a creative take on SFDA by utilizing diffusion models for style-aware augmentation. The two-stage framework and spatial calibration mechanism are technically sound and well-motivated.\n\n**Empirical performance**: In my opinion, this is the main strength of the paper. The proposed method produces strong results on 4 benchmark datasets with significant and consistent gains over prior work in the setting of SFDA. Extensive ablations demonstrate the contribution of each component.\n\n**Clarity**: The high-level intuition behind each design component is clear, especially in describing the use of diffusion for feature stylization and progressive learning from clean to noisy pseudo-labels.\n\n**WEAKNESSES**\n\n**Practicality**: The method introduces multiple components (e.g., multiple loss terms, a style sampler trained using DDIM, a learned confidence regressor, a spatial calibration module), which may be cumbersome to train in practice and require careful tuning.\n\n**Minor redactional issues**: Some parts of the paper suffer from broken English and formatting issues, e.g., \n- Abstract: “.. a challenging tasks …”\n- Page 6 (experiments): “Comperative Methods”\n- Appendix C: “Thess results demonstrate …”, “demnostrated”\n\nQUESTIONS:\nHow exactly is the class condition provided to Stable Diffusion? The paper denotes conditioning on y, but since Stable Diffusion is typically text-conditioned, did the authors provide the class name as a text prompt? Or did they finetune an embedding for each class?\n\nHow is the training runtime of DPTM for each progressive refinement iteration or until producing good performances?\n\nI’m still not fully convinced that DPTM benefits from the random assignment of the labels for non-trust samples as a starting point, instead of the pseudo-labeling mechanism like on the trust samples. Starting with completely random labels for the non-trust set might hinder reproducibility and stability. Could the authors clarify on this?\n\nLIMITATIONS:\nYes, the authors mention that the proposed method is inherently constrained by the generation capability of the diffusion models. \n\nHowever, some other potential practical burden of implementing such a multi-stage system is under-addressed. \n\nIt would be great to include a discussion on deployment challenges (e.g., memory/GPU needs, sensitivity to hyperparameters) and consider if the use of synthetic target features could inadvertently cause performance drift in critical systems.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}