Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jinxu Lin
authors: Linwei Tao
authors: Minjing Dong
authors: Chang Xu
reviews: 
Comments: The paper introduces a new data attribution method called "Diffusion Attribution Score (DAS)" to quantify the influence of training samples in diffusion models, specifically addressing the challenges of data attribution in generative tasks. Experimental results demonstrate that DAS achieves state-of-the-art performance in LDS across various datasets and diffusion models, with strategies explored to accelerate DAS computation for large-scale models.
Strengths: 1. Compared to previous work, the proposed DAS method shows significant improvements in experimental results.
2. The article provides anonymized source code, demonstrating the reproducibility of the proposed approach.
3. The data attribution problem in diffusion models studied in this paper is meaningful.
Weaknesses: 1. The author mentions "these approaches measure the divergence between predicted and ground truth distributions, which leads to an indirect comparison between the predicted distributions and cannot represent the variances between model behaviors." and provides a brief explanation in Section 3.1. However, as an important argument of the paper, the vague explanation given in lines 178–180 is insufficient to convincingly illustrate the shortcomings of previous work.
2. Similar to W1, lines 191–194 lack adequate support, whether theoretical justification, appropriate citations, or experimental evidence.
3. The logical connection from Sections 3.2 to 3.3 is unclear. The motivation for each sub-module, the problems they aim to solve, and why the proposed methods were chosen are not explained. Additionally, the theoretical proof follows similar derivations as [1] without clearly explaining the motivation behind each step, making it difficult to follow the authors' reasoning.



[1] Park, Sung Min, et al. "TRAK: Attributing Model Behavior at Scale." *International Conference on Machine Learning*. PMLR, 2023.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces a novel metric called Diffusion Attribution Score (DAS) to more accurately quantify the attribution of training samples to generated images. While the paper is tackling an important problem of accurate data attribution for diffusion models, it has areas for improvement, such as justification of their proposed and more rigorous evaluation of the method as described below.
Strengths: - The paper addresses the important problem of attributing diffusion models’ output to training samples.
- The metrics in quantitative evaluation look good, all showing significant gaps accounting for the confidence interval or standard deviation (though further clarity on these intervals would be helpful as they are not noted).
Weaknesses: - The equations in Section 3.1 only show that DAS is independent of data distribution, unlike D-TRAK. However, it has not been shown how significant it is. While DAS seems to be working well in the experiment results, the theory behind it appears weak.
- The evaluation solely relies on the LDS metric, which is based on the strong assumption of additive data contributions. There have been discussions on the effectiveness of LDS for evaluating data attribution algorithms, such as https://arxiv.org/abs/2409.18153 Adding other metrics or justifying its use looks necessary.
- In Figure 1, diffusion model outputs from the same random seeds are presented. The rationale behind the comparison seems to be that the same random seed should lead to a similar output when models are trained on datasets of similar distribution. However, this may not be true since models are retrained, and they are essentially different ones. How can you ensure that two diffusion models trained on the same dataset generate the same output for a random seed when other randomness, such as batch sampling, is not controlled?
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: (1) This paper examines the choice of the model output function in D-TRAK, demonstrates the theoretical shortcomings of D-TRAK's choice, and proposes using the predicted noise of a diffusion model as the output function (i.e., the output of the noise predictor). (2) With the proposed model output function, the leave-one-out attribution is approximated with Taylor expansion and Newton's method. (3) Additional tricks such as moving the expectation w.r.t the timestep inside the norm and using CLIP score to select candidate important data are proposed. Finally, the proposed method (i.e., DAS) achieves better linear datamodeling scores than existing attribution methods.
Strengths: - This paper addresses the gap left by Zheng et al. (2024) and demonstrates why the simple diffusion loss is not an appropriate model output function for TRAK-based methods (Equation 8).
- The proposed, theoretically more principled method DAS actually leads to improved linear datamodeling scores.
- This paper proposes to reduce the number of candidate training samples for computing data attribution scores, using CLIP score as the criterion. This approach can significantly improve computational efficiency and is shown to work fine (Table 7).

References
- Zheng et al. (2024) - Intriguing properties of data attribution on diffusion models.
Weaknesses: - Although the paper shows that $L_{\text{simple}}$ is not an appropriate model output function, it is not directly shown why $L_{\text{Square}}$, as proposed in Zheng et al. (2024), is more appropriate. Specifically, Equations (9) and (10) are not exactly equivalent. In this sense, this paper does not fully explain the intriguing findings in Zheng et al. (2024). A more direct theoretical comparison between Equations (9) and (10) would make the contribution greater.

- In lines 153-155 and Equation (7), the paper states that D-TRAK uses the simple loss $L_{\text{simple}}$ as the output function. However, the main experimental results from Zheng et al. (2024) come from D-TRAK implemented with $L_{\text{Square}}$ as the output. This mismatch in writing needs to be resolved.

- Quantitative results are missing in the experiment where the top 1,000 positive influential samples are removed. Something like the counterfactual evaluation in Zheng et al. (2024) would suffice.

References
- Zheng et al. (2024) - Intriguing properties of data attribution on diffusion models.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2