Kati8/ICLR_2023_2025 huggingface dataset:
authors: Neel Alex
authors: Shoaib Ahmed Siddiqui
authors: Amartya Sanyal
authors: David Krueger
reviews: 
Comments: This paper presents BaDLoss, a new backdoor detection method that exploits the difference training dynamics between clean and backdoor samples by injecting specially chosen probes into the training data. These probes model anomalous training dynamics, and BaDLoss tracks the loss trajectory for each example in the dataset to identify unknown backdoors. By removing identified backdoor samples and retraining, BaDLoss can mitigate the backdoor attacks.
Strengths: 1. The proposed method works based on observing the significantly vary training dynamics between clean and backdoor samples, which is quite novel and interesting.
2. Overall, the method's performance seems to outperform other baselines.
Weaknesses: 1. [method's presentation, major] I personally find the presentation of Section 3 quite hard to follow since there are no algorithm or figure to describe the method, or even a formulation.
2. [lack of experiments, major] There are only 3 types of backdoor attack (patch-based, blending-based, warping-based) that are considered in the experiments, so I am not sure if the defense is effective with all attacks. I think there should be more backdoor attack approaches included in the experiments as well as related works discussion, such as sample-specific ([1]), optimized trigger ([2]), or frequency domain attack ([3]). Moreover, there is no abaltion study/discussion about different choices for the hyperparameters used in the paper (detection threshold, k for the kNN classifier). 
3. [underwhelming experimental results, major] The clean acc. of BaDLoss is significantly degraded in the cases of SIG, WaNet, and multi-attack on CIFAR10. With those underwhelming clean acc. (~60%), I doubt that the model can be considered functional, especially on such "easy" dataset like CIFAR10.
4. [results' presentation, minor] There are many numbers in Table 2 and Table 3 but the best results are not highlighted. The authors should highlight the best results, or maybe report the average clean acc. and ASR drops of each defense method.

[1] Li, Yuezun, et al. "Invisible backdoor attack with sample-specific triggers." (ICCV 2021)  
[2] Zeng, Yi, et al. "Narcissus: A practical clean-label backdoor attack with limited information." (ACM CCS 2023)  
[3] Wang, Tong, et al. "An invisible black-box backdoor attack through frequency domain." (ECCV 2022)
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper proposes a backdoor detection method named BadLoss. It focuses on the threat model that modifies the training dataset and detect them via loss dynamics. Specifically, it needs a probe set which has potential trigger patterns and use them to detect poisoned samples. After deleting the backdoored samples, it uses the clean set to retrain the model. The experiments validate the effectiveness of their method.
Strengths: 1.	Addressing the datasets backdoor attack is still an interesting and realistic direction.
2.	Detecting multi-trigger backdoor attacks is an efficient way to deal with large scale dataset.
Weaknesses: 1.	Results have only marginal improvement. For example, the badloss cannot maintain a consistent high clean accuracy for the sinusoid attack.
2.	Ablation study is needed. For example, how do you choose the threshold and why you choose that. Would the probes set affect the performance significantly? How do you choose the k in kNN classifier.
3.	Using loss is not a novel idea to detect backdoors, and there are many similar works.
Li, Yige, et al. "Anti-backdoor learning: Training clean models on poisoned data." Advances in Neural Information Processing Systems 34 (2021): 14900-14912.
Guan, Jiyang, et al. "Few-shot backdoor defense using shapley estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair