Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yuwei Niu
authors: Shuo He
authors: Qi Wei
authors: Zongyu Wu
authors: Lei Feng
reviews: 
Comments: The paper presents BDetCLIP, a novel test-time backdoor detection method for multimodal contrastive learning models like CLIP. The approach leverages contrastive prompting to differentiate between clean and backdoored images by analyzing the distribution differences in cosine similarity with benign and malignant class description texts. Extensive experiments demonstrate the method's superior effectiveness and efficiency compared to state-of-the-art detection methods.
Strengths: The authors claim that this is the first work on backdoor detection during the inference phase.
The paper introduces a novel approach by using contrastive prompting, which is innovative and leverages the strengths of large language models.
This work offers a computationally efficient solution for real-world applications, surpassing previous methods by large margins.
Weaknesses: The experiments focus on specific attack types and large multimodal models. Including a broader range of attacks and models could enhance the robustness of the evaluation.
Some technical aspects, such as the choice of thresholds and specific parameter settings for both CLIP and attack methods, could be explained in greater detail. 
While the method is computationally efficient, its applicability in highly dynamic environments or with evolving backdoor techniques might require further exploration. Discussing potential challenges and solutions for such scenarios could improve the work's practical relevance.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper introduces a new method for detecting backdoors in CLIP models at test time. The main idea is to use how backdoored samples remain insensitive to changes in text descriptions. The method generates two types of prompts with LLMs and measures the difference in cosine similarities between images and these prompts to detect backdoors. It's efficient because it doesn't require parameter updates.
Strengths: - The paper highlights a key gap in current backdoor defenses for multimodal models:  
  - While previous methods focus on training or fine-tuning, which require high computational costs, test-time detection hasn't been explored yet. This leads to a new research direction that's both interesting and practical.

- The contrastive prompting mechanism is effective because:
  - It uses the multimodal features of CLIP, addressing a limitation of previous unimodal detection methods that ignore the text modality.
  - The use of benign and malicious prompts provides a natural way to measure semantic sensitivity without modifying the model.
Weaknesses: - The method relies on the assumption that backdoor samples are insensitive to changes in text descriptions while clean samples are sensitive. However:
  - Limited Analysis of Multi-target Attacks: The analysis only covers single-target backdoor attacks. If backdoor triggers cause different outputs based on context (e.g., indoor vs outdoor scenes), the assumption of semantic insensitivity may not hold, exposing a gap between simplified models and real-world cases.
  - Simplified Trigger Representation: Current assumption: $x^*=(1-M) \odot x+M \odot \Delta$, where $\Delta$ is a simple pixel-pattern trigger. Modern backdoor attacks may use semantic-level triggers that naturally align with textual descriptions, yet the paper does not analyze how this alignment impacts the distribution difference  $\Omega(x)$.

- Prompt Quality Variation: There is no mechanism to ensure prompt quality is consistent across different categories or evaluations, and no metrics to assess prompt quality. The effect of prompt variation on detection reliability is also not examined.
- Gaps in Theory: There’s no clear definition of what makes a “good” prompt, nor an analysis linking prompt properties to detection performance.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposed BDetCLIP to defend against backdoor attacks on CLIP. Different from cuttent methods, this is the first attempt at a computationally efficient backdoor detection method in the inference stage. The experiments show that BDetCLIP performs well in terms of both effectiveness and efficiency.
Strengths: -	This paper proposed a inference-stage method, which does not access to the pre-traing or finetuning, and also the training data.
-	The method is well-motivated, i.e., the visual representations of backdoored images are insensitive to both benign and malignant changes in class description texts. 
-	Overall, the ablation studies are well-organized, illustrating the effectiveness of the proposed method.
Weaknesses: -	This paper only focused on the classification task, assuming that there a closed category space. However, VLMs are good at open-set classification tasks.
-	The authors introduce a method that helps to choose the threshold $\epsilon$ in Appendix B. However, the selection method requires a small set of clean validation data. It would be better to include the sensitivity analysis of the threshold $\epsilon$ to show the robustness of $\epsilon$.
-	The investigation of adaptive attack is missing. Since the proposed defense is based on the property of backdoor samples, it is necessary to go deep into adaptive attacks.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3