VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models
Authors: Zihao Zhu
Mingda Zhang
Shaokui Wei
Bingzhe Wu
Baoyuan Wu
Venue: ICLR 2024 poster (2024)
Keywords: Noisy label detection
Poisoned sample detection
data-centric AI
multimodal large language model
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We propose a universal framework that capable of detecting poisoned samples, noisy labels and hybrids of them simultaneously.

Abstract: The role of data in building AI systems has recently been emphasized by the emerging concept of data-centric AI. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable.
Hence, it is critical to detect dirty samples to improve the quality and realiability of dataset. 
Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other fields.
In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. 
To capture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.
It consists of three consecutive modules: the visual question generation module to generate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the inconsistency.
Extensive experiments demonstrate its superior performance and generalization to various categories and types of dirty samples.
The code is available at [https://github.com/zihao-ai/vdc](https://github.com/zihao-ai/vdc).

Published as a conference paper at ICLR 2024
VDC: VERSATILE DATA CLEANSER BASED ON VISUAL-LINGUISTIC INCONSISTENCY BY MULTI- MODAL LARGE LANGUAGE MODELS
Zihao Zhu', Mingda Zhang', Shaokui Wei', Bingzhe Wu², Baoyuan Wu School of Data Science, The Chinese University of Hong Kong, Shenzhen Tencent Al Lab {zihaozhu, mingdazhang, chaokuiwei]er @link.. cuhk. edu. cn bi bingghewu@tennent.ccom com
wubaoyuan@cuhk, edu. on
ABSTRACT
The role of data building Al systems has recently been emphasized by the emerging concept of data-centric AL. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable. Hence, detect dirty samples to improve the quality and realiability of dataset. Ex- isting detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other fields. In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. To cap- ture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large lan- guage models (MLLM) in cross-modal alignment and reasoning. It consists three consecutive modules: the visual question generation module to gener- ate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the incon- sistency. Extensive experiments demonstrate its superior performance and gener- alization various categories and types of dirty samples. The code is available https: httptttttthtugtg//g////ttgtg.h com/ zihao-ihh---ee ai si/vdc.
critical
INTRODUCTION
The emerging concept of data-centric AI (DCAI) highlights the pivotal role of data in constructing advanced Al systems (Zha et al., 2023). The quality and reliability of data crucial factors influ- encing model performance. Nevertheless, in the real world, dataset can be susceptible to undesirable flaws (Whang al., 2023).
For instance, dirty samples may be introduced into the datasets intentionally or unintentionally. In this paper, we comprehensively examine three categories of dirty samples as follows:
Category I: Poisoned Samples. In the context of backdoor attack, malicious attackers intentionally manipulate partical clean samples by embedding triggers and changing the ground-truth labels to target labels, thereby generating poisoned samples. Deep neural networks (DNNs) trained on the dataset with such poisoned samples will be injected with backdoor, i.e., predict any poisoned sample as the target label during the inference stage, while maintain accuracy the clean samples. Category II: Noisy Labels. In scenarios crowdsoureing or web crawling, human annotators or automatic annotation robots may make mistakes accidentally, resulting in the presence of dirty sam- ples with corrupted labels. Training DNNs using the dataset with such noisy labels will significantly degrade the overall performance.
Category III Hybrid Dirty Samples. An even more critical concern arises when the attackers poison datasets that initially contain noisy labels. In this case, the datasets comprise both poisoned
"Corresoonding Author

---

Published as a conference paper at ICLR 2024
samples and noisy labels. Models trained such datasets will encounter both malicious backdoor attack and performance degradation simultaneously.
The presence of above dirty samples makes the DNNs vulnerable and unreliable. To enhance the robustness and performance of DNNs, the detection of dirty samples is crucial in the lifecycle of DCAL Recent research have been proposed on the noisy label detection (Northcutt al., 2021; Zhu et al., 2022; Yu et al., 2023) or poisoned sample detection (Hayase et al., 2021; Tang et al., 2021; Qi al., 2023) respectively. However, they frequently exhibit limitations in terms of generalization: 1). Inconsistent generalization across different categories of dirty samples. We empirically find that detectors designed for detecting poisoned samples are ineffective when applied datasets with noisy labels. and vice versa. Moreover, both types of detectors prove inadequate for hybrid dirty samples. (See Table in Sec 5.2.3). 2). Inconsistent generalization across different types of dirty samples in the same category. For noisy label detection, research has shown that symmetric noisy labels are more readily detectable than asymmetric ones (Cheng et al., 2021). Likewise, for poisoned sample detection, sensitivity to various triggers has been demonstrated in Wu et al. (2022). Therefore, developing a universal framework capable of detecting multiple types of dirty samples concurrently, including noisy labels and poisoned samples, is urgent challenge for DCAL
We find a notable commonality of noisy labels and poisoned samples lies in visual-linguistic in- consistency between visual contents and associated labels, i.e., the semantics of visual modality and that of language modality of label do not match, even when the poisoned samples are embedded with triggers. Given the exceptional capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning, we resort to MLLM to measure this semantic inconsistency between modalities. To this end, we propose a universal detection framework called Versatile Data Cleanser (VDC). It consists of three consecutive modules: the visual question generation (VQG) module generate insightful visual questions about the image based on the associated label; the visual question answering (VQA) module to obtain the semantic information of the image by an- swering the generated questions with MILLM; followed by the visual answer evaluation (VAE) module to measure the inconsistency by evaluating the matching score between the semantics of the image and labels. Since VDC does not involve the training process with specific dirty samples, is endowed with the universal capacity to detect various categories and types of dirty samples.
We summarize our main contributions: 1). We identify the commonality of various dirty samples inconsistency between visual contents and associated labels. 2). To quantify this inconsistency, we propose a versatile data cleanser that leverages the impressive capabilities of multimodal large language models. 3). Experiments show that VDC consistently exhibits superior performance for detecting poisoned samples, noisy labels, and hybrids of them.
RELATED WORKS
Poisoned Sample Detection. The rise of backdoor attacks in machine learning has posed signifi- cant security threat, including embedding malicious triggers into clean training samples (Wu et al., 2023). Several recent studies have explored detecting and mitigating the presence of poisoned sam- ples in datasets. Chen et al. (2018) proposes to use K-means to separate the clean and poison clusters in the latent space. Tran et al. (2018) and Hayase al. (2021) utilize robust statisties detects poi- soned samples based on spectral signature. Gao et al. (2019) observes the randomness of predicted classes for perturbed inputs. Zeng et al. (2021) proposes to detect artifacts of poison samples in the frequency domain. Chen et al. (2022) focuses on sensitivity metrics for distinguishing poisoned samples from clean ones. Qi et al. (2023) proposes confusion training to decouple benign corre- lations while exposing backdoor patterns to detection. Most of these approaches require training on the poisoned dataset or external clean subset, which depends on the types of poisoned samples, while our proposed method is more robust and generalizable to various types of poisoned samples.
Noisy Label Detection. Human-annotated labels are often prone to noise, and the presence of such noisy labels will degrade the performance of the DNNs. Several approaches have been pro- posed to detect noisy labels (Ghosh et al., 2017; Bahri et al., 2020; Berthon et al., 2021). Northcutt et al. (2021) proposes to exploit confident learning to estimate the uncertainty of dataset labels. CORES (Cheng et al., 2021) progressively sieves out corrupted examples via proposed confidence regularizer. Zhu et al. (2022) proposes data-centric solution based on neighborhood information
2

---

Published as a conference paper at ICLR 2024
Input
Visual Question Answering
Visual Answer Evailuation
Paisoned Image:
Targei Label:
airplane
Describe the Image ofly.
Please detem ine if the
Cuupptty [Epected answed
(allowigg ia Caption and Label rofor to the me cojoct
Visual Question Generation
Caption, [Outputa]
Multimodal Large Language Model
[Expected arswer)
quections muttimeds large
String Matching
language to dendty object smage Alplane
Outpptty)
Large Model
the maga reature yellow ear parted on pavement, th its hoad be
Manual Design
Large Language Model
No Response and completely
General Questions
Label specifii Questions
Junc imaga brkifi::
cdjec the capable carving cargo
cifferent types of yansportziion
False
b The slject the mage desigried for the
Dweritte the image in desst
the take tu artaayal
False
Fallee
Fallow
9,
How would you sur mario the car tant the imags an
Fabse
True
Oy the obaet the image disigned flying the ar?
Multimodal Large Language Model
fow words?
prion
Dost the objectt the imoge fotte wiips lor gersnwingg kl??
Vote site-base Ensemble
the glown image
Outport::
No, The object is car.
(Expected answer answer: akillaoe)
(Expected Answorr you)
Poisoned Sample
The framework of Versatile Data Cleanser. Given the image and label, the visual question generation module first generates general and label-specific questions respectively. Then the visual question answering module answers the generated questions based on the image. Last, the visual question evaluation module evaluates the correctness of answers and makes the final judge based on the vote-based ensemble.
Figure I:
to detect noisy labels. BHN (Yu et al., 2023) leverages clean data by framing the problem of noisy label detection with clean data as multiple hypothesis testing problem.
Existing poisoned sample detection and noisy label detection methods are limited to performing well in their respective domain. Instead, our paper proposes a universal detection framework capable of detecting various types of dirty samples simultaneously.
PRELIMINARIES: DIRTY SAMPLE DETECTION
In this section, we first define the setup of dirty sample detection task, including poisoned samples and noisy labels, and then clarify the goals of this paper.
Setup. We consider a standard classification problem given the dataset D = {(x;; 1 that contains N samples i.dd sampled from X' Y, where æ; X' denotes the input feature, y/ {l....,kt K} is the label of x, The classification task aims to learn a classifier fo : X + y. In the realdworld, however, when collecting dataset, some samples may be corrupted due human mistakes or malicious goals, thereby generating dirty samples with corrupted labels in the dataset. Therefore, in the real-world, D is the fusion of dirty dataset D = {@,,, Tt) } and clean dataset D = ((.,, 9:) N. i.e., D' = δ U D, where (æ;, 7i) is dirty sample and M is the number of dirty samples, (Z,. 0c) is a clean sample. We formulate two types of dirty sample in the following:
Poisoned Sample: Poisoned sample denotes the that its visual feature is maliciously ma- nipulated by the attacker, i.e., ã, := g(wx) * it, where g(-) is the generation function, such as blending (Chen et al., 2017) and wrapping- wrapping-based transformation (Nguyen & Tran, 2021). Mean- while, the label is changed to the target label by the attacker, i.e., Ds yt + 9i-
Noisy Label: Noisy label represents the sample that its label is annotated incorrectly, while its
visual feature remains unchanged, i.e., Z, 2;, H. E y # ÿ., where ) represents noisy version of y. Following Yu et al. (2023); Zhu et al. (2022), we focus on the closed-set label noise that y and Y are assumed to be in the same label space. This situation is common when human annotators are asked to select the most appropriate label from a preset label set.
Goal. Unlike most existing works that can only detect noisy labels or poisoned samples, our goal is to design universal detection framework that can be applied to various categories of dirty samples.
3

---

Published as a conference paper at ICLR 2024
4 METHODOLOGY: VERSATILE DATA CLEANSER
We find that what poisoned samples and noisy labels have in common is that the visual features of the poisoned samples are inconsistent with their given labels. For example, an image containing 'car" is wrongly labeled as "dog', which can be detected by comparing the semantics of the visual content of the image and that of the given label. For the poisoned sample, although the trigger is embedded into the image, its underlying semantics has not been modified. We refer this commonality as "visual-linguise" inconsistency". Thanks to the surpassing abilities of multimodal understanding and reasoning of MLLM, propose Versatile Data Cleanser, called VDC. to capture the visual- linguistic inconsistency based on MILM. To the best of our knowledge, VDC is the first universal framework that is capable of detecting both noisy labels and poisoned samples simultaneously. As shown in Figure 1, it consists of the following consecutive modules:
Visual Question Generation (VQG): VQG module first generates insightful visual questions related to the given labels based on the template and LLM, which is detailed in Sec 4.1.
Visual Question Answering (VQA): Then VQA module resorts to MLLM to answer the gen- erated visual questions about the image to acquire the semantics of the visual content, which is detailed in Sec 4.2.
Visual Answer Evaluation (VAE): The VAE module assesses visual-linguistc inconsistency by evaluating the matching score between the semantics of the image and label, detailed in Sec 4.3.
4.1 VISUAL. QUESTION GENERATION
We propose to obtain semantic information of the visual content by asking MILMM visual questions. Therefore, the first step is how design insightful questions based the given label yi, which is formulated as follows:
4 =
== Fogg(Us
(I)
where might be corrupted label ÿ. or ground-truth label Di. Q denotes the j-th question and A denotes expected answer, and N₂ denotes the number of questions. In order to comprehensively and fully understand the semantics of images, two different types of questions are considered in VDC, including coarse-graines general questions and fine-grained label-specific questions.
General Questions. General questions can serve as means to acquire holistic semantic under- standing of an image from a global perspective, such as "Please describe the image briefly.". The expected answers to these general questions align with the given label. Since the general ques- tions remain consistent across various labels, they are generated by random selection from a set of predefined templates, as outlined in Table 10 in Appendix E.
Label-specific Questions. Besides, the label-specific questions related to the given labels aim to extract more localized semantics from the image, encompassing aspects of common sense features, attributions, functions, geography, history, culture, and etc. For example, given the label "airplane", an apt question is "Is the object in the image designed for flying in the air?". Designing most label- specific questions necessitates a level of expertise about the label that may exceed the capacity of a human annotator. When dealing with multitude of labels, such ImageNet with 1,000 classes, manually designing for each label becomes impractical. Hence, we utilize LLM like ChatGPT (Ope- nAl) automatically generate these questions, depending on its expansive open-world knowledge. The well-designed prompts and generated questions are detailed in Appendix D and E.
4.2 VISUAL QUESTION ANSWERING
The next step involves responding to the generated questions in Sec 4.1 based on the input image I; acquire the semantics of the visual content. This process is often referred to as the visual question answering (VQA) task, which can be formulated as follows:
R Fogua((44 (2/ xx:)
where Rd indicates the response of VQA model for the question Q/ Answering these questions necessitates the capabilities of natural language generation and external knowledge beyond the vis- ible content of image. Therefore, we resort to MILM as VQA model owing to its remarkable
4

---

Published as a conference paper at ICLR 2024
capabilities of visual and language understanding and reasoning, which has been demonstrated in wide range of risuua-language tasks.
4.3 VISUAL ANSWER EVALUATION
Afterward, for a suspicious input sample (æ W:), we obtain a of questions, expected answers, and responses, Le., {Q', A R 1 The subsequent step is to assess visual-linguisti consistency by evaluating the matching score between the semantics of the image and label. We first judge the correctness of the response of MLLM, i.e., whether it aligns with the expected answer, which can be formulated follows:
of Prac
(3)
where e; denotes the correctness, i.e., true or false. For label-specific questions with deterministic expected answers, we use string matching to evaluate the response. If the word "yes" is present in the response, the result should be true, otherwise if the response contains the word "no", the result should be false. Nevertheless, for general questions, string matching is insufficient to determine correctness. In such cases, we employ ChatGPT as specialized evaluator through meticulously designed prompts, which is a commonly adopted approach in the evaluation of LLM Chang et al. (2023).
Vote-based Ensemble. Then the matching score 5; of sample (æ,, y:) is computed as the proportion of questions answered correctly, which are formulated as follows:
I(e, = true)
N₂
where 1-) denotes identity function. If the score is less than the threshold sample (æ), v1) is detected as a dirty sample and then removed from the dataset.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets. We evaluate ASRon three benchmark datasets, CIFAR-IO (Krizhevsky et al., 2009) and two ImageNet (Russakovsky al., 2015) subsets: (1) For ImageNet-I00. we randomly choose 100 classes from ImageNet, in which 500 images per class for training and 100 images per class for testing. (2) For ImageNet-Dogg evaluate the effect of similarity of classes, we randomly choose 10 classes of dogs from ImageNet, in which 800 images per class for training and 200 images per class for testing.
Dirty Samples Generation. Denote the ratio of dirty samples in the whole dataset by 1). Two types of dirty samples are considered in the evaluation, which are illstrated as follows:
Poisoned Samples. We consider six representative backdoor attacks to generate poisoned sam- ples: (1) Visible triggers: BadNets (Gu et al., 2019), Blended (Chen et al., 2017). TrojanNN (Liu et al., 2018). (2) Invisible triggers: SIG (Barni et al., 2019), SSBA Li et al. (2021), WaNet Nguyen & Tran (2021). For all attacks, we randomly choose the same number of images from all classes except target class to add trigger, and then change the labels as target label. The example and settings of each attack are detailed in Appendix C.2.
Noisy Labels. We experiment with two popular synthetic noisy model models: the symmetric and asymmetric noise: (1) Symmetric noisy label is generated by uniform flipping, i.e., randomly flipping a ground-truth label to all other possible classes (Kim et al., 2019). (2) Asymmetric noisy label is generated by flipping the ground-truth label the next class, iee, (i mod K) + where K denotes the number of classes.
Evaluation Metries. We report the detection results with two key metrics: true positive rate (TPR) and false positive rate (FPR) following Qi et al. (2023). TPR means the recall of detected dirty sam- ples, representing the capacity to successfully detect dirty samples within the dataset. FPR denotes the ratio of clean samples erroneously identified as dirty samples, highlighting the susceptibility to produce false alarms. An ideal detection method should exhibit a higher TPR and lower FPR. Let va 1 indicate that the i-th sample is detected as dirty sample. Moreover, when retraining on
5

---

Published as a conference paper at ICLR 2024
Table I: Comparison of TPR (%) and FPR (%) for poisoned sample detection CIFAR-I0. 1j 0.09. i.c.
500 poisoned samples per class, Average is the mean of results different triggers. Top 2 are bold.
Dutaset: CIFAR-10
1
0.09
(500 polsoned samples per class)
Method
Clean Data
BodNets TPR1
Blended
SIG
TrojanNN TPRT
SSBA
WaNet
Averae
FPR]
TPR[
FPR.
TPR
FPR.
FPR.
IPR1
FPR
TPR
FPR.
TPRC
FPR.
STRIP
4%
94.22
10.99
52.82
11.12
100.00
10.98
99.73
10.05
81.87
9.33
3.82
10:45
74
10.49
S8
4%
61.62
48.85
61.40
48.87
60.89
48.92
59.53
49.06
58.02
49.21
57.22
49.29
59.78
49.03
SCAn
96.49
2.82
93.49
2.80
99.47
2.85
92.49
2.83
2.99
Frequency
4%
88.
18:71
82.80
18.70
48.07
20.79
100.00
11:40
85.84
19:81
40.02
20.61
74.29
18.34
CT
4%
97.24
0.18
97.78
1,02
99.16
0.74
100,00
0.13
98.31
0.10
95.16
0.70
97.94
0.48
D-BR
0%
87.13
3.36
23.93
7.60
94.40
2.56
90.85
10.28
10.07
8.93
10.18
8.87
51.09
6.93
SPECTRE
0%
94,00
20.62
95 31
20.49
8.
29.1 11
80.07
22 00
97.
20.28
88. 24
21 19
77 20
22.29
VDC (Ours)
99.93
2.75
2.75
2.75
2.75
Table 2: Comparison of TPR (%) and FPR (%) for poisoned sample detection on ImageNet-100. 17 = 0.099,
Lee, 50 poisoned samples per class, Average is the mean of results of different triggers. Top are bold.
Dataset: Imageeet-1100
!
0.099
poisoned samples per class)
Method
Clean
BadNets TPRt
Blended
SIG
TiojanNN
SSBA
WaNet FPR.
Average
FPR
TPRT
FPR_
TPR
TPRC
EPR.
TPR1
FPR.
TPR1
TPR:
FPR.
STRIP
4%
92.20
12.56
99.19
11.79
100.00
10:95
100.00
13.14
99.74
11.26
3.11
11.32
82.37
11.84
SS
4%
48.12
50.21
44.95
50.55
44.95
50,55
44.95
50.55
44.95
50.55
45.13
50,53
45.51
50.49
SCAn
97.01
1.66
97.58
2.46
99.21
1.21
99.92
1.77
87.39
1.36
58.97
2.54
90.01
1.83
Frequency
4%
1.52
1.59
1.31
1.59
1.72
1.59
95.05
1.59
3.41
1.59
0.04
1.59
17.18
1.59
CT
94.16
0.37
0.37
0.58
0.39
0.00
80.67
D-BR
0%
86.43
23.56
9.56
10.03
76.09
15:07
16.53
9.06
11.09
10:15
10.08
9.87
34.96
12.96
SPECTRE
48.57
50.
95
50.55
44.95
50.55
44.97
50.55
44.95
50.55
45.09
50. 54
45.58
50.48
VDC (Ours)
0%
99,92
1,55
99.94
1.55
99.90
1.55
99.96
1.55
99.98
1.55
99.94
1.55
99.94
1.55
the purified dataset, we report the attack success rate (ASR) and the clean accuracy (ACC) of the retrained model.
Implemented Details. We adopt ChatGPT based on GPT-3.5-turbo (OpenAl) LLM and Instruct- BLIP (Dai et al., 2023) as MLLM in VDC. For all datasets, we generate two general questions. The number of label-specific questions is six for mageNNet-100 and four for CIFAR- 10 and ImageNet- Dog. The threshold a is set as 0.5 across all experiments. The noisy ratio 1j for noisy labels is set as 0.4. We poison 50 and 500 samples per class for CIFAR-10. 5 and 50 per class for ImageNet- 100, and 80 per class for ImageNet-Dog. We retrain on the purified dataset with ResNet-18 (He et al., 2016). Additional details can be found in Appendix C.I.
Compared Baselines. For poisoned sample detection, we compare with 7 baselines, in which STRIP (Gao al., 2019), SS (Tran et al., 2018), SCAn (Tang et al., 2021), Frequency (Zeng al., 2021) and CT (Qi et al., 2023) require external clean subset to execute, while SPECTRE (Hayase al., 2021) and D-BR (Chen al., 2022) do not require any clean subset. For noisy label detec- tion, we compare with 5 baselines, including BHN (Yu et al., 2023), CL (Northcutt et al., 2021), CORES Cheng et al. (2021), SimiFeat- V and SimiFeat-R (Zhu al., 2022), in which BHN relies on a clean subset to perform. The detailed settings of each baseline can be found in Appendix C.3,C.4.
5.2
EXPERIMENTAL RESULTS
5.2.1 RESULTS ON DETECTING POISONED SAMPLES
In this section, we first conduct a comprehensive evaluation on the poisoned samples detection. The results on CIFAR-10, ImageNet-100 100 and ImageNet-Dog with different poisoning ratios are presented in Tables 1,2,13,14 (Refer Tables 13,14 Appendix F.2). For a fair comparison, all baselines requiring clean data utilize 4% clean subset. The results demonstrate the effectiveness of our proposed method from the following aspects:
Consistent Effectiveness Against Various Types of Poisoned Samples. From the results on CIFAR-10 10 in Table 1, we find that VDC consistently exhibits superior performance under various types of poisoned samples without relying on any clean subset, demonstrating the generalization of VDC. In contrast, other detectors are sensitive to different types of triggers. For example, VDC achieves average TPR of 99.91 % against all backdoor attacks, while SPECTER experiences a sig-

---

Published as a conference paper at ICLR 2024
Table 3: Comparison of TPR (%) and FPR (%) for poisoned sample detection ImageNet-Dogg 1 0.09, Le., 80 poisoned samples per class, Average the mean results of different triggers. Top are bold.
Dataset:
ImageNet-Dog
0.09
(80 poisoned samples per class)
Method
Clean Data
BadNetts
Blended TPR
SIG
TrojanNN
SSBA
WaNet
Average
TPR1
FPR.
FPR.
TPR
FPR_
TPR1
FPR]
TPR:
FPR.
TPR1
FPR.
TPR1
FPR.
Strip
4%
93.19
10.88
82.78
11.18
97.08
11.95
98.61
11.32
95.42
10.62
3,47
9.42
78.43
10.90
SS
4%
19,86
21.66
17.64
21.88
21.94
21.46
23.89
21.26
23.75
21.28
12.50
22.39
19.93
21.66
SCAn
4%
98.06
0.01
72.36
9.15
78.61
0.03
62.22
0.23
84.44
0.15
12.54
2.12
68.04
1,95
Frequency
4%
83,89
45.48
50.00
43.45
44.03
45.48
95.97
44.64
61.94
45.45
36.53
45.65
62.06
45.36
CT
92.50
84.31
0.58
15.41
0.99
98.06
0.44
88.89
0.29
0.00
0.92
63,20
0.70
D-BR
0%
8.61
8.85
9.31
8.85
10.83
9.22
9.72
9.01
8.75
9.15
8.47
9.12
9.28
9.03
SPECTRE
77.64
47.27
99.86
45.07
45.30
96.94
49.68
46.30
VDC (Ours)
0%
98.89
4.12
97.50
4.12
98.61
4.12
99.31
4.12
98.89
4.12
98.89
4.12
98.68
4.12
Table 4: Comparison of TPR (%) and FPR (%) for noisy label detection on the three datasets under different types of noisy labels, where noisy ratio = 0.4. Top 2 are bold.
CIFAR-10 0.4
ImageNet-100
0.4
ImageNet-Dog 0.4
Clean Data
Method
Symmetric TPR1
Asymmetric TPR
Symmetric TPR1
Asymmetric TPR
Symmetric TPR
Asymmetric
FPR.
FPR
FPR.
FPR.
FPR
TPR1
FPR..
BHN CORFS
20%
80.88
2.98
83.13
3.24
57.04
1.94
16.24
0.96
14.36
0.23
22.54
0.75
0%
92.11
4.85
5.36
4.47
77.22
0.05
0.07
84.44
23.87
24.47
CL
0%
85.05
8.75
82.49
4.50
67.32
19.07
43.62
17.82
90.78
71.37
61.97
46.86
Simileet-V
0%
98.80 88.80
4.13
59.67
7.43
98.31
5.52
55.67
17.65
89.59
11.73
51.85
22.10
SimiFeat-R R
0%
99.16
5.11
79.46
15.18
99.27
8.22
69.59
27.25
95.86
17.90
66.39
35.07
VDC (Ours)
0%
98.81
2.61
99.60
2.62
94.79
1.55
92.34
1.55
97.30
7.90
91.97
7.90
nificant fluctuation with a difference of 87.15% between its highest and lowest TPR. Additionally, VDC achieves competitive results terms of FPR, averaging only 2.75%, which indicates that VDC has a low propensity to incorrectly identify clean samples as dirty samples.
Consistent Effectiveness Across Datasets. Comparing the results of ImageNet- 100 in Table 2 and CIFAR-10 in Table 1, when facing larger dataset with more labels, VDC maintains performance with the average TPR still reaching 99.94%. On the contrary, other baselines are unstable on dif- ferent datasets, such as SPECTRE decreases from 77. 77.20% to 45.58%. To explore the effect of the similarity of classes, we evaluate on a fine- fine-grained dataset ImageNet-Dog From the results in Table 3 in Appendix F.1, VDC shows evident improvement compared other baselines.
Consistent Effectiveness Across Poisoning Ratios. We also evaluate with lower poisoning ratios on CIFAR-10 10 and ImageNet- 100 study the effect of poisoning ratios. Compare Table 1 with 0.09 and Table 13 with = 0.009 on CIFAR-10, we find that the performance of VDC has almost fluctuation, while other methods are greatly affected by the poisoning ratio. A similar phenomenon on ImageNet-100 can be found in Table and 14.
5.2.2 RESULTS ON DETECTING NOISY LABELS.
In this section, we evaluate VDC on noisy label detection, another common type of dirty samples. The results on CIFAR-10, 10, ImageNet-109 100 and ImageNet-Dog are shown in Table 4, verifying that VDC also performs well on detecting noisy labels from the following points:
Consistent Effectiveness Against Various Types of Noisy Labels. By comparing the performance on the symmetric and the asymmetric noisy labels, we note that asymmetric a more challenging setting. Even though some baselines behave well on detecting symmetric noisy labels, such as SimiFeat- and SimiFeat-R, they may reach low TPR on the symmetric noisy labels. However, VDC consistently works well on the asymmetrie noisy label. For example, VDC achieves 99.60% TPR on detecting asymmetric noisy labels on CIFAR-10, while SimiFeat-V only has 59.67% TPR.
Consistent Effectiveness Across Datasets. From the results the three datasets in Table 4, note that VDC performs consistently well on different datasets, while other methods perform worse on ImageNet- 100 and Imagenet-Dog, which indicates the robustness of our proposed method.

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of TPR (%) and FPR (%) for detecting the mixture of poisoned sampels and noisy labels on CIFAR- 10, where poisoning ratio yp and noisy ratio 7/2 are set as 0.1. Top are bold.
Dataset: CIFAR-10
poisoning ratio
0.09
noisy ratio 12
0.1
Method
Clean Data
BadNetts TPRt
Blended TPR
SIG
TrojanNN
SSBA
WaNet
Average
FPR.
FPR.
TPR
FPR_
TPR1
FPR.
TPR:
FPR.
TPR1
FPR.
TPR1
FPR.
STRIP
4%
50.59
12.41
27.40
11.74
50.83
12.09
50.31
12.72
43.54
9.39
4.98
11.64
37,94
11.67
SS
4%
51.63
49.61
52.77
49.34
53.80
49.10
50.91
49.78
51.45
49.65
50.57
49.86
51.86
49.56
SCAn
4%
45.38
0.00
45.64
3,80
50.71
1.77
47.46
0.00
44.67
0.01
43.89
4.07
46.29
1.61
Frequency
4%
52.22
18.65
49.21
18.66
34.08
20,71
53.18
11.44
51.44
19.73
30.14
20.53
45.05
18.29
CT
45,22
47.88
0.39
48.37
49.41
1.61
46.67
0.38
46.00
0.69
47.26
0.60
D-BR
0%
31.92
0
12.09
1,07
1.20
1.42
16.13
17.73
0.01
0.02
3.08
3.20
10.74
3.91
SPECTRE
22.07
22.49
22.29
26.09
22.41
22.32
21.67
22.49
23.96
25.05
23.44
BIIN
20%
68.40
1.27
69.11
1.34
70.35
1.35
72.61
1.12
67.81
1.26
69.43
1.34
69.63
1.28
0.51
36.33
CORES
0%
66.73
2.29
47.59
2.46
30.41
15.94
47.27
2.18
48.03
2.55
49.76
2.92
48,30
4.72
SimiFeat-V V
80.02
4.71
77.94
5.06
66.72
5.31
52.60
4.89
85.73
4.72
87
4.80
75.08
4.92
SimiFeat-R
81.36
4.57
79.12
5.48
66.23
6.17
52.42
4.93
80.85
5.24
89.19
4.93
74.86
5.22
VDC (Ours)
0%
99.42
2.79
99.40
2.79
99.39
2.79
99.42
2.79
99.41
2.79
99.43
2.79
99.41
2.79
Table 6: Comparison of ASR (%) and ACC (%) for training on the purified CIFAR-10 with poisoning ratio ry = 0.09. Top 2 are bold.
Method
BadNets ASR,
ACC1
Blended ASRL
ACC1
ASR
SIG
ACC1
TrojunNN ASRL
ACC1
ASR,
SSBA
ACC1
ASR]
WaNett
ACC1
ASR
Average
ACC1
No detection
99.99
98.16
92.66
95.38
92.89
98.00
Strip
1.82
92.38
98.18
92.9
0.23
93.15
81.38
93.61
79.69
92.26
94.68
92.95
59.33
92.88
SS
87.2
80.77
85.19
93.61
SCAn
0.97
93.39
32.21
93.53
20.11
92.5
7.51
93.34
17.67
92.55
5.17
93.04
13.94
93.06
Frequency
75.71
92.05
87.3
91.9
91.95
1.27
93.12
66.58
90.2
91.88
91.6
70.44
91.80
CT
0.82
92.83
1.93
93.4
1.26
92.91
2.31
93.2
3.19
93.09
2.36
93.09
1.98
93,09
D-BR
88,14
93.23
95,78
91.44
99.97
93.82
100
93.21
97.42
92.54
95.54
92.33
96. 14
92.76
45.57
99.9
86.76
4.77
18.53
88,
87.76
VDC (Ours)
0.86
93.32
1,23
92.24
1.24
93.15
4.41
93.88
1.12
93.11
0.94
93.57
1.63
93.21
5.2.3 RESULTS ON DETECTING HYBRID DIRTY SAMPLES
In the real world, when an attacker poisons a realistic dataset, the dataset may already contain noisy labels. Therefore, in this section, we further evaluate the effectiveness of detectors when the dataset contains both poisoned samples and noisy samples, in which poisoning ratio is 0.09 and noisy ratio is 0.1 The results on CIFAR-10 are shown in Table 5, The following insightful points can found from the results:
Consistent Effectiveness Against Hybrids Poisoned Samples and Noisy Labels. In this more challenging scenario. VDC still shows leading advantages compared with other methods, with aver- age TPR reaching 99.41%. However, methods designed only for poisoned sample detection perform poorly when detecting mixture of various dirty samples, such as SCAn decreasing from 95.46% to 46.29%. In the meantime, methods designed only to detect noisy samples also underperform in this case, such as CL decreasing from 85.05% to 36.33%, which further illustrates the effectiveness and robustness of our proposed method.
5.2.4 TRAINING ON THE PURIFIED DATASETS
After detecting and removing dirty samples from the origin dataset, we normally train DNNs on the purified datasets verify the detection effect. The results on the purified datasets initially contain poisoned samples, noisy labels, and hybrid dirty samples are shown in Tables 6, 2,15166,1 17. By accurately detecting dirty samples, VDC indeed prevents the trained model from being interfered by dirty samples, ie., maintaining low ASR and high ACC compared with other detectors.
A CLOSER LOOK AT VDC
In this section, we provide further analysis and ablation studies of VDC and show some limitations.
Effect of the Type of Visual Questions. Figure 2a illustrates the influence of visual question types generated in VDC. We conducted experiments separately only using general questions label-
8

---

Published as a conference paper at ICLR 2024
100
01.
915
1101
60.2
44
87h
30.0
15.0
15.0
12.1
14
00
1pp0 of quesiion
Type quesiions
(a) Effect question types.
(b) Effect of question numbers.
(c) Effect of MILM.
Figure 2: Ablation results on the different aspects of VDC. (a) shows average results on CIFAR-10 with 7 0.09 under different types of visual questions, where G denotes general questions and S denotes label- specific questions. (b) shows average results on ImageNet-100 with 7 0.099 under different numbers of visual questions. (c) shows results of various poisoned samples on CIFAR-I0 with v 0.09 under different multimodal large language models.
specific questions while keeping all other settings constant. We observe that using only one type of question makes the model perform worse. In addition, label-specific questions slightly more important than general questions.
Effect of the Number of Visual Questions. We investigate the effect of the number of visual ques- tions generated in VDC. Figure 2b shows the detection results w.t.. various number of questions. We find that VDC's performance improves as the number of questions increases. But more ques- tions also lead to more inference time. Therefore, it becomes crucial to strike a balance between these two factors.
Effect of the Multimodal Large Language Model. In Figure 2c, we substitute the multimodal large language model in the VDC with Otter (Li et al., 2023), another recently open-sourced MLLM, to investigate the impact of MILM. Although the performance differs from those obtained with InstructBI.IP, still outperforms the majority of baselines. with the TPR for all poisoned samples consistently exceeding 96%, which further verifies the effectiveness of VDC.
Computational Complexity. Unlike other baselines that require training, VDC requires only inference of LLM and MLLM. Let K represent the number of classes, No, and N,, denote the number of general questions and label-specific questions respectively, T and T" denote the time of one inference of LLM and MILM. The overall time complexity O(TKN, N,,) + ( + N₂ )N) O(TN,, N), in which three terms correspond to the com- plexities of VQG, VQA, and VQE respectively. With the development of lightweight LLM,, such as quantization (Yao et al., 2023), the inference speed of LLM will increase, leading to a further reduction in the computational cost of VDC.
be expressed as
Limitations. 1) VDC hinges on the inconsistency between visual content and labels, making it inappropriate for detecting samples without corrupted labels, such clean-label backdoor attack. 2) Although ensembling technique has been employed in our framework to mitigate the risk of abnormal questions and answers, LLM and MLLM may still yield incorrect replies. However, the performance of VDC will also improve in the future as L.M progresses.
7 CONCLUSION
In this paper, we propose to detect dirty samples with corrupted labels by exploiting semantic incon- sistency between visual content and associated labels. To this end, we design versatile data cleanser (VDC), a universal detection framework harnessing the surpassing capabilities of large language models and multimodal large language models, which capable of detecting various categories and types of dirty samples. Experimental results validate the consistent superior performance of VDC in poisoned sample detection and noisy label detection. addtion, VDC still maintains effective- ness even when the dataset contains the hybrid dirty samples. Furthermore, anticipate that as large language models continue to evolve at a rapid pace, VDC will demonstrate further enhanced performance in the future.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China un- der grant No. 62076213, Shenzhen Science and Technology Program under grants No. RCYX20210602
Outstanding Youth Program of Guangdong Natural Science Founda- tion, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.
REFERENCES
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Confer- ence on Machine Learning, pp. 540-550, 2020.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing, 2019.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International conference on machine learning, pp. 825-836. PMLR, 2021.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang. Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.031000 2023.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint
2018.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In Advances in Neural Information Processing Systems, volume 35, pp. 9727-9737, 2022.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning, 2023.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings the AAAI conference on artificial intelligence, volume 31, 2017.
Tianyu Gu, Kang Liu, Brendan Dolan Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
Jonathan Hayase. Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: Defending against
backdoor attacks using robust statistics. In International Conference on Machine Learning, pp- 4129-4133 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Max Jaderberg, Karen Simonyan, Andrew Zisserman, al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF international conference computer vision, pp. 101- 110, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint W:Xiv:3366.0525,, 2023.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 230.008884, 2023.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium, 2018.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping-basee based backdoor attack. In International Conference on Learning Representations, 2021.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70: 10:1373-144 1411, 2021.
OpenAl. Openai api. URL https: httpppaaffooo openai com com/docs
Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. To- wards a proactive [ML] approach for detecting backdoor poison samples. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1685-1702, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Cosputer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2551 2015.
Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of (DNNs} for robust backdoor contamination detection. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1541-1558, 2021.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
Steven Fuijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal, 32(4):791-813, 2023.
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu. Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Thirty-sixth Conference Newral Information Processing Systems Datasets and Benchmarks Track, 2022.
11

---

Published as a conference paper at ICLR 2024
Baoyuan Wu, Li Liu, Zihao Zhu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Adversarial machine learning: systematic survey of backdoor attack, weight attack and adversarial example. arXiv preprint wxXV:22302.09457 2023.
Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study post-training quantization for large language models. arXiv preprint (rxXiv2333.083000 2023.
Chenglin Yu, Xinsong Ma, and Weiwei Liu. Delving into noisy label detection with clean data. In Proceedings the 40th International Conference Machine Learning, volume 202, pp. 40290- 40305, 2023.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In Proceedings of the IEEE/CVF international conference on computer vision, pp. (6473-1648) 2021.
Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang. and Xia Hu. Data-centric ai: Perspec- tives and challenges. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp. 945 948, 2023.
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In International conference machine learning, pp. 27412-2742) 2022.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX OVERVIEW
The overall structure of the Appendix is listed as follows:
Appendix B: Naive Approach with CLIP
Appendix C: More implementation details.
Appendix C.1: Details of training on the purified datasets.
Appendix C.2: Details of poisoned sample generation.
Appendix C33: Details of baseline Poisoned sample detectors.
Appendix C.4: Details of baseline Noisy label detectors
Appendix D: Prompts used in ChatGPT.
Appendix E: Examples of generated questions.
Appendix E.1: Examples of general questions.
Appendix E.2: Examples of label-specific questions.
Appendix F: Additional experimental results.
Appendix F.1: More results of poisoned sample detection.
Appendix F.2: More results of training on the purified datasets.
B A NAIVE APPROACH WITH CLIP
As we identified in the manuscript, how to measure the visual-lingussti inconsistency between the visual content and associated labels is the key to detect dirty samples. A naive approach to quantify such semantic inconsistency is directly using CLIP (Radford et al., 2021). We first encode the input image using image encoder in the CLIP, and get the image representation I. The associated label transformed into sentences, "a photo of [label}". Then the text representation T extracted from the sentence via text encoder in the CLIP. Cosine similarity between I and T treated as the matching score. If the matching score is less than a certain threshold, the input sample can be considered as dirty sample. In the implementation, we choose ViT-B/32 as the image encoder and the threshold is set as 0.2. The results are shown in Table 7, where VDC-CLIP represents the naive approach with CLIP. We find that the TPR using only CLIP is far from our proposed VDC, indicating the need for more advanced detection frameworks instead of only using CLIP.
Table 7: Comparsion of TPR (%) and FPR (%) the poisoned sample detection. VDC-CLIP denotes the naive approach with CLIP.
Dataset
Method
BadNets
Blended
SIG
TrojunNN
SSBA
WaNNl
TPR
FPR
TPR
EPR
TPR
FPR
TPR
FPR
TPR
FPR
TPR
FPR
VDC-CLIP
41.89
1.98
48.02
1.98
28.37
1.98
38.07
1.98
51.95
1.98
34.60
1.98
CIFAR-10 10 0.09
VDC (Ours)
99.93
2.75
99.87
2.75
99.84
2.75
99.93
2.75
99.91
2.75
99.96
2.75
CIFAR-10 0.009
VDC-CLIP
VDC (Ours)
100.00
41.55
2.83
2.72
99.56
49.33
2.83
2.72
29.33
99.78
2.83
2.72
100.00
39.56
2.83
2.72
52.00
99.78
2.83
2.72
100.00
36.78
2.72
2.83
ImageNet- 0.000
VDC-CLIP
80.81
1.84
77.52
1.84
82.34
1.84
70.14
1.84
77.90
1.84
82.20
1.84
VDC (Ours)
99.92
1.55
99.94
1.55
99.90
1.55
99.96
1.55
99.98
1.55
99.94
1.55
VDC-CLIP
1.75
77.00
71.11
1.75
77.78
ImageNot- 100 y 0.0099
VDC (Ours)
99.50
1.55
100.00
1.55
99.80
1.55
100.00
1.55
100.00
1.55
99.80
1.55
3.23
3.23
3.23
ImageNet-Drry 0.09
VDC (Ours)
98.89
4.12
97.50
4.12
98.61
4.12
99.31
4.12
98.89
4.12
98.89
4.12
C MORE IMPLEMENTATION DETAILS
C.I DETAILS OF TRAINING ON THE PURIFIED DATASETS
After successfully detecting dirty samples in the dataset, need to normally training the purified dataset further verify the effectiveness of detectors. our experiments, we choose ResNet-18 as the target model. For all datasets, the training epochs is set 100 and adpot SGD optimizer. For
13

---

Published as a conference paper at ICLR 2024
CIFAR-10, the batch size of 128 and the inital learning rate of 0.1 and decreases it by the factor after 50, 75 epochs. For ImageNet-100 and ImageNet-Dog, the batch size is 64, the inital learning rate is 0.1 and decreases by the factor of 10 after 30, 60 epochs.
C.2 DETAILS OF POISONED SAMPLE GENERATION
In this section, we present the settings for generating poisoned samples in backdoor attacks that are evaluated in the main manuscript. For all backdoor attacks, we choose class 0) as the target label.
BadNets BadNets (Gu et al., 2019) stands as seminal work in the realm of backdoor attacks, which introduces the concept of substituting specific pixels within a clean image with a well- designed trigger, thus yielding poisoned image. In experiments, for 32 X 32 image in CIFAR-10, we select a 3 x 3 white square patch located in the lower-right corner of the image to serve as the trigger. In the case of images with dimensions 224 224 from both ImageNet-100 and ImageNet- datasets, we utilize a white square patch with dimensions 21 x as the trigger.
Blended Blended Chen et al. (2017) firstly adopted the blended injection strategy to generate poisoned samples by blending benign input instance with the key pattern. The choice of the key pattern can be an arbitrary image. In our experiments, we use a "Hello Kitty" cartoon image (see Figure 3) trigger, and the blending ratio is set 0.1.
Figure 3: The Hello Kitty pattern used in Figure 4: The trigger mask used in TrojanNN. Blended.
SIG SIG (Barni et al., 2019) proposes horizontal sinusoidal signal designed by v(i,)) j) A sin (2 ff///////// /m), j m, < i < I, for certain frequency f. on the clean image, where m the number columns of the image and the number of rows. In the evaluation, we set A 20, f 6 for all datasets. The overlay backdooor signal is applied all the channels. In this case, the backdoor is almost, though not perfectly, invisible.
TrojanNN TrojanNN attack Liu et al. (2018) starts by choosing a trigger mask, which is a subset of the input variables that are used inject the trigger. Then it searches for value assignment of the input variables in the trigger mask so that the selected neuron(s) of the target model can achieve the maximum values. The identified input values are essentially the trigger. our evaluation, as shown in Figure 4, we choose to use the Apple logo as the trigger mask and ResNet- 18 as target model.
SSBA SSBA (Li et al., 2021) generates sample-specife invisible additive noises as backdoor trig- gers by encoding an attacker-specified string into clean images through an encoder-decoder network. Following the settings in (Li et al., 2021), we use U-Net (Ronneberger et al., 2015) style DNN as the encoder, a spatial transformer network (Jaderberg et al., 2015) as the decoder. The encoder- decoder trained for 140,000 iterations and batch size is set 16.
WaNet WaNet (Nguyen & Tran, 2021) uses small and smooth warping field in generating poi- soned images, making the modification unnoticeable. In our experiments, we adopt elastic image warping proposed in (Nguyen & Tran, 2021).
14

---

Published as a conference paper at ICLR 2024
Examples of Various Poisoned Samples As shown in Figure 5, choose image from ImageNet- 100 and visualize the examples various poisoned samples mentioned above.
(a) BadNets
(b) Blended
(c) SIG
(d) TrojanNN
(e) SSBA
(1) WaNet
Figure Examples of various types of poisoned samples.
C.3 DETAILS OF BASELINE POISONED SAMPLE DETECTORS
In this section, present the settings of poisoned sample detection baselines compared in experiments.
STRIP STRIP (Gao al., 2019) detects poisoned sample by checking whether superimposing the input image over set of randomly selected images makes those new image's class label harder to predict. If so, the input is considered to be normal and otherwise. In our evaluation, the FRR preset to be 0.1
SS SS (Tran et al., 2018) identifies spectral signatures of all known backdoor attacks to utilize tools from robust statisties to thwart the attacks. The upper bound on number of poisoned training set examples is set as 0.1.
SCAn SCAn (Tang et al., 2021) utilizes several statistical methods to estimate the most likely parameters for the decomposition and untangling models and then detect infected label through a likelihood ratio test. The threshold user for split clean samples in each classes is set as Euler's number e.
Frequency Frequency-basee detection (Zeng al., 2021) trains binary classifier based training set that contains DCT transformations of clean samples and samples with digital manipula- tions. For CIFAR-I0, 10, We directly use their provided pretrained detection model. For ImageNet-100 and ImageNet-Dog, we train a 6 layer CNN with the same settings as CIFAR-10.
CT CT (Qi et al., 2023) proposes confusion training that applies an additional poisoning attack to the already poisoned dataset, actively decoupling benign correlation while exposing backdoor patterns to detection. In our experiments, we set confusion factor = 20, the number confusion iterations 6000, the number of confusion training rounds K 6.
15

---

Published as a conference paper at ICLR 2024
D-BR We only use the training set into clean, poisoned and uncertain samples, according to the FCT metric. In our evalua- tion, we set a 0.2, ap is set the true poisoning ratio.
(SD) module in D-BR. SD module splits the whole
SPECTRE SPECTRE (Hayase al., 2021) uses robust covariance estimation amplify the spec- tral signature of corrupted data. In our experiments, is set as 4, poison fraction is set as 0.1.
C.4 DETAILS OF RASELINE NOISY LABEL DETECTORS
In this section, we present the settings of 5 noisy label detection baselines compared in our experi- ments.
BHN BHN (Yu et al., 2023) defines the p-values based on the neural network with the clean data. The p-values are then applied to the multiple hypothesis testing detect corrupted examples. In our evaluation, we set leave ratio as 0.4. We use ResNet- 18 for datasets, and training epochs set to be 200.
CORES CORES (Cheng al., 2021) trains ResNet-34 the noisy dataset and uses its proposed sample sieve to filter out the corrupted examples. In our experiments, we adopt its default setting during training and calculate the F1 of the sieved out corrupted examples. The training epochs is set as 40.
CL CL (Northeutt al., 2021) detects corrupted labels by firstly estimating probabilistic thresh- olds to characterize the label noise, ranking examples based on model predictions, then filtering out corrupted examples based on ranking and thresholds. In our experiments, we train ResNet- 18 on the noisy dataset and call the functions of Cleanlab to detect noisy labels.
SimiFeat-V and SimiFeett-R SimiFeat-V (Zhu al., 2022) uses "local voting" via checking the noisy label consensuses of nearby features to determine if the example is corrupted. Simileea- (Zhu al., 2022) scores and ranks each instance based on the neighborhood information and filters out guaranteed number of instances that are likely to be corrupted. In the evaluation, the KNN paprameter is set 10 and epochs is set as 21.
D PROMPTS USED IN CHATGPT
In this section, we present the prompts that we used query ChatGPT in our paper. Table shows the prompts used for the generation of label-specific visual questions for different datasets. Table shows the prompts used for the evaluation of the response of MILM.
E EXAMPLES OF GENERATED QUESTIONS
In this section, we show some examples of generated visual questions in the visual question genera- tion module of VDC.
E. EXAMPLES OF GENERAL QUESTIONS
Table 10 shows the general questions used for acquiring holistic descriptions of the image, with some prompts sourced from (Liu al., 2023).
E.2 EXAMPLES OF LABEL-SPECIFIC QUESTIONS
Table 11 and 12 show the examples of label-specifie visual questions on ImageNet- 100.
thub. htte:////gthhhocc////aaaaaaaaaaa
16

---

Published as a conference paper at ICLR 2024
Table The prompts used for the generation label-specific visual questions for different datasets with ChatGPT. [label;] represents the label name of class i, {n} denotes the number questions of each lable.
Dataset:
CIFAR-10, ImageNet-100
Prompt:
Please generate some visual questions to ask a multimodal large language model to identify the label of an image is correct. These questions will help determine if the object in the image corresponds to the given label. Remember that the goal is to ask questions that would lead to a 'yes' answer if the label is correct.
The labels are [labell)"......(lbbbee )'] generate {n} the most insightful questions for each lahel.
For example, the label is 'airplane', the possible questions could be: Can the object in the image be used to fly in the air?
Dataset:
ImageNet-Dog
Prompt:
Please generate some visual questions ask a multimodal large language model to identify if the label of an image is correct. These questions will help determine if the breed of the dog in the image corresponds to the given label. Remember that the goal is to ask visual questions that would lead to 'yes' answer if the label is correct.
The labels are [' [l(label)) ) )...., "[abell JJ. generate {n} different questions for each label, such as the breed, attributes. The questions of each label should be used to judge different breeds.
For example, the label is 'Chihuahua', the possible questions could be: Does the dog in the image have any distinct features of a Chihuahua?
Table 9: The prompts used for the evaluation of the response of MILM with ChatGPT. (label} rep- resents the label name, [response) represents the response of MILM visual qunestion answering module.
Prompt:
Assume you helpful and precise assistant for evaluation. Please judge whether the "Caption" of an image and one of the 'Label' refer to the same object. Answer with yes or no. Caption: "reepposee"" Label: (label)"
17

---

Published as a conference paper at ICLR 2024
Table 10: The list of general questions for image description.
Describe the image in detail.
Describe the image briefly.
How would you summarize the content of the image in few words?
Provide detailed description of the given image.
Describe the image concisely.
Provide brief description of the given image.
Offer a succinct explanation of the picture presented.
Summarize the visual content of the image.
Give a short and clear explanation of the given image.
Share concise interpretation of the image provided.
Present compact description of the photo's key features.
Relay brief, clear account of the picture shown.
Render a clear and concise summary of the photo.
Write a terse informative summary of picture.
Create compact narrative representing the image presented.
Please generate detailed description of the dog in the image, including the breed of the dog, its specific attributes, unique features that can distinguish it from other breeds. (for ImageNet-Dog)
F ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide more experimental results that mentioned in the manuscript.
F.1 MORE POISONED SAMPLE DETECTION RESULTS
Table 13 shows the detection results CIFAR-10 with poisoning ratio r) 0.009, i.e., 50 poisoned samples per class.
Table 14 shows the detection results ImageNet-100 with poisoning ratio 1) 0.0099, ie., poisoned samples per class.
The results show the consistent effectiveness of VDC across different datasets and poisoning ratios.
F.2 RESULTS OF TRAINING THE PURIFIED DATASETS.
Table 15 shows the normally training results on the purified CIFAR-10 with poisoning ratio 1) == 0.009, i.e., 50 poisoned samples per class.
Table 17 shows the normally training results on the purified CIFAR-10 with poisoning ratio 7) = 0.09 noisy ratio 7/2 0.1.
The results show that our proposed VDC can indeed improve the reliability and usability of DNNs trained with dirty samples.
18

---

Published as a conference paper at ICLR 2024
Table 11: Examples of label-specific questions on ImageNet-100.
Label:
cock
Questions:
Is the object in the image belong to the type of cock? Is the object in the image a type poultry?
Is the object in the image commonly found on farms or in rural areas?
Does the object in the image have a comb on top of head? Is the object in the image known for its distinctive crowing sound?
Does the object in the image have sharp spurs on its legs?
Label: Questions:
goldfinch
Is there a type of bird in the image? Is the object in the image a type of finch?
Does the image feature small bird known for its vibrant yellow and black coloration?
Is the object in the image a type finch with bright plumage?
Is the bird in the image small passerine species?
Is the bird in the image belong to carduelis?
Label: Questions:
scorpion
Does the object in the image have a venomous stinger? Is there a scorpion in the image? Is the object in the image a type of arachnid? Is the creature in the image venomous?
Is the creature the image commonly found in desert regions?
Does the image show scorpion?
Label:
koala
Questions:
Is there a koala in the image?
Does the object in the image have a round face with large, fluffy ears? Is the object in the image known for its ability to climb trees?
Does the animal in the image primarily feed on eucalyptus leaves?
Is the object in the image typically gray in color? Is the object in the image commonly found Australia?
Label:
flamingo
Questions:
Is the image showing flamingo?
Is the object in the image commonly found in wetland habitats? Does the object in the image have a long, curved neck?
Is the bird in the image tall with long legs?
Is the bird in the image known for its vibrant pink plumage
Is the bird in the image known for standing on one leg?
Label: Questions:
gorilla
Is the creature in the image type of gorilla? Is the animal in the image known for its intelligence? Is the animal in the image known for its strength? Is the animal in the image primate?
Is the object in the image commonly found in forests or jungles?
Does the object in the image have a large and robust body?
Label: Questions:
dumbbell
Does the image show dumbbell? Is there a dumbbell in the image?
Is the object in the image often used to build muscle strength? Is the object in the image associated with fitness training? Is the object in the image commonly hold with hands? Is the object in the image a type of exercise equipment?
19

---

Published as a conference paper at ICLR 2024
Table 12: Examples of label-specific questions on ImageNet-100.
Label:
hatchet
Questions:
Is there a hatchet in the image?
Does the image show hatchet?
Is the object in the image a type of cutting tool?
Is the object in the image typically used for chopping?
Is the object in the image often used for splitting wood? Is the object in the image typically held with hand?
Label: Questions:
stethoscope
Is the object in the image a stethoscope? Does the object in the image have a distinct Y-shaped design? Is the medical instrument in the image stethoscope? Is the device primarily used by doctors?
Is the tool in the image commonly used to listen to heartbeats? Is the device in the image associated with medical examinations?
Label:
broccoli
Questions:
there broccoli in the image?
Is the object in the image a vegetable with thick, edible stalk? Is the vegetable in the image broccoli? Is the vegetable in the image green?
Is the object in the image commonly used in salads? Is the object in the image often cooked or consumed for its health benefits?
Label: Questions:
space shuttle
Does the image show a space shuttle?
Is the object the image capable launching vertically into space?
Is the object in the image equipped with powerful rocket engines for propul- sion?
Is this spacecraft used to transport astronauts and cargo? Is the object known for its missions to the International Space Station?
Is this vehicle that was used by NASA for space exploration?
Label: Questions:
pomegranate
Is the fruit in the image pomegranate? Is the fruit the image typically red or reddish in color? Does the fruit in the image have tough outer rind?
Is the fruit the image typically used to make juices and other beverages? Does the object in the image have crown-like structure the top? Does the object the image have a segmented interior filled with clusters of juicy, ruby-red seeds?
Label: Questions:
radio telescope
Is the object in the image a type of radio telescope?
Does the object in the image have a large dish or antenna-lic like structure?
Does the object in the image have parabolic spherical reflector to focus radio waves?
Is the device in the image used for radio astronomy? Is the equipment in the image designed to receive radio waves from the universe?
Does the image depict a device that contributes to radio astronomy re- search?
20

---

Published as a conference paper at ICLR 2024
Table 13: Comparison of TPR (%) and FPR (%) for poisoned sample detection CIFAR-10. 0.009, i.e., 50 poisoned samples per class. Average is the mean of results different triggers.
Dataset: CIEAR-10
7=
0.009
poisoned samples per class)
BadNets
Blended
SIG
TrojanNN
SSBA
WaNet
Average
Method
Data
TPR:
FPR.
TPR
[PR]
TPR]
FPRJ
TPR1
[PR]
TPR:
IPR.
TPR
FPR.
TPR]
TPR.
STRIP
4%
86.22
11.67
4.22
10.86
99.56
11.27
99.78
9.84
65.11
9.87
2.44
9.96
$9.56
10.58
4%
97.56
12.72
99.78
12,70
100.00
12.69
3.33
13.57
99.33
12.70
92.00
12.77
82.00
12.86
SCAn
92.22
2.28
2.83
2.81
88.00
2.28
34.54
2.65
2.46
Frequency
4%
89.11
21.51
84.22
21.55
48.67
21.76
100.00
19.32
85.56
21. 66
39.78
21.72
74.56
21.25
CT
97.56
100.00
1.01
100.00
3.92
100.00
1.82
76.00
2.58
95.51
2.05
D-BR
0%
0.44
0.91
0.00
0.90
0.00
0.90
11.11
0.78
1.11
0.91
1.33
0.89
2.33
0.88
SPECTRE
VDC (Ours)
0%
100.00
2.72
99.56
2.72
99.78
2.72
100.00
2.72
99.78
2.72
100.00
2.72
99.85
2.72
Table 14: Comparison of TPR (%) and FPR (%) for poisoned sample detection ImageNet-100. 7) = 0.0099, i.e., poisoned samples per class, Average is the mean of results different triggers.
ImageNet-100
poisoned samples per class)
BodNets
Blended
SIG
TrojanNN TPR1
SSBA
WaNNet
Average
Method
Data
TPR
TPR?
TPR:
TPR1
TPR
FPR
TPR1
FPR.
STRIP
4%
89.70
11.94
79.39
11.36
100.00
11.03
98.99
11.09
99.60
11.61
1.62
12.50
78.22
11.59
4%
48.08
49.92
54.14
49.86
47.47
49.92
48.08
49.92
49.49
49.90
50.91
49.89
49.70
49.90
SCAn
4%
87.47
88.89
98.99
86.46
2.12
97.37
92.56
2.35
Frequency
4%
1.62
1.57
1:21
1.57
1.62
1.57
94.75
1.57
3.03
1.57
0.00
1.57
17.04
1.57
CT
D-BR
0%
1.01
1.99
1.41
1.65
0.00
1.98
0.81
1.81
1.21
1.77
1.21
1.82
0.94
1.84
60.20
VDX (Ours)
0%
99.80
1.55
100.00
1.55
99.80
1.55
100.00
1.55
100.00
1.55
99.80
1.55
99.90
1.55
21

---

Published as a conference paper at ICLR 2024
Table 13: Comparison of ASR (%) and ACC (%) for training on the purified CIFAR-I0 with poi- soning ratio 0.009.
RadNets
Blended
SIG
TmjjnNN
SSBA
WaNett
Average
Method
ASR
ACC1
ASR
ACC1
ASRL
ACC1
ASR.
ACCT
ASR,
ACC1
ASRJ
ACCT
ASR,
ACC1
No detection
91.83
93.67
74.00
93.63
99.64
93.50
99.99
93.56
72.86
93.70
13.18
93.37
75.25
93.57
Strip
0.27
93.15
10.79
14.83
93.06
SS
0.97
92.62
0.98
92.9
0.41
92.77
99.94
92.74
1.21
92.76
0.89
93.15
17.40
92.82
1.59
93.09
0.23
93.19
92.89
21.44
93.73
4,88
93.32
Frequency
0.86
92.54
5.83
93.15
98.44
92.4
2.17
92.69
2.33
93.01
6.32
91.62
19.33
92.57
CT
0.79
93.24
0.71
93.94
0.12
93.7
3.96
93.17
0.57
93.76
1.32
93.55
1,25
93.56
99.58
99.98
93.73
93.65
SPECTRE
0.87
92.89
1.26
92.94
0..21
92.99
4.1
92.96
1.06
92.92
1.07
92.9
1.43
92.93
VDC (Ours)
0.61
93.29
0.69
93.73
0.31
93.14
3.10
93.47
1.02
93.72
0.76
93.74
1.08
93.52
Table 16: Comparison of ACC (%) for training on the purified datasets with noisy labels.
CIFAR-10
0.4
ImageNe-100
0.4
ImageNet-Dog
0.4
Method
Symmetric
Asymmetric
Symmetric
Asymmetric
Symmetric
Asymmetric
No detection
61.84
56.09
31.21
32.65
28.45
31.35
BHN
88.71
89.21
40.12
44.25
31.65
38,90
CORES
84.68
84,70
38.41
41.87
18.70
37.05
CL.
87.82
57.94
39.19
46.98
18.25
30.00
SimiFcat-V
89.39
74.70
37.81
41.68
33.70
34,90
SimiFeat-R
90.48
80.54
39.70
31.86
28.80
VDC (Ours)
90.75
90.89
66.6
69.32
46.54
48.80
Table 17: Comparison of ASR (%) and ACC (%) for training on the purified CIFAR-10 with poi- soning ratio ih 0.09, noisy ratio 7 0.1.
Dutasel: CIFAR-10
poisuning ratio
0.09
moiey ratio (b
BadNets
Blended
SIG
TrojanNN
SSBA
WaNet
Averuge
Method
ASR
ACC1
ASR!
ACC1
ASR
ACC²
ASRJ
ACC1
ASR
ACC1
ASR
ACC1
ASRJ
ACC1
86.54
Strip
85.84
96.13
85.24
85.97
84.35
57.91
84.66
85.25
42.06
85.22
SS
94.38
78.33
95.31
81.33
99.96
81.58
99.97
80.56
92.81
78.45
69.07
77.34
91.92
79.60
SCAn
2.18
86.9
4.87
85.75
4.91
85.17
4.2
86.99
3.97
86.12
5.44
83.41
4.26
85.72
Frequency
76.38
CT
2.46
85.41
1.53
86.49
0.97
85.26
55.96
86.1
9.18
84.49
5.39
86.46
12.58
85.70
D.BR
90.72
86.09
96.3
86.59
99.86
86.04
100
85.46
96.59
86.16
94.93
85.11
96.40
85.91
SPECTRE
BIIN
73.04
91.22
50.73
91.68
99.98
85.61
100.00
85.13
95.77
84.54
88.11
83.40
84.61
86.93
CL
95.86
88.92
98.10
90.29
99.97
86.01
99.99
85,58
96.38
85.12
91.19
84.19
96.92
86.69
CORES
95.88
81.94
96.53
84.95
99.96
85.44
100:00
86.
95.18
84.42
93.90
84.22
96.91
84.51
SimiFent-V V
1.21
92.38
71.01
92.21
99.96
84.76
99.99
85.42
95.74
84.07
92.97
84.35
76.81
87.20
SimiFeat-R R
1.04
92.79
67.91
92.34
99.97
85.23
99.99
85.05
95.46
84.47
88.73
82.90
75.52
87.13
VDC (Ours)
1.01
92.58
1.13
91.73
3.07
91.67
4.59
92.79
1.39
92.06
0.99
92.63
2.03
92.24
22


Reviews:
[{"review_id": "VV87JWnLKE", "reviewer": "Reviewer_N7YG", "summary": "This paper proposes a working pipeline for noisy data detection. Compared with existing works focusing on noisy data or noisy label, this work aims to obtain an integrated framework to handle a comprehensive scenario including various noisy cases. Specifically, it terms it as visual-language inconsistency. Leveraging on several prompt techniques, the proposed framework achieves promising results compared with other baselines.", "questions": "Please refer to the strengths and weakness for details. Even if I am concerning some points in weakness, I recognize other aspects of this paper mentioned in strengths. Overall, I lean to vote for an acceptance for now and I would like to check other reviewers' comments to discuss and make my final decision.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Noisy or dirty data detection and cleaning is an important research topic. It is becoming even more critical for recent machine learning research since the data scale is always getting larger.\n2. The proposed framework wisely utilize the advantage of current large-scale model to benefit the dirty data detection task.\n3. Comprehensive empirical results show the framework superiority compared with other baselines.\n4. The whole draft is in a good format for readers.", "weaknesses": "1. I mainly concern about the technical contribution in this draft. The wise combination of prompting and dirty data detection is interesting. However, it still based on the visual-language understanding from large-scale pretrained model. Only based on such powerful tools relatively diminish this paper novelty. In addition, the key point of this paper is proposing an integrated detection pipeline instead of only focusing on sample or label. This point looks like a trivial combination which is incremental compared with previous settings. Is this setting practical and necessary for real-world scenarios?\n2. Adding more recent published works for comparison may further help to support the paper contribution. Currently, only one or two baselines are published within past one year.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpFKkIcfAp", "reviewer": "Reviewer_481L", "summary": "To address common noise data and backdoor attack problems in deep learning, the paper designed a data cleaning tool called VDC using a multimodal large language model. By designing generalized visual question answering questions and label-specific visual question answering questions, VDC can remove dirty and poisoned data from the dataset based on the inconsistency between the semantic of the obtained question results and the semantic of the image itself. The paper compared VDC with backdoor attack methods and noisy learning methods, demonstrating the effectiveness of VDC.", "questions": "+ How is the threshold for excluding dirty data determined in the paper?\n+ Has the author considered the scenario where MLLM itself serves as a detector rather than a data cleaner?\n+ The paper's method relies heavily on MLLM, which in essence depends on more data and better labeling. Therefore, it may not be necessary to test models that use more clean data on small datasets, as this may go against the goal of advancing deep learning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper's approach is innovative, using popular multimodal large models to replace manual data cleaning work.\n+ The paper unified the treatment of noisy data and poisoned data from backdoor attacks, which is relatively rare in previous research.\n+ The paper's extensive experiments demonstrate the effectiveness of VDC in handling dirty data.", "weaknesses": "+ The overall content of the paper seems to be more about using out-of-distribution methods to filter data, and perhaps this should be reflected in the accuracy of sample selection.\n+ While the paper effectively addresses the issue of dirty data using MLLM, it seems to have a bias towards reporting the application of MLLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oa4JbDyZ22", "reviewer": "Reviewer_vuG8", "summary": "The paper proposes a general \"label\" cleaning/filtering approach and aims to detect three types of errors: \"poisoned samples\", \"noisy samples\", and \"hybrid dirty samples\". The paper takes advantage of the exceptional capability of multimodal large language model (MLLM) and casts the error detection problem into a three-step scoring pipeline. The pipeline consists of 1) visual question generation, 2) visual question answering, and 3) visual answer evaluation. The main argument is that, unlike prior arts, the proposed approach can detect all three types of label errors and achieves better performance in common benchmarks, including ImageNet-100 and CIFAR-10.", "questions": "1. The name of “visual question generation” is confusing. From my understanding to the paper, there is no visual signal in this stage. Can the author confirm this?\n2. What are the accuracy of each question?\n    - Figure 2 shows the TRR of the general and label-specific questions. I am curious of what specific question is challenging for the instruct-BLIP. Is there any specific question that is always challenging to the instructBLIP and, therefore, removing those questions actually help the overall performance\n3. Given that the method leverages a ensemble of the MLLM responses, instead of taking the average accuracy as score as in Eq. 4, does the author think that using techniques like label aggregation further improves the overall performances?\n4. To make a fair comparison with other baselines, I suggest the authors could compare baselines with MLLM as well. For example, CL usually trains their classifier with leave-one-out or cross validation, which limits the size of the train dataset. However, one can also uses off-the-shelf classifier and apply CL on it.\n5. Do the authors consider the data-leakage problem? Does the train dataset used to train the instruct-BLIP accidentally include the images in ImageNet or CIFAR? If so, how do accurately validate the performances?\n6. Do the authors observe any accumulated error in the three-step pipeline? For example, the label-specific question include some noisy questions, which leads to poisoned answers in the second stage.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method leverages the recent trend of MLLM to the label error detection literature\n- The proposed method is training-free", "weaknesses": "- The empirical comparison is not fair.\n    - The propose approach is using instruct-BLIP (larger network trained on larger dataset), while the baseline is usually using less-expressive network trained on smaller datasets, e.g., CL is using ResNet-18 and trained on CIFAR dataset.\n- The claim that the proposed method mitigates all three types of label noises is too strong.\n    - The proposed approach is general, but so do other baselines. For example, SimiFeat-V leverages the feature similarity to detect noisy labels, which is also applicable to the scenario of “poisoned samples” as long as the feature extractor is trained on a poison-free dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)