MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection
Authors: Xi Jiang
Jian Li
Hanqiu Deng
Yong Liu
Bin-Bin Gao
Yifeng Zhou
Jialin Li
Chengjie Wang
Feng Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Anomaly Detection
Multimodal Large Language Model
Industrial Inspection
Primary area: datasets and benchmarks
TL;DR: We established the first industrial benchmark for MLLMs and revealed several current limitations of MLLMs through extensive experiments.

Abstract: In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, a full-spectrum MLLM benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9\%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research. 
The code and data are available at https://github.com/jam-cc/MMAD.

Published as a conference paper at ICLR 2025
MMAD: A COMPREHENSIVE BENCHMARK FOR MUL- TIMODAL LARGE LANGUAGE MODELS IN INDUSTRIAL
ANOMALY DETECTION
Xi Jiang'
Jian Li
Hanqiu Deng" Yong Liu²
Bin-Bin Gao
Yifeng Zhou
Jialin Li Chengjie Wang21 Feng Zheng
Southern University of Science and Technology Tencent YouTu Lab
University Alberta *Shanghai Jiao Tong University
jiangx2020@mail sustech. edu. en, hangiul@ualberta.com [swordli, choasliu, danylgao, joefzhou, jarenli, jasonejwang] @tencent. com, f. z.zengg@ieee.c org
ABSTRACT
In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have high potential renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive bobblem-solving. skills in many domains, MLLMs' abil- ity industrial anomaly detection has not been systematically studied. To bridge this gap, present MMAD, full-spectrum MLLM benchmark in industrial Anomaly Detection. We defined seven key subtasks of MILMM in industrial inspection and designed a novel pipeline generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-arr MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial re- quirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and de- fects. We further explore two training-free performance enhancement strate- gies to help models improve in industrial scenarios, highlighting their promis- ing potential for future research. The code and data are available at https: //github. com/ wwt.uuub..coo///acc//////// CCMMAA/MADD
INTRODUCTION
Automatic vision inspection is a crucial challenge realizing an unmanned factory (Benbarrad et al. 202T). Traditional AI research for automatic vision inspection, such as industrial anomaly detection (IAD) (Jiang et al. 2022b Ren et al. 2022 Zhang et al. 2024b), typically relies on dis- criminative models within the conventional deep learning paradigm. These models can only perform trained detection tasks and cannot provide detailed reports like quality inspection workers. Addi- tionally, when production lines requirements change, traditional methods necessitate retraining redevelopment. The development of MLLMs (Jin et al. (2024) has the potential to alter this situation. These generative models can flexibly produce the required textual output based on input language and visual prompts, allowing us to guide the model using language similar to instructing humans.
Nowadays. multimodal large language models, represented by GPT-4 (Achiam et al. 2023), can already do many human jobs, especially high-paying intellectual jobs like programmers, writers, and data analysts (Eloundou et al. 2023). In comparison, the work of quality inspectors is simple, typically not requiring a high level of education but relying heavily on work experience. Therefore, we are greatly interested in the question:
How well are current MLLMS performing as industrial quality inspectors?
Cooreeponndnnce Feng Zheng (icienn@@icceorrg)

---

Published as a conference paper at ICLR 2025
Anomaly Descrimiiation
MVT AD
Gonded
Object Analysis
Defect Classification
Objict Analysis pesaay
Discomiiation Saai
Annmaly
Normal images 1229
pco
Offject Conaniiation
Abgormal Inrages
Defert Cloca
[4r08]]
MV LOCO AD
Object Classifiation
Dofect Localization
Detecl Anulyais [4116]
[50766]
Loxe atication
Defert
Defect Deveription [4512]
(4499)
Defect Analysis
Defect Description
Ruman
AnomelyGPP
LL-W-115-130
VIAA
GPT-4o
Cambrian-1
IntemVL2-720
Figure I: Left: Innermost layer: image components, middle layer: subtasks composition, outermost layer: object categories. MMAD covers key subtasks and 38 representative categories of IAD. Right: Results of 5 representative MLLMs and Human. The left-skewness indicates that models perform well object-related questions but poorly on questions related to defects.
Recent studies have attempted to explore this issue. Through instruction following mechanisms (Dai jet al., 2023; Liu et al., 2024b), some reports evaluate IAD examples with MLLMs, demonstrating
the advantages of MLLMs in generalization and flexibility (Zhang et al.) 2023a Cao et al. 2023
Xu et al. 2024]. Unfortunately. they only tested a few qualitative examples with no quantitative results. Other studies, such as AnomalyGPT (Gu et al. (024) and Myriad (Li et al. 2023), specif- ically trained MLLMs to understand the outputs of traditional IAD models. However, they use the traditional evaluation, which measures the ability of expert models rather than MLLMs. The lack of unified test data and output format also led to an incomplete comparison with other general models. So, a comprehensive benchmark to compare the quantitative results of MLLMs in IAD necessary.
In this paper, we introduce the first-ever benchmark for MILMS in IAD tasks. Current MILMs are challenging evaluate directly on IAD tasks because existing publicly available datasets only contain visual perception annotations and category labels, lacking rich semantic annotations. To ad- dress this issue, we designed comprehensive pipeline. First, we utilized GPT-4V to generate rich semantic annotations with visual annotations and language interaction. Based on the semantic an- notations, we generated questions and options for testing, which were manually reviewed to ensure their reasonableness and accuracy. Ultimately, we collected 8,366 samples from 38 classes of indus- trial products across public datasets, generating a total of 39,672 multiple-choice questions key subtasks, illustrated in the left panel of Figure [ Our benchmark encompasses representative image data and language- based evaluation methods, providing a fair and reasonable assessment of MILMS' performance TAD. Figure
2 provides some examples.
With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of sittee-f--haa art (SOTA) MLLMs, including the GPT-4 series and Gemini 1.5 series (Reid et al. 2024), well as open-source image models like VLternVL2 (Chen et al. (202)) and LLa LLaVA-NeXT (Liu et al. (022a)) and industry anomaly detection models like AnomalyGPT (Gu et al.) 2024). As shown in the right panel of Figure [ our experiments demonstrate that GPT-4o is the best-performing model, reaching 74.9%. but as the scale of models increases, some open-source models are gradually approaching the capabilities of these commercial models. However, industrial scenarios often demand high ac- curacy, and the current accuracy levels are far from meeting practical standards. Additionally, both open-source and proprietary models perform significantly worse when answering questions related to anomalies and defects compared to questions about objects, which reveals an important capa- bility weakness of MILMs. Notably, the model specifically designed for IAD, AnomalyGPT (Gu) et al. 2024), performed the worst overall, as it could only handle anomaly discrimination tasks was trained and failed respond adequately to questions in other subtasks. This indicates that enhancing MLLMs' capabilities in LAD through training requires larger-scall scale industrial data and the provision of more comprehensive industrial knowledge.
2

---

Published as a conference paper at ICLR 2025
Anomaly
Clasdifcation
There is defect the object. What the of the defect?
Is there defect in the ohject?
B: No.
B: Discolmiiim
C: Wrinklingg
Object Clasilication
Defect Localization
What kind product the image? section garlen hose: R: A cross-section of triphrse elecriial
There is defect in the object. Where is
defect?
A: Top left candle
Top right candle C:
bundle fiber optic cables.
Dr piece of computer hardware.
D: Bottom right candle
Object Analysis
Defect Description
are the sulcomponcnts the
There is defect in the object. What appearance
breakfast box?
defect?
Oranges, nectarine, granola,
A: The cap slightly ajar on one side B: The completelly missing botile dented
bamana siices
B: Oninge, apples, cercal, and dried thit
Granola, yogurt benies
D: The label is poeling off'
D): Bread, cheese, and vegetables
Defect Analysis
What is tikely purpose emphasis traditional design elements on the eigarette box?
There is defect in abject. What is the potential effect of the defect??
convey modemity
Reduced performance
B: To highlight the product's origin
B: Incretsed power consupption
C: To aitract emplasize the product's quality
C: Improper insertinn
D: Shorter lifospm
Figure 2: Examples of subtasks of MMAD. Each question presented in a multiple-choice for- mat and includes several distractor options. We present different categories of objects in various examples to demonstrate the diversity.
Beyond standard testing, we also conducted experiments on different settings and scaling laws, which aid in further comparative studies. Given the current shortcomings of MLLMs in addressing anomalies and defects, we explored two performance boost schemes that do not require additional training: Retrieval-Augmenter Generation (RAG) and expert agent, the previous augmentation and the other through visual augmentation. These methods can improve MLLMs' performance in IAD some extent, but they are still limited by the fundamental capabilities of the models. Overall, current MILLMs are not yet capable of effectively performing the duties a quality inspector. They require further supplementation with IAD knowledge and enhanced capabilities for fine-grained understanding and gross-comparison of multiple images.
through text
The contributions of this paper are summarized follows:
To the best of our knowledge, our proposed MMAD is the first evaluation benchmark for MLLMs in the task of IAD. This benchmark fills the gap the application of MLLMs in the industrial domain and sets new challenges for their capabilities.
We introduce a novel pipeline for generating semantic annotations for visual anomaly de- tection data, addressing the issue that MLLMs cannot be directly evaluated on IAD datasets.
We comprehensively evaluate the performance of representative MLLMs MMAD, high- light the weaknesses current MLLMs fine-grained industrial knowledge and multi- image understanding, and provide two general boost schemes.
2 RELATED WORK
2.1 MULTIMODAL BENCHMARKS
In recent years, substantial efforts on benchmarks have been dedicated to exploring MLLMs LLi & Lu 2024) from various perspectives. However, the capabilities assessed by these benchmarks differ significantly from those required for IAD. Firstly, most MLLM benchmarks primarily focus single-ime -mage input scenarios, such MME (Fu et al.] 2023), MMBench (Liu et al., 2023b), and MMVP (Tong et al., 2024b). In contrast, industrial quality inspection necessitates the ability to process multiple images, industrial knowledge highly specialized and often requires additional
3

---

Published as a conference paper at ICLR 2025
Table 1: Statistics on the composition and quantity of MMAD data.
Sampled Images
Generated Questions
Object Categories
Defect Categories
Image Source
Specialty
MVTec AD
A variety of objects and textures Including both structural and logical anomalies Containing multiple instances and complex structure Multiple goods each category
[erggaan 2019
1691
8338
15
73
MV LOCO AD
Bergmann et 2022
1566
7624
5
89
Zou et al. 2022
VISA
2141
10622
12
67
2968
13088
6
15
Zhang al. 2024a
SUM
8366
39672
38
244
images for product recognition. Secondly, most benchmarks emphasize the general capabilities of MLLMs rather than the specific needs of particular domains. For instance, TextVQA (Singh et al. 2019) focuses on text recognition in images, MATH-Vision (Wang et al.] (024c) and MATH-
VISTAQLA al. 2023) evaluates mathematical reasoning in visual contexts, and Video-MME (Fu iet al.) 2024) emphasizes video understanding. Most importantly, these benchmarks do not ad- dress the industrial domain. For example, MMMU (Yue et al. (024) covers lots of fields, such as Art, Business, Science, Social Science, and Engineering, but not industry. Seed-Bench (Li et al.) 2024a) and CompBench (Kil et al.] 2024) test multi -mage comparison capabilities, but both in- volve fictional tasks and cannot provide realistic references. In contrast, the medical field, which has tasks and data formats similar to the industrial domain, already has numerous benchmarks for MILMs, including Asclepius (Wang et al.) 2024d), GMAI-MMBench (Chen et al.) (022bb) and PMC-VQA (Zhang et al. (203bb). Therefore, proposing the first MLLMs benchmark for the indus- trial domain would fill significant gap.
2.2 INDUSTRIAL ANOMALY DETECTION
Traditional IAD research primarily aims to address a significant issue in industrial visual inspection: discriminating and localizing defects without samples of defects. Consequently, traditional LAD methods typically involve training on a large number of normal samples and then using outlier detection techniques to identify anomalies in test samples. Common approaches include memory bank-based methods (Jiang et al.) [202aa] Wang et al.] 2025), reconstruction-basse methods (Deng)
Li 2022 Jiang et al. 2024a), and methods based on training with synthetic anomalies (Zavrtanik
et al. 2021 Recent research has begun to focus on the generalization capability of IAD. Leveraging vision-language models such as CLIP, some few-shot, and even zero-shot models have emerged, such as AnomalyCLIP (Zoou et al. 2023 M3DM-NR (Wang et al. (024a) and InCTRL (Zhu & ( Pang 2024).. However, these discriminttive models heavily rely on predefined anomaly concepts in the CLIP model, limiting their ability to generalize to scenarios. For instance, models trained on structural anomalies struggle to detect logical anomalies (Bergmann et al.) 2022). The emergence of
MLLMs may help address this challenge, as they can understand complex textual inputs and provide diverse responses in conjunction with visual components.
Some studies have already demonstrated the advantages of open input-output formats of MLLMs in IAD. For example. LogiCode (Zhang et al., (024c) uses MLLMs for logical reasoning and automat- ically generates code address different types of logical anomalies. Xu et al.) (2024) proposes that designing visual and language prompts can directly enhance the performance of MIL.Ms in IAD tasks. Additionally, some research has begun to train MILMSs directly on IAD datasets, such as AnomalyGPT (Gu et al. 2024), Myriad (Li et al.) 2023). and FabGPT (Jiang et al. (024b). How- ever, these three models heavily depend on the capabilities of expert models and cannot freely extend their capabilities by changing inputs. Moreover, we have found that due to the small amount of IAD data (Wang et al. (2024b), MLLMs like AnomalyGPT are highly prone to overfitting. In conclusion, current LAD methods rarely meet our paradigm,
focus our evaluation general MLLMs.
3 THE MMAD DATASET
3.1 DATA COLLECTIONS
An excellent industrial quality inspector should be able to adapt to the inspection tasks of different products, the skills involved in visual inspection are similar. Therefore, our designed benchmark
4

---

Published as a conference paper at ICLR 2025
Caption Instruction
VQA Instruction
You inspector checks...
Image-caption Pair
You are given asage and its caption... muliple-choice questions... Format requirement...
Plave doserioe the FIRST image detiil...
Visšon Prompt
Question Definition
Image
Examples
Highlight
The finst questton should
GT (musk))
distinguist whether there the object, such as yes, the sccond question should
image cross-scctional view of en cleetrical cable with three
LT Manual Filtering
conducttors (green/ycllow, blue, and brown) enclesed in insulating sheuth. cunductor is deformed, with strands beut and splayed out from the central axis, tightly packed amuggammnt defect imguet cable's electrical performance and pose a safety risk."
Scarch
Nearest
Normal
Template
Questions, Options und Answers
Q1: Is there any defect the object?
Knowledge
devisting
A: Yes.
Objeett cotegary: "cable":
R No.
Defect categary:
"bent wire".
Defect localization:
image"
Q2: There is defect in the object. What
The VQA data generation pipeline for IAD. We utilize images from the open-source IAD dataset and leverage GPT-4V to automate the generation of question-answer texts. Initially, the model is prompted provide detailed captions for IAD images by summarizing visual cues and textual prior knowledge. Based on these image-caption pairs, the model then generates questions across different subtasks according predefined question definitions and examples, simultancously creating multiple-choice questions with several distractor options. Finally, manual verification is conducted filter out low-quality VQA pairs, resulting in high-quality VQA data for the IAD.
Figure 3:
needs to cover multiple scenarios of IAD. We achieved data diversity by collecting and sampling from four different IAD datasets with distinct focuses, resulting in over 38 product categories and 244 defect types, as detailed in Table [ Among these, MVTec AD (Bergmann et al.) 2019) is one of the most prominent datasets for IAD, encompassing multiple categories of objects and textures, where we use finer annotations from Defect Spectrum (Yang et al. (023). MVTTec LOCO AD (Bergmann et al., (022) focuses on logical anomalies, thereby testing the model's 's understanding of logical-level anomalies. The VisA (Zou et al. (0222) dataset includes multiple instances and complex examples, reflecting more intricate IAD scenarios. The GoodsAD (Zhang et al.) 2024a) dataset primarily consists of industrial goods, and due to the varying appearances of finished products from different brands, significantly expands the number of object categories.
3.2 QUESTION DEFINITION
To evaluate whether MLLMs can fulfill the role of an industrial quality inspector, it necessary to test wide range of capabilities. On the production line, quality inspector must not only identify defective samples but also classify and grade defects, analyze causes, and diagnose faults. Therefore, in addition to basic anomaly detection, we have designed four anomaly-related subtasks and two object-related subtasks, as detailed below:
Asking whether sample has defects. These ques- tions are binary classification problems that test the ability to judge anomalies in samples. Defect Classification: Asking about the type defect. verifies the model's basis for anomaly discrimination while also testing its understanding of industrial defect categories.
Anomaly Discrimination/Detecle
Defect Localization: The model needs to specify the exact location of the defect, further verifying the model's basis for anomaly detection. Since most MLLMs cannot directly out- put masks, we standardize the testing by using textual descriptions to indicate the location.
Defect Description: Describing the appearance characteristics of the defect. It simulates the steps of collecting information on the size, color, and other attributes of defects in actual production. This information can be used to determine the defect's 's grade according to production standards and infer whether the defect related to equipment malfunction.
Defect Analysis: This involves analyzing the potential impact of the defect on the product, which is used to further determine the defect's severity level.
Object Classification: Categorization of industrial products. An understanding of indus- trial products aids in recognizing normal characteristics and identifying anomalies.

---

Published as a conference paper at ICLR 2025
IAn Datrorts
Multimodal Large Language Models
Multimedal Large Language Model
Krwndedge
Yake Tapuc
lapus
Andtermett
Queyy tmage
Norrel cimep (Loba)
Fighight
Thos afterti whert
Qeap Sxage
Nornal Sarzic
(1oskor)
Falleing denkk Peeeider ame adfurtedd slje furminsde
Expen
Doseppton Do
(A)
Agent
Figure Illustration of two proposed boost methods of MLLMS in MMAD.
This involves questioning the composition, position, appearance, and function of the object, aiming to assess the detailed understanding of the specific product.
Object Analysis:
Some examples are shown the Figure To better evaluate the outputs of MLLMs, we use multiple-choice questions, a method proven effective in previous research (Li et al., 2024a Liu al., 2023b). To avoid biases inherent in language models, we randomize the options in our ques- tions and use text matching to select the closest option when the model cannot correctly follow instructions and output the answer's letter.
3.3 DATA GENERATION
Due to the lack of semantic annotation in open-source IAD datasets, the currently collected data cannot be directly used to evaluate MLLMs. Therefore, we designed a novel pipeline to generate evaluation questions for each IAD image. As shown in Figure 3 our process leverages the text generation capabilities of GPT-4V (Achiam et al., 2023), combined with rule-based program out- puts, language prompts, and human filtering to ensure the reliability of the generated content. First, we generate rich textual descriptions for each image. Since GPT-4V is primarily trained on natural scenes and may struggle with industrial quality inspection, prompts. In the visual prompts, we highlight the ground truth mask in red on the original image to make the model aware of the defects. Additionally, we retrieve the most similar normal image to serve as comparison template, using a similarity metric combining the SSIM score (Wang et al.) 2004) and the Bhattacharyya distance (Bhattacharyyal 1943) of color histograms. The language prompts include object and defect category labels and textual descriptions of defect positions within the image, utilizing the nine-grid division proposed by Gu et al. (2024). To ensure diversity, we designed instructions for caption generation with variations in the prior knowledge provided, pre- venting the captions from merely copying the supplied text. Once we have the captions for each image, we generate questions, options, and answers based on predefined subtasks. Unlike natural images, each LAD image corresponds to multiple questions, which are generated simultaneously and then filtered through manual verification. This verification process involved 26 individuals and took over 200 working hours.
provide additional visual and textual
3.4 BoOST METHONS
Relrieval-Augmented Generation (RAG). RAG is method that combines information retrieval and generation enhance the performance of language models, particularly for tasks requiring external knowledge (Zhao et al. 2024). Knowledge related to IAD is often specialized and rarely encountered during the training of Multimodal Language Models (MLLMs). Therefore, we propose a RAGG method tailored for IAD, as illustrated in Figure 4 a). Experts, with the assistance of large language models, first summarize the existing IAD datasets. For each category, they summarize the characteristics of normal samples and the features of each possible anomaly. The domain knowledge summarized from all datasets forms a retrieval database. During testing, the query image is used to retrieve the relevant category knowledge, which is then incorporated into the text prompts.

---

Published as a conference paper at ICLR 2025
Table Performance comparison of both proprietary and open-source MLLMs in MMAD with the standard I-shot setting. Anomaly Discrimination uses the average accuracy of normal and abnormal categories. (*For the methods not supporting multi-image input, the O-shot result reported.)
Anomaly
Defect
Object
Model
Scale
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
25.00
25.00
28.57
Human (expert)
95.24
75.00
92.31
83.33
94.20
86. 11
80,37
86.65
(ondinary)
71.25
Claude-3 3.5-sonnet
60,14
60.14
48.81
67.13
79.11
85.19
79.83
68.36
Gemini- 1-5-hh
66.53
79.71
Gemini 1ommini-1.5.pe
68.63
60.12
58.56
70.38
82.46
89.20
82.25
73.09
GPT-4o-mini
79.74
GPT-4o
68.63
65.80
55.62
73.21
83.41
94.98
82.80
74.92
AnomalyGPT
65.57
27.49
27.97
36.86
32.11
84
35.82
36.52
Owen-V1--COn VT-CCht
7B
53.65
31.33
28.62
41.66 66
63.99
74.46
67
51.66
LLaVA-1155
69.53
Cambrian-1 1
8B
55.60
32.53
35.39
43.46
49,
78, 18.15
67.
51.64
SPHINX
LLaVA LLaVA-NEXT -itellneeeeeeee
7B
57.64
33.79
47.72
$1.84
67.93
81.39
74,91
59,32
LntemLM-XComme Mo--Cooops----11
74.34
LLaVA LLaVA-One Vision
Visiin
7B
51.77
46.13
41.85
62.19
69.73
90.31
80.93
63.27
MiniCPM-V2.6
66.25
InterVL22
8B
59.97
43.85
47.91
$7.60
78.10
74. 18
80.37
63.14
1LvVA-15
13B
49.96
38.78
46.17
58.17
73.09
73.62
70.98
58.68
LLaVA-NeXT
34B
57.92
48.79
52.87
71.34
80.28
81. 12
77.80
67.16
InterV11.2
76B
68.25
54.22
56.66
66.30
80.47
86.40
82.92
70.75
Expert Agent. Tool agent is independent module within large model, responsible for specific functions (Xi et al. 2023). In the evaluation of MMAD, exploring whether expert models can enhance MLLMs an intriguing topic. We designed a simple method to investigate its feasibility. As shown in Figure bb) we treat the IAD model as visual expert model. Since the output anomaly map is difficult understand by MI.LMs, we visualize it and then input it into the MILMMs. We use SOTA models in IAD, such as AnomalyCLIP and PatchCore, experts and test the differences among 3 visualization methods. Additionally, we applied a special expert, ground truth, which directly uses the mask as the output of the expert model to verify the upper limit.
4 EXPERIMENTS
4.1 EVALUATION SETTINGS
We default a 1-shot setting, where, in addition the query image, we randomly provide a normal image from the dataset and inform the model that it can use this image as template. The template image helps the model understand the normal state of objects, which is essential for anomaly de- tection. We also provide O-shot and few-shot settings for further comparison. Besides, randomly providing normal images may lead to misunderstandings in some object categories due to multiple normal states existing. We introduce 1-shott setting, which retrieves the most similar image from the normal data as a template. This setting tests the model's ability to perform pairwise comparisons of industrial images.
For commercial models, we conducted tests using APIs, specifically versions claude-3-5-sonnee 20241022, gemini-1.5-flahh-e 05-13. For open-source models, we adapted and tested AnomalyGPT (Gu et al.] 2024). Qwen- VL-Chat (Bai al., 2023), LLaVA-1.5 (Liu et al. 2023a). Cambrian- (Tong al. 2024a), SPHINX (Lin et al. (022). LLaVA-NeXT (Liu et al. 2024a), LLaVA-NEXTTImternnn Interllave Li et al. (2024b), InternI M-ammmMMMMmmmpppi LM..MM--ooee2---VV (Dong et al. 2024), MiniCPM-V2.6 (Yao et al., 2024). and VL.2 (Chen et al., 2023). We endeavored to maintain consistency with the default hyper- parameters and prompt methods provided. For models exceeding 20B parameters, we employed appropriate quantization methods. Each image's multiple VQA tasks were tested independently, and caches were cleared after each test. Some models only support single-image input by default. For 1.555- and SPHINX, we only tested single-image input (i.e., 0-shot) performance. We will randomize the letters and order of the options and the accuracy of responses metric. If the model does
gemini-1.5-pro-000 gpt-4o-miin-22024-07-11 and gpt-4o-2024-
modified the framework to support multi-image input, while for Cambrian-1

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison with and without template images. (l-shott indicates using retrieved image as the template to provide more helpful visual information.)
Model
Scale Sctting
Anomaly
Defect
Average
Discrimination
Classification
Localization
Deseription
Analysis
0-shot
58.43
49,93
53.11
63.07
82.83
68.58
I-short
62.22 (+3.79)
[52.29 (+2.37)
50.99 (-2. 12) 65.52 (+2.45) 83.41 (+0.59)
69.59 (+1.0)
LLaVA-1.5
7B
0-sbot
50.79
36.94
35,94
50.86
70:45
54.91
1-shar*
50.99 (+0.21)
(+0.20)
35.63 (.0.3))
50.75
(.0.5 70.01
(.0.4))
54.66 (:0.25)
LLaVA-1.5
13B
sbot
49.98
38.77
48.64
$7.65
72.47
58.58
I-sbor*
49,97 (-0.01)
39.44 (+0.67)
45.22 (-3.42)
58.64
(+0.99)
73.28
(40.81)
58.55 (-0.02)
LeXV
34B
O-sbot
60.25
51.57
55.49
71.62
80.43
68.45
1-sbot
56.80 (-3.45)
48.877 (-2.70)
52.80 (-2.69)
70.82
(-0.79)
80.00
(-0.43)
67.16 (-1.28)
LLaVA-NEXT.intertewee
7B
O-shot
58.39
48.98
51.51
66.64
60.04
I-sbor*
57.71 (-0.68)
35.87 (-1.11)
48.
(-0.86)
52.40
(+0.89)
67.88
(+1.24)
59.82 (-0.22)
Internl. M M-XComposer22VV V1.
7B
0-sbot
58.33
43.10
54.56
$7.84
75.30
62.78
I-sbot'
55.87 (-2.46)
42.76 (-0.34)
49.68 (4.88)
(+0.20)
76.52 (+1.233
61.99 (-0.79)
LntemVL2
40B
0-shot
66.37
48.54
53,39
64.05
79.01
69.23
I-shor*
67.69 (+).32)
551.15 (+2.61)
55.81
(+2.42)
66.81
(+2.76)
79.83 (+0.82)
70.71 (+1.49)
0-sbot
64.30
51.19
54.20
63.46
79.92
69.41
Lnter
768
I I-shot*
69.23 (+4.93) 54.69 (+3.50) 57.20 (+3.00) 66.75 (+3.29) 80.46 (+0.54)
71.31 (+190)
provide any option, we will automatically match the closest option the output the answer. It is worth noting that, in the anomaly discrimination subtask, due to the imbalance distribution, we will separately calculate the accuracy of normal and abnormal samples and then use their mean as the final accuracy.
4.2 EXPERIMENTAL RESULTS
We compare the performance of over a dozen models, including commercial APIs, interleaved MLLMs, industrial MLLMs, and vision-centric MLLMs, as shown in Table 2 All models out- perform the random baseline. The open-source models perform the best, with the average accuracy of the GPT-4o and Gemini-1.5-pp models reaching 74.9% and 73%, respectively. However, their cost-efficient counterparts, GPT-4o-min and Gemini 1.5--i-- only achieved 66.3% and 68. 9%, re- spectively, falling short of the best open-source model, InternVL.2-76B, which achieved 70.8%. AnomalyGPT performs poorly overall, primarily due to its training on the IAD task in a fixed guestion-and-anse format, leading to severe overfitting issues. demonstrates decent perfor- mance in anomaly discrimination because we specifically adapted the question format to suit its training. Similarly, the vision-centric MLLMs, Cambrian-1 and SPHINX, do not exhibit superior performance on the fine- tre-grriied visual tasks of MMAD, likely due to their foundational language models not being advanced enough. Among the general open-source MLLMs, earlier models like Qwen- VL-Chat and VLa underperform compared to newer models like VaVA-OneeVision and MiniCPM-V2 V2.6, indicating that advancements in general capabilities benefit performance on IAD tasks. MMAD uses default 1-shot format, providing a normal image for comparison with the test image. Thus, multi-image understanding, especially image comparison, is crucial, while LLaVA-NEXTT-mnerleeve trained for this, does not perform outstandingly. LLaVA-NeXT-344 and InternVL22-765 due to their larger scales, achieve the top two performances among open-source models, highlighting the importance of model size.
Human evaluation. We conduct preliminary human evaluation using 177 examples randomly sampled from the entire benchmark. Eight evaluators were divided into two groups: 3 industrial anomaly detection researchers as experts and 5 ordinary participants. As shown in Table 2 ordi- nary outperform the best models by 4%, and experts by 12%. Humans excel in anomaly and defect-related tasks, highlighting MMAD's challenges and MLLMs' limitations in anomaly detec- tion. However, in object-related VQA, GPT-4o surpasses humans due to its broader knowledge base, with examples provided in the appendix B
8

---

Published as a conference paper at ICLR 2025
Detection
Anomaly Detect
Defoct Classiicaioon
Ocfect Opssifictionn
85
50
pescrittion
Analysi
Analyss
Ciject Classiicatioon
Osject Clssilication
Accuracy
Cbject Analysts
Average
Accuracy
75
Object Analysis
Average
60
70
60
10
50
268
728
s-shot
2ishot
4
8-shot
Modal Scale
Normal Sample Count
Figure 5: Left: Scaling law of model size in MMAD evaluation. We use the Intern series as examples. Right: The performance trends of Intern with varying counts normal samples.
4.3 FURTHER ANALYSES
Can MLLMs effectively utilize template normal images? Anomaly detection often benefits from template images to understand normal patterns. To examine whether MLLMs can utilize tem- plates, conducted comparative experiment. As shown in Table we evaluated models in O-shot and I-shot settings. In O-shot, no template image is provided, while in 1-shot*, the closest normal image the image is used a template for comparison. Results indicate that most models fail to leverage template information, often resulting in performance drops. Due to cost constraints, we tested only the cost-effective Gemini-1.5 1.5--lash among commercial models, which showed notable gains, suggesting open-source models excel in contextual image understanding. Among open-source models, only the larger-scale Intern VI.2 effectively utilized template information.
How significant is the impact of model scale performance? In the main experiment results, we observed that larger MILMs generally exhibit better performance. We evaluate the scaling law using the InternV1.2 series models, which utilize the performing open-source models. As shown in the left panel of Figure performance significantly improves with increasing model size, with the average accuracy difference between the largest and smallest models reaching 23.37%. Specifically, the classification performance of industrial objects improves the most with increasing size, and there are notable enhancements in several subtasks related to anomalies and defects. Although the performance improvement trend slows as the model size increases, further enlarging the model remains a promising option, albeit one that must be balanced against cost considerations.
training data and are among the best-
Can increasing the number of images further enhance performance? For traditional few-shot IAD models, increasing the number of normal samples significantly enhances the performance of anomaly detection and localization. Therefore, we investigate whether MLLMs can leverage ad- ditional normal images. Experiments with InterVVL2-76B (Figure [5] right) show performance im- proves from ()- to I-shot, but further increases yield minimal gains (only slight improvements in anomaly/defect subtasks up to 4-shot). Beyond 8-shot, the performance in some subtasks even be- gins to decline, possibly due to information overload caused by too many input images, leading to confusion in the MLLMs. After providing multiple normal samples, the model needs to first sum- marize the normal characteristics and then compare them with the samples under inspection, while such tasks rarely appear in current MLLM instruction tuning data. This highlights the need for future research focusing industrial applications' unique requirements.
4.4 EXPLORATION
Input Domain Knowledge to MLLMs. One significant challenge identified in our evaluation is the lack of industrial knowledge in MILMs for specific tasks. During the training, MLLMs rarely encounter knowledge related industrial quality inspection, while different products correspond to different specific knowledge. In practical applications. senior experts typically train workers. We use RAG to provide domain knowledge guidance to ML.MM to simulate this process. As shown in Table

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison different MLLMS with and without RAG.
Anomaly
Defect
Object
Model
RAG
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
LLaVA-NeXT-34B
/
56.7 (-1.199
68.22 (-1944) $7.36 57.36 (14.99)
52.87
13.12
71.34
82 (41 96)
91.78 10,66) 81.35 (+3.55)
72,97 (+5.81)
67.16
VL2-222
/
67.96
(+0.68)
67.32 (-17.55)
49.77
49.44
(+1.77)
70.84 (48.22)
62.62
80.92
26)
93.81 (+5.64)
88.16
83.31 (+1.92)
81.39
68.61
V1224V1.2-40
70.01 (+5.56)
64,45
70.09 (119,33) 56.89
50.57
53.42
(+3.47)
73.29 (+7.12) 83.26 (+3.70)
66.17
79.56
(+5.85)
90.65
84.41 (+2.05)
82.36
76.35 (+6.75)
69.59
VL2-766
/
68.25
54,22
56.66
66.30
80.47
86.40
82.92
70.75
(-1.57)
70.95 (-16.73)
60.57 (+391)
(19.00)
82.71 (42.24)
91.21 (45.31)
85.29 (42.36)
76.18 (45.43)
(4. 30
31.19
54.20
63.46
79.92
99:34 34
83.48
69.41
(O-shor)
/
63.46
(1.30)))
59.11 (1492) 74.44 (110977) 82.47 (12.55)
91.78 (14.44 84.83 (.333)
75.37 (15.96)
the inclusion of RAG significantly improves performance on MMAD across multiple models, with the most notable improvements in anomaly classification and object classification. This is primarily because domain knowledge contains extensive category information. The performance in defect localization also improves, indicating that models can leverage textual knowledge to enhance their perception of images. Notably, the performance in anomaly discrimination shows substantial improvement for Intern VL2-40B, surpassing all other models in this metric.
4
Model Collaboration. Given that MLLMs have been specifically trained on IAD data, their anomaly detection capabilities are relatively weak. Detecting anomalies is not particularly challenging for existing IAD models. Therefore, we are in- terested in whether MLLMs can per- form answering with the help of the visual outputs of expert models. As shown in Table [ tested three agents: using AnomalyCLIP, Patch- Core, and directly using ground truth (GT) as experts, while also exper- imenting with various visualization methods. The two IAD models only improved defect localization but resulted in declines in performance for anomaly discrimination, defect classification, and defect description. In contrast, using GT improved all performance met- rics, with the enhancement in defect localization far surpassing that of the two expert models. This indicates that the outputs of the current expert models are not sufficiently accurate to be used by MLLMs. Additionally, expert models will output anomaly maps for normal images, leading to overkill in anomaly discrimination. Among the various visualization methods, simple bounding box (bbox) to highlight the anomaly region proved to be the most effective approach overall. Di- rectly converting the GT mask into textual descriptions of defect location and size resulted in better performance in both discrinination and localization.
Figure Performance comparison of different agent and visualization types. Based model is Intern VL.2-40B.
Expert Model
Visualization Type
Anomaly
Defect
Diserimination
Classification Localization Description
haseline
63.84
$1.58
$2.94
66.43
AnomalyCLIP AnomalyCLIP AnomalyCLJP
bbox
-5.83
-0.35
+2.66
contour
6640
-1.19
+2.52
-386
highlight
-4.11
-6.92
+7.44
-6.61
PatchCore
bbox
+1.00
PatchCore
contour
-12662
-0.02
-0.48
-2.0
PatchCore
highlight
-3.87
-7.23
45.01
-5.72
GT
bbox
+21.29
GT
contour
110.86
16.45
+20.44
+5.16
GT
highlight
+9.95
41.46
+16.59
+1,21
GF mask
-1.05
48.34
GT
text
+24.89
+1.36
+28.01
-0.23
5 CONCLUSION
In this work, we introduce MMAD, the first benchmark for MILMSs in the IAD field, aimed at ex- ploring the feasibility of using MLLMs for industrial quality inspection. Our evaluation of over ten SOTA MLLMs yielded less than optimistic results, revealing their weaknesses in industrial scenar- ios, particularly the lack of industrial knowledge and the ability to perform fine-grained comparisons across multiple images. Models may require extensive data for targeted improvement. Moreover, further investigations suggest that MLLMs have the potential to address some of these issues through additional enhancements. We hope that MMAD will inspire future research into improving the relevant capabilities of MILMs and promote the development of practical applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Tajeddine Benbarrad, Marouane Salhaoui, Soukaina Bakhat Kenitar, and Mounir Arioua. Intelligent machine vision model for defective product inspection based on machine learning. Journal of Sensor and Actuator Networks, 10(1):7, 2021.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. Beyond dents
and scratches: Logical constraints in unsupervised anomaly detection and localization. Interna- tional Journal of Computer Vision, 130(4):947-969, 2022.
Anil Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distribution. Bulletin of the Calcuuta Mathematical Society, 35:99-110, 1943.
Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. Towards generic anomaly detection and understanding: Large-scale visual-linguistie model (gpt-4v) takes the lead. arXiv preprint urXiv:2311.027883 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision- language models? In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL http::://ppeerevieeee www.ervviieii.eeeeeeeeeeeee forum? d=evP 9mxNNxJ
Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng. Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint 2408:22008.0366 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- tervvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. 14238, 2023.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth survey of chain of thought reasoning: Advances, frontiers and future. In The 62nd Annual Meeting of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand. August 11-16, 2024. Association for Computational Linguistics, 2024.
htt.t:tt:///// http///axvvvv//aa org/ abs/ 2009.0331 15402
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty Conference Neural Information Processing Systems, 2023. URL battps://openreviee wtwp:p/awwwi.eeeeeeeeoeeeeeeeeee www.oooonnoowwooooeoooo
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 9737-9746 2022.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li. Kai Chen, Conghui He. Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm -coopoo-xoomosse2:::: Mastering free-form text-image composition and comprehension in vanion-l large model. arXiv preprint arXiv:2401. 16420, 2024.
11

---

Published as a conference paper at ICLR 2025
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303 10130, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Mme: comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. 12223006613394,, 2023. URL. wwt.orrg//ooooo Corpus ID 259243928
https://api. sementicschon
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-moda llms in video analysis. arXiv preprint arXiv:2405.21077 2024.
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-langue languae models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1932-1940, 2024.
Xi Jiang, Jianlin Liu, Jinbao Wang, Qiang Nie, Kai Wu, Yong Liu, Chengjie Wang, and Feng Zheng. Softpatch: Unsupervised anomaly detection with noisy data. Advances in Neural Information Processing Systems, 15:35433-1554 1445, 2022a.
Xi Jiang, Guoyang Xie. Jinbao Wang. Yong Liu, Chengjie Wang, Feng Zheng, and Yaochu Jin. A survey of visual sensory anomaly detection. arXiv preprint arXiv:2202.070003 2022b.
Xi Jiang, Ying Chen, Qiang Nie, Jianlin Liu, Yong Liu, Chengjie Wang, and Feng Zheng. Toward multi-class anomaly detection: Exploring class-aware unified model against inter-class interfer- ence. arXiv preprint 243::24033.12113 2024a.
Yuqi Jiang, Xudong Lu, Qian Jin, Qi Sun, Hanming Wu, and Cheng Zhuo Fabgpt: An ef- ficient large multimodal model for complex wafer defect knowledge queries. arXiv preprint arXiv:2407.108100 2024b.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey, arXiv preprint 2rXiv:2405.11739 2024.
Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Compbench: A comparative reasoning benchmark for multi- modal llms. arXiv preprint arXiv:2407.16877 2024.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024a.
Feng Li, Renrui Zhang. Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-imterleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07880 2024b.
Jian Li and Weiheng Lu. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408. 08632, 2024.
Yuanze Li, Haolin Wang. Shihao Yuan, Ming Liu, Debin Zhao, Yiwen Guo, Chen Xu, Guangming Shi, and Wangmeng Zuo. Myriad: Large multimodal model by applying vision experts for indus- trial anomaly detection. arXiv preprint arXiv:2310. 19070. 2023.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wengi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint X3XX:23311005777 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
12

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next next: Improved reasoning, oct, and world knowledge, January 2024a. URL https://
1lava-vaz www.git.gg..... wt/blo///aaieeeeee 20tp:/224---000 01 30- llava-next/
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint vrXiv:2307.06281, 2023b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Zhonghe Ren, Fengzhou Fang, Ning Yan, and You Wu. State of the art in defect detection based on machine vision. International Journal of Precision Engineering and Manufacturing- Green Technology, 9()):661-691, 2022.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-Is A fully open, vision-centric exploration of multimodal llms. arXiv preprint urXiv:2406.1687 2024a.
Shenghang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms, 2024b.
Chengjie Wang, Haokun Zhu, Jinlong Peng, Yue Wang, Ran Yi, Yunsheng Wu, Lizhuang Ma, and Jiangning Zhang. M3dm-nr: Rgb-3d noisy-resistant industrial anomaly detection via multimodal denoising. arXiv preprint arXiv:2406.. 022663, 2024a.
Chengjie Wang. Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jiangning Zhang. Zhihao Gu, Shuguang Qian, Mingang Chen, and Lizhuang Ma. Real-iad: A real-world multi-view dataset for bench- marking versatile industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22883-22892, 2024b.
Chengjie Wang, Xi Jiang, Bin-Bin Gao, Zhenye Gan, Yong Liu, Feng Zheng, and Lizhuang Ma. Softpatch+: Fully unsupervised anomaly classification and segmentation. Pattern Recognition, 161:111295, 2025.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Mcasuring multimodal mathematical reasoning with math-vision dataset, arXiv preprint arXiv:2402.1 14804, 2024c.
Wenxuan Wang, Yihang Su, Jingyuan Huan, Jie Liu, Wenting Chen, Yudi Zhang, Cheng-Ye Li, Kao- Jung Chang, Xiaohan Xin, Linlin Shen, al. Asclepius: A spectrum evaluation benchmark for medical mudti-modal large language models. arXiv preprint arXiv:2402. / /217, 2024d.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He. Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.0788 2023.
13

---

Published as a conference paper at ICLR 2025
Xiaohao Xu, Yunkang Cao, Yongqi Chen, Weiming Shen, and Xiaonan Huang. Customizing visual- language foundation models for multi-modal anomaly detection and reasoning. arXiv preprint arXiv:2403.11083, 2024.
Shuai Yang, Zhifei Chen, Pengguang Chen, Xi Fang, Shu Liu, and Yingcong Chen. Defect spectrum: A granular look of large-scale defect datasets with rich semantics. arXiv preprint arXiv:2310.13316 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained reconstruc- tion embedding for surface anomaly detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8330-8339, 2021.
Jian Zhang, Runwei Ding, Miaoju Ban, and Linhui Dai. Pku-goodsad: supermarket goods dataset for unsupervised anomaly detection and segmentation. IEEE Robotics and Automation Letters, 2024a.
Jiangning Zhang, Xuhai Chen, Zhucun Xue, Yabiao Wang, Chengjie Wang, and Yong Liu. Explor- ing grounding potential of vga-oriented gpt-4v for zero-shot anomaly detection. arXiv preprint arXiv:2311.02612 2023a.
Jiangning Zhang, Haoyang He, Zhenye Gan, Qingdong He, Yuxuan Cai, Zhucun Xue, Yabiao Wang, Chengjie Wang, Lei Xie, and Yong Liu. Ader: comprehensive benchmark for multi-class visual anomaly detection. arXiv preprint arXiv:2400.033222. 2(3), 2024b.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023b.
Yiheng Zhang, Yunkang Cao, Xiaohao Xu, and Weiming Shen. Logicode: an llm-drivees framework for logical anomaly detection. arXiv preprint 2406:20000604888,,. 2024c.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. eerrivall-augment generation for ai-generated content: A survey, arXiv preprint arXhv:2402.19475 2024.
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310. 189661 2023.
Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learn- ing with few-shot sample prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 77826-17836, 2024.
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot -hfffthe-differeece self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision, pp. 392 408. Springer, 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSION OF SETTINGS
Vision Input
Vision Input
Text Imput
Lespector whether You docalé
Instruction:
compane
Image
Nocnal Sample
Query Image
Few Noraal Samplee
Query Image
irage kample,
Netrest
Saarch
Randam
Following the Simageeiia
Normal Set
Normal Set
okjez??
MLLM
(b) 1-shot+
(d) Few-shot (2, 4, 8)
Vision Input
any defect object??
Image Tag:
Following dimagoiin
Nonmal Sample
Query Image
Defect
Query
Random
Input
Sample 1
Sample
Image
Normal
Query Imuge
Yes.
(a) Default Setting (I-shot)
(c) O-shot
(e) In-Context Learning
Figure 7: Diagram of different settings. The query image shows an abnormal screw with a manipu- lated front, which is difficult to identify without compared sample.
Unlike other benchmarks, MMAD features various testing settings simulate different scenarios in industrial quality inspection for MLLMs. We use 1-shot as the default setting, as a single normal image theoretically provides significant information compared O-shot. This approach also used in our data annotation process. Increasing the number of normal images in few-shot settings may pose challenges for some MILMs, particularly those not fine tune-t for multi-image tasks. In indus- trial production lines, the number of available normal samples often exceeds the context length of MLLMs. such cases, similarity retrieval can filter out irrelevant images, leading us propose 1-shot setting, where the most informative image is selected images can sometimes be collected in production lines. To determine whether these abnormal im- ages aid MLLMs in making judgments, we introduce abnormal image prompts in the 1-shot setting and then query the test image, termed in-context learning, as illustrated in Figure
template. Additionally, abnormal
A.2 ANALYSIS WITH IN-CONTEXT LEARNING
As shown in T tested the performance under the in-context learning setting and compared it with other settings. We chose Intern VI. VI. 22240B as the base model due to its superior context-handling abilities demonstrated in previous tests. The experimental results indicate that adding abnormal im- ages in the in-context learning setting indeed helps MLLMs in anomaly detection, improving the performance from 63.84 33.84% to 66.47%. However, adding one more normal image does not signifi- cantly change this performance. In subsequent tas-tasks related to defect identification, in-context learning did not seem to provide any benefit. We believe this may be due MLLMs fully lever- aging the information from abnormal images. Marking defects in abnormal images and combining them with textual prompts might better assist MILMs. The 1-shot setting, on other hand, showed comprehensive improvements across sub-tasks, indicating that the quality of prompts is far more important than their quantity.
A.3 ANALYSIS OF CHAIN OF THOUGHT
Chain of Thought (CoT) is commonly used method to enhance the logical reasoning abilities of MLLMs (Chu et al. 2024). To investigate whether current MLLMs lack reasoning when address- ing MMAD problems, introduced straightforward CoT approach. The process involves three
15

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison of different settings in MMAD based on Intern VL.2-40B.
Setting
Anomaly
Defect Localization
Object
Average
Discrimination
Classification
Description
Analysis
Classification
Analysis
1-shot
63.84
51.58
$2.94
66.43
79.75
90.76
82.41
69.67
1-shot+
67.81
54.18
55.99
68.14
81.75
90.76
82.98
71.66
2-sbot
63.75
50.38
54.36
67.22
79.54
90.49
81.85
69.66
In-Context Leamiing
66.47
50.56
51.78
64.87
78.55
91.33
81.65
69.32
Table 6: Performance comparison of different MILMSs with and without CoT.
Anomaly
Defect
Object
Model
CoT
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
IntemVL2-40B
/
64.45
50.57
53.42
66.17
79.56
90.65
82.36
69.59
59.42
46.10
51.92
$7.66
79.80
80.55
85.29
65.82
68.25
54.22
56.66
66,30
80.47
86.40
82.92
70.75
IntemVL2-76B
/
68.18
54.61
58.64
68.89
79.95
90.51
85.25
72.29
steps: first, the model identifies objects in the image: second, it compares the differences between the template image and the query image; and finally, it determines whether the identified difference constitutes a defect in the object. We incorporated set of rules and adjusted the instructions cordingly, dividing the CoT responses into two stages. As shown in the Table 6 VLternVL2-766, the best-performing open-source MILM we tested, achieved 1.5% improvement in CoT-based performance. However, its anomaly detection accuracy, which is the most critical metric, showed no improvement. On the other hand, Intern V1.244V experienced performance decline after intro- ducing CoT, potentially due to insufficient stability in the language model's reasoning capabilities.
A.4 ANALYSIS OF VISION DISABLE
Some studies (Tong et al. 2024a Chen et al. 2024a) have proposed that determining whether a benchmark requires visual input be solved has been persistent challenge in vision-language research. To validate MMAD, we masked the visual components and compared the performance of MLLMs with and without visual input. The results, shown in the Table ? indicate that most subtasks in MMAD are highly dependent on the visual components. Performance significantly decreases when the visual input is removed.
A.5 DETAILS HUMAN SUPERVISION
Our textual data is generated by MILM, so its accuracy requires human supervision. In our designed process, human supervision is responsible for filtering the final model-generated multiple-chote questions. We first perform preliminary filtering through the program and then enable annotators to conduct an item-by-item review. total of 26 personnel were involved in the review process, all of whom are researchers in industrial inspection or computer vision and possess a certain level of expertise. However, it should be noted that the industrial inspection field is highly specialized, with significant differences between products. Our annotators' understanding may differ from the professionals understanding of where the original image data originates. We have developed a tool to enable annotators filter out problematic multiple-choice questions quickly. As illustrated in Figure 8 we provide the original annotation information through visual and textual means to help annotators accurately identify objects and defects. At the same time, annotators do not need to correct every issue; they simply mark the erroneous questions for exclusion, significantly improving efficiency.
A.6 DIVERSITY ANALYSIS OF DATASET
To systematically analyze the diversity of the dataset, we separately calculated the frequency of phrases appearing in questions and options, presenting them the form of word clouds in Figure 9 In the question text, "defect" appeared most frequently, followed by "object", reflecting that our benchmark is constructed for industrial inspection scenarios, focusing on seven sub-tasks. In addition to common words such "image", "type"" and "appearance", the questions also include
16

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparison of different MILMs with and without vision.
Anomaly
Object
Model
Vision
Discrimination
Classification
Loculizution
Description
Analysis
Clussification
Analysis
Average
Enable
59.97
43.85
47,91
57.60
78.10
74.18
80.37
63.14
I2tem
Disuble
49.71
32.28
41.38
52.20
75.34
41.94
57.13
50.00
IntemVL2-400
Enable
63.84
51.58
52.94
66.43
79.75
90.76
82.41
69.67
Disahle
24.43
36.35
46.39
68.05
37.29
55.75
Enable Disahle
68.25
54.22
56.66
66.30
80.47
86.40
82.92
70.75
IntemVL.2
49.93
25.92
28.54
43.88
70.41
35.89
57.53
44.59
Instruction
Image and Groundtruth
Questions
Cli these defeect in adjez??
Vision Prompt
Anewar
Foggn rocumaccs
Category: bottle
Text Prompt
CB Twwenlarrliie alg WWn thtccbeee
Filter Switches
broken,jke Jarge
Questions & Answers
37 0009
Tipi
CS. There
algpec What applasaatce diffect?
Pogeess
Dateat ftanga::
NVTac: D- 13I!
YEA Controller & Statistic
NVT Locc 1012 2413
Arawar
cuturmus beitory wort et
tas atheccel ocmata eacaior
Submission Butföns
Subrit Lamp 3o mod))
Figure 8: Illustration of human filtering tool. The tool comprises several functional areas. We use green bounding boxes to highlight defects as vision prompts, and we provide rough text prompts for object categories and defect types.
diverse expressions with lower frequencies. For example, expressions indicating position such as "relative position", "arrangement" and "defect located". In the text of the options, "Yes" and "No" are the standard answers for anomaly discrimination questions, thus appearing most frequently.
Ob
JoJerr
ffeeect eett
gotential ef ef
Top
Trigg nuts center
bottle
Nie
SHODt! packag
type
product..yy
kind defi
O
purpose
rightt
product
e
conta ner
5 de
Ni
Rackaging
Bot tom left
appearance
CaP
jbjec
Top center
BOTE Botton right Af fect Top RightBelugg IN
cbrrcteristie
Word Cloud Afl Questions
(b) Word Cloud of All Options
Figure 9: Word frequency statistics presented in the form of a word cloud, with separate statistics for questions and options.
17

---

Published as a conference paper at ICLR 2025
Beyond these two words, the diversity of options is evident, including specific descriptions of various positions or expressions of different types defects.
Table 8: Performance comparison of different MLLMs in Anomaly Discrimination/Detectton Tasks.
Model
Scale
Accuracy
Recall
Precision
F1
Human (expert)
95.24
94.25
98.89
96.43
Human (ordinary)
86.90
87.07
94.35
89.30
claude-3.5 1ouuee-3.5-s00 3.5-sonnet
60.14
30.87
76.75
Gemini-1.5 1.5-flash
58.58
67.41
72.40
Gemini-1.5-p 1.ppro
68.63
45.47
86.84
57.60
GPT-44ominni
64.33
65.47
73.04
68.67
GPT-4o
68.63
67.37
75.68
71.04
AnomalyGPT
7B
65.57
82.11
74.45
76.68
Qwen-VL-Chal VL-Chat
7B
53.65
43.95
65.39
47.28
LLaVA-1.5
7B
51.33
94.79
62.72
75.32
Cambrian-1
8B
55.60
22.28
74.10
31.85
SPHINX
7B
53.13
99.74
LLaVA-NEXT-mntereee
7B
57.64
16.58
90.83
InternL XComposer2-VV VL
7B
55.85
75.87
LLaVA-Oni Vision
7B
$1.77
78.19
MiniCPM-V2.6
57.31
34.38
70.98
45.31
VL.2
59.97
30.25
79.22
41.23
LLaVA-1.5
49.96
99.79
62.00
76.28
LLaVA-NeXT
34B
$7.92
46.27
69.98
$4.44
IntemVL2
76B
68.25
55.81
83.52
64.40
100
(export)
95
Human (ordinary)
LaW.-NExTiiterreewee-
Gemini-1 S-pro
Precision
85
IntemVL27768
80
InternVL
LLavA-nmVision778
InterLMMXCCmmge
2 21 sonnet
GPT-4o
75
Combbrinn
AnomslyGPT-78
GPI-4o-min mini
MiniCPMaV2 LAV
Gemini-1.5 5-fflshh
Qwen-VVCChtt77
65
LLa 10-1--11111 -3-8
Recall
Figure 10: Bubble chart with recall and precision axes in anomaly discrimination/detctiion task. The size the bubbles represents the number of parameters in the model. For commercial models or humans where the number of parameters is unknown, the bubbles are set the maximum size.
A.7 COMPREHENSIVE METRICS FOR ANOMALY DISCRTMINATIIO//DEEECCCO TASKS
To comprchensively evaluate the performance of various models in the anomaly discrimina- tion/detection task, we follow the traditional anomaly detection setup, treating the anomaly class as the positive class and the normal class as the negative class. We measured recall, precision, and Fl-score. As shown in Table S by analyzing the recall and precision metrics, we can identify some reasons for poor accuracy performance in certain models. For instance, both SPHINX and LLaVA- On Vision have recall rates below 10%, indicating that these models frequently misclassify anomaly samples as normal, leading to high rate of missed detections. On the other hand, LLa ALaVA-1.5 has a high recall but low precision, suggesting a high rate of false positives. Humans, however, outper- form MI.L MLLMMs across all metries, with human experts achieving over 94% and ordinary individuals
18

---

Published as a conference paper at ICLR 2025
achieving over 87%. This disparity is visualized in the bubble chart, shown in Figure 10 where there is a sigmificant gap between the bubbles representing humans and those representing various MLLMs, and considerable differences among the MLLMs themselves. Additionally, it can be ob- served that the model AnomalyGPT, which is specifically trained for anomaly detection, performs better than most models but still suffers from significant false positive issue.
B CASE STUDY
Hit:
You are an industrial inspector who checks products by images. You judge whether there is a defect in the query image and answer the questions about it. Answer with the option's letter from the given choices directly!
The first image is the normal sample, which can be used a template to compare. The last image is the query image.
Question:
Should the query picture be classified as normal or abnormal? Abnormal
Normal
Give short analysis and then answer letter:
< only included in qualitative analysis
Intern LL44BB:
The query picture shows three objects with copper-colored spheres on top, which typical for normal product. They appear be some kind of artificial experimental items rather than a standard product. Therefore, the query picture should be classified as abnormal.
Answer: A. Abnormal.
GPT-4o:
Comparing the query image the normal template, the green section in the query image shows visible crack separation in its structure, which is not present in the normal image This defect indicates that the product does not meet the standard criteria. Answer:
Analysis:
Figure 11: A case of question-and aawtiiio--nd--nssseee result in qualitative analysis. Since MMAD is test- ing with multiple-choice questions, we cannot directly analyze the answers. Instead, we use the chain-of-thought approach to encourage the model to provide its own analysis. In this example, the Intern VI.-40B model did not notice the damage to the cable, yet it still chose the correct answer from the options. In comparison, GPT-4o can output proper analysis and letter. Subsequent issues such as anomaly classification, localization, and description can further test whether the model's 's cor- rect answers are coincidental. Therefore, comprehensive evaluation, rather than simple anomaly detection, is essential.
19

---

Published as a conference paper at ICLR 2025
Hit:
You are an industrial inspector who checks products by images. You should judge whether there is a defect in the query image and answer the questions about Answer with the option's letter from the given choices directly!
The first image is the normal sample, which can be used as template to compare. The last image is the query image.
BB>
I
Question:
There defect in the object. What is the type of the defect?
A: Incorrect labeling
B: Structural deformation
C: Missing component
D: Discoloration
Give a short analysis and then answer a letter. < only included qualitative analysis
GPT-4o:
Analysis: Comparing the query image to the normal sample, the query image is missing one washer. The normal sample has two bolts, two nuts, and two washers, whereas the query image has only one washer.
Answer: C: Missing component
Human:
Analysis:
It is easy to see that the end of the long screw in the inspected image has been stained green.
Answer: D: Discoloration
A case of question-and- result in qualitative analysis. A human quickly notice the defect in the query image, while the model focuses on the number of components. This illusion tends to occur when the defect is minor, and the object's composition is complex.
Figure 12:
20

---

Published as a conference paper at ICLR 2025
Hit:
You are an industrial inspector who checks products by images. You should judge whether there is a defect in the query image and answer the questions about
Answer with the option's letter from the given choices directly!
BAT
Question:
What is the purpose function of this electronic circuit board?
A: Audio processing unit
B: Power supply for electronic devices
C: Lighting control system
D: Wireless communication module
Give short analysis and then answer letter. <- only included in qualitative analysis
GPT-4o:
Analysis: This is a TP4056 module, commonly used as a lithium battery charger circuit. It features a micro-USB input (IN+) and battery connection points (BAT and BAT-). The presence of components like the TC4056A IC and the layout indicates its in charging and regulating power supply for lithium-ion batteries.
Answer: B
Human:
I don't 't know. looks like all the options are possible.
X
case of guestion-and-anseee result in qualitative analysis. For object-related ques- tions, GPT-4o possesses extensive knowledge and can analyze object information based on subtle clues. However, most humans do not have such comprehensive knowledge. For instance, non- specialists may not recognize BAT+ and BAT-, let alone the TC4056A IC chip.
Figure 13:
21

---

Published as a conference paper at ICLR 2025
Image Information:
object category: "cable", ;
T
defect category: "a combination of multiple defects ; defect localization: "center of the inage"
Template Image
Captioned Image
RGB mask
Prior Knowledge
Caption:
This close-up image of cross-section of cable. The cable consists three separate conductive wires, each insulated different colored sheath: green with yellow stripe. blue, and brown. The template normal image provided:
encased gray onter sheath. Based on comparison with
There appear multiple defects present object.
The green and yellow wire exhibits a defect in the form of bent strands. These are visible at the approximate top center of the image, where the copper strands stick out at an irregular angle compared to the undisturbed arrangement seen in the normal image. Another anomaly is present in the brown wire, where there seems to be damage to the insulation. This defect located at the bottom center of the image, appearing deviation from the uniformly round shape of the corresponding wire in the normal sample.
These anomalies could potentially affect the performance and safety of the cuble.
Figure 14:
A good case image caption. We prompt the Al assistant that the image contains multiple defects and provide corresponding annotations. The Al assistant distinguishes different defects in the caption, describing their appearance and location individually.
Image Information:
object category:
"screw bug" "
bb
defect category:
"screw too long" "
defect localization:
"center of the image' Prior Knowledge
Template Image
Captioned Image
RGB
Caption:
The query image depicts sealed plastic bag containing hardware Inside the bag, are two bolts, four mits, and two washers. The bolts appear metallic with threads for fastening, and
and waskers accompany them as components for securing or the material being fastened.
distributing the
Upon comparing the query image with the template normal image, there a noticeable anomuly related to the length of the bolts. In the query image, one bolt, positioned in the bottom part of the bag, appears longer than the corresponding holt in the normal image. Specifically, the defect is observed in the central bolt, which extends from the bottom left to the bottom center region of the bag. This anomaly disrupts the uniformity expected from suandadd hardware where bolts should be of matching lengths for consistent application in construction ussembly tasks. The appearance of the anomaly is characterized by visibly extended thread area beyond what obwerved the normal comparison inage.
Figure 15: A bad case of image caption. Although we informed the AI assistant that screw is too long, it failed to analyze that one screw should be long and the other short. This error primarily due to the insufficient number of template images, and such misunderstandings will be corrected during manual filtering.
22

---

Published as a conference paper at ICLR 2025
Sample Images:
||II
Domain Knowledge:
<Normal Characteristics>
Description: The ultrasonic sensor modile, typically characterized by its rectangular blue printed circuit board (PCB), features two round ultrasonic transducers positioned symmetrically, which function as the transmitter and receiver. The mochle includes central electromic component likely responsible for signal processing. survaadedd by clear white labeling indicating each pin's function, such VCC (power supply). Trig (trigger), Echo (echo signal), and GND (ground). Upon inspection, there are no visible defects or anomalies present; components, inchuding solder points, appear well-formed and intact. The noise fentures, which may inchude slight variations
minor color differences, are rypical manufacturel electronic components and not indicate defects. overall condition conveys a standard representation a fully functional ultrusonic sensor good working order. free from any damage, discoloration, misalignment.
Figure 16: A of domain knowledge. Domain knowledge includes descriptions of the character- istics of normal samples. In this example, the text describes the components and appearance of this category of PCBs, as well as some common types of noise that may be present.
Sample Images:
Domain Knowledger
<Logical Anomalies>
Description: The observed defects primarily exhibit inconsistencies related to product labeling, fill levels, or the representation of contents within the juice bottles. Common visual features for distinguishing these logical anomalies include:
Label Misaligument: Many images showed labels where graphical elements (such as fruit toons) were misplaced or not centered, deviating from their expected positions. This often causes confusion regarding brauding and product identity.
Missing Labels: Several instances indicated the absence of labels essential for branding, ingredient information, or flavor ideutification. Missing labels create visual gaps on the product, which can mislead consumers about the productss is contents.
Fill Level Deviations: Anomalies relating to insufficient or excessive juice levels compared standard expectations were found. These incidents presented notable visual differences where expected liquid content was overflown, underfilled, or completely absent. Label Content Mismatch: Some products contained labels that represented a different fruit than actual juice color, indicating a logical misrepresentation of product's identity. This ereated strong disconnect between the imagery the liquid contents inside the bottle
All these feutures are significant determining presence and type of logical unomalies across evaluated juice bottle batch.
Figure 17:
case of domain knowledge. Domain knowledge includes descriptions of defects. In this example, the text outlines four major logical anomalies that may occur in these juice bottles. The actual dataset contains some rare logical anomalies that are not described, such as the incorrect icon placement in the first image. We do not expect domain knowledge to cover all possible anomalies; its primary role is to provide MI.LMs with an understanding
types of anomalies.
23


Reviews:
[{"review_id": "qi1zzzuRTE", "reviewer": "Reviewer_nkBV", "summary": "The authors introduce MMAD, a benchmark specifically designed for anomaly detection in industrial settings to evaluate the capabilities of multimodal large language models (MLLMs). The benchmark encompasses multiple subtasks, wirth 8,336 images and 39,672 questions. Evaluation of current state-of-the-art models demonstrates that performance levels (approximately 70% accuracy) are not yet acceptable by industry standards, highlighting significant gaps in current models’ capabilities. To address these shortcomings, the authors propose the use of Retrieval Augmented Generation for domain-specific knowledge and visual “Expert Agents” to enhance MLLMs’ perception of anomalies. MMAD highlights the current limitations of MLLMs in fine-grained industrial knowledge and provides a benchmark for evaluating future methods in this domain.", "questions": "See Weaknesses Above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Benchmark: MMAD is the first benchmark of its kind in industrial anomaly detection, specifically tailored for evaluating MLLMs. It covers a wide range of tasks related to industrial anomaly detection.\n\nDiverse subtasks and Categorization: The benchmark includes numerous subtasks and defect types, ensuring fine-grained evaluations across different aspects of anomaly detection. \n\nThe paper is well-written and easy to follow.", "weaknesses": "**Lack of Details on Human Supervision:** Given that this is a dataset-centric paper, there is a lack of detailed information about the exact processes used for dataset filtering and the criteria applied during manual verification. More transparency in the data curation process would strengthen the credibility and reproducibility of the benchmark.\n\n**Insufficient Diversity Analysis:** While the dataset is generated from multiple seed datasets and includes diverse subtasks, there is no systematic analysis of the dataset’s diversity (e.g., semantic similarity measures). Furthermore, there is a lack of analysis on the diversity of the questions generated by the LLM. It is unclear whether the questions follow only a few fixed templates or truly represent a wide range of scenarios, which could limit the benchmark’s effectiveness in evaluating models comprehensively.\n\n**Lack of Comprehensive Analysis:** The paper generally lacks in-depth analyses, such as qualitative assessments, error analyses, or ablation studies, which are crucial for evaluating the quality and limitations of both the dataset and the proposed methods. For instance, although the authors note that models perform poorly on defect-related questions, they do not provide detailed insights into the specific types of errors or misconceptions the models exhibit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6jbty3bNv", "reviewer": "Reviewer_69ie", "summary": "This paper introduces the first benchmark for industrial anomaly detection with multimodal LLMs. It defines 7 subtasks for industrial anomaly detection and creates a novel pipeline for generating semantic annotations of visual anomaly detection data. From the experiments, the best performing model GPT-4o reaches 74.9% accuracy, but it is still way below the industry needs and demonstrates the potential for future research.", "questions": "1. It is interesting to see how RAG improves the performance with domain knowledge, as anomaly detection is a multimodal task, not a direct text retrieval task. How could prompting with domain knowledge affect the model performance, as context windows are now getting larger?\n2. It is interesting to use an expert model to assist MLLMs - are there other ways of providing the expert information beyond the visualizations listed such as highlight, contour and BBox? It is intuitive to see how BBox can massively improve localization, but wonder if other direct text or other modal information can improve the performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, this paper is clear, well-motivated and provides a new benchmark for industrial anomaly detection with multimodal LLMs. The tasks selected from the benchmark are comprehensive for identifying industrial anomaly, including defect detection, classification, localization, description, analysis, etc. They also contribute to the diversity and novelty of benchmark tasks. In addition, the paper highlights a few explorational techniques that could improve model performance, such as RAG on domain knowledge, few shots learning and model collaboration. Those explorations are helpful for industrial practitioners using the benchmark in production.", "weaknesses": "The paper could benefit from discussions with regards to the following points:\n1. This paper fills a gap in the specific domain of industrial anomaly detection by providing a dedicated Multimodal LLM benchmark, and the benchmark has clear value in real-life industrial quality inspections. However, it is not clear how the industrial anomaly detection tasks are fundamentally different from other general visual Q&A tasks, and how this benchmark creates significant contribution to the general ML community beyond its specific application in industrial anomaly detection. There are many other venues that are great fit for industry-specific benchmarks, such as application-focused workshops and NeurIPS dataset and benchmark track.\n2. The benchmark constructs the 7 key subtasks in the form of multiple choices. This format could be limiting or misguiding, as it is dependent on how the choices are created, the number of choices provided to the model, and the model could provide the correct answer without truly understanding the answer. For instance, in the defect classification, location, and analysis tasks, accurate open-text answers and locations are much harder to generate and evaluate than multiple choices, but those formats are directly useful in production settings. \n3. In addition, the benchmark only reports on accuracy based on multiple choices, but fails to report recall, F1 score, etc. which would be important in the anomaly detection setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVFVdPbYta", "reviewer": "Reviewer_YdC1", "summary": "The authors introduce MMAD—the first large-scale industrial anomaly detection dataset, specifically targeted at evaluating multimodal large language models. The dataset contains seven main subtasks: anomaly discrimination, defect classification, object classification, defect localization, object analysis, defect description, and defect analysis, which are all highly relevant to the IAD problem. The authors evaluate multiple out-of-the-box MLLMs on their dataset, as well as using RAG and expert model-based improvements. Finally, the authors present a very thorough evaluation of model performance by analyzing whether template images help in anomaly detection, how model scale contributes, and how the number of images affects the results.", "questions": "I am not an expert in industrial anomaly detection; as such, I do not have a good intuition for what exactly the mechanism by which this dataset contributes to e.g. manufacturing actually is; the authors seem much more well-versed in this than I am, so it would be interesting to hear more detailed thoughts. Besides this, however, I do not have many questions, as the paper is relatively clear and self-contained.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed dataset is original, and seems relatively high-quality. Importantly, it seems to be filling an important gap in real-world MLLM evaluations, which I think makes for a strong claim of significance.\n\nThe paper is generally clear and well-written, and the figures are very well-constructed.\n\nThe thoroughness of the evaluations is impressive; I especially appreciate Table 3 and the corresponding analysis of template image usefulness. Additionally, I am surprised that RAG is this helpful—and I'm glad the evaluations are detailed enough to observe this effect! I also appreciate the mention of specific model versions in evaluation, such as \"gpt-4o-mini-2024-07-18\".", "weaknesses": "The paper does not present any human baseline—while every example undergoes manual inspection for a theoretical 100% accuracy, it would be interesting to know what industrial experts achieve. Additionally, several frontier MLLMs are not evaluated, such as Claude Sonnet 3.6.\n\nSome minor nitpicks/thoughts on style (feel free to disregard):\n\n- Line 42: \"represented by\" is a somewhat strange term—maybe \"such as\" would be more accurate?\n- Line 43: I would maybe rephrase this; \"finish\" is an unclear term\n- Line 49: probably \"mechanisms\"\n- Line 50: either the citations should not be parenthetical here, or it's unclear to me what the subject of the sentence is\n- Line 79: I'm unsure what this sentence means; what is \"unfair\" about the comparison?\n- Line 81: I guess you have already introduced it, so \"committed\" is a somewhat strange word to use here (e.g., why not just \"We introduce\")\n- Line 93: would add a space before the parenthetical\n- Line 346: this is probably not the usual definition of \"autonomous agent\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ls4Srfzzto", "reviewer": "Reviewer_dwRW", "summary": "The authors introduce MMAD, the first benchmark for evaluating Multimodal Large Language Models (MLLMs) in industrial anomaly detection (IAD). MMAD contains seven key subtasks relevant to industrial inspection with a dataset of 8,366 industrial images and 39,672 questions. \n\nTheir experiments show that while commercial models like GPT-4o perform best, achieving 74.9% accuracy, they still fall short of industrial requirements. Their analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. The authors propose two training-free strategies to boost MLLM performance.", "questions": "Since the benchmark is generated from existing benchmarks, why is it described as the “first-ever comprehensive” benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow.\n2. The benchmark contains 39,672 questions for 8,366 industrial images, which is helpful for thoroughly evaluating the quality of MLLMs in an IAD setting.\n3. Multiple LLMs were evaluated, and the training-free performance enhancement strategies appear to be useful.", "weaknesses": "1. The contribution are rather weak. While the authors introduce a new benchmark, it primarily relies on images from several existing benchmarks as seed data, using LLMs to generate a synthetic benchmark. This approach does not introduce significant advancements in terms of new methodologies or techniques. As such, claiming the benchmark as \"first-ever comprehensive\" is kinda misleading. \n2. Moreover, the two enhancement strategies proposed are fairly standard and appear to be instantiations of existing solutions applied to the IAD domain without incorporating substantial “domain-specific” optimizations. Given the prevalence of similar techniques in the broader fields, I am skeptical about whether this paper will hold strong interest for the ICLR audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)