MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection

Authors: Xi Jiang, Jian Li, Hanqiu Deng, Yong Liu, Bin-Bin Gao, Yifeng Zhou, Jialin Li, Chengjie Wang, Feng Zheng
Primary area: datasets and benchmarks
Keywords: Anomaly Detection, Multimodal Large Language Model, Industrial Inspection

Abstract:
In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, a full-spectrum MLLM benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9\%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research. 
The code and data are available at https://github.com/jam-cc/MMAD.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729512280976

thread_type:
official_review

<thread_object>
creator: reviewer_dwrw

<content>
summary:
The authors introduce MMAD, the first benchmark for evaluating Multimodal Large Language Models (MLLMs) in industrial anomaly detection (IAD). MMAD contains seven key subtasks relevant to industrial inspection with a dataset of 8,366 industrial images and 39,672 questions. 

Their experiments show that while commercial models like GPT-4o perform best, achieving 74.9% accuracy, they still fall short of industrial requirements. Their analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. The authors propose two training-free strategies to boost MLLM performance.

strengths:
1. The paper is easy to follow.
2. The benchmark contains 39,672 questions for 8,366 industrial images, which is helpful for thoroughly evaluating the quality of MLLMs in an IAD setting.
3. Multiple LLMs were evaluated, and the training-free performance enhancement strategies appear to be useful.

weaknesses:
1. The contribution are rather weak. While the authors introduce a new benchmark, it primarily relies on images from several existing benchmarks as seed data, using LLMs to generate a synthetic benchmark. This approach does not introduce significant advancements in terms of new methodologies or techniques. As such, claiming the benchmark as "first-ever comprehensive" is kinda misleading. 
2. Moreover, the two enhancement strategies proposed are fairly standard and appear to be instantiations of existing solutions applied to the IAD domain without incorporating substantial “domain-specific” optimizations. Given the prevalence of similar techniques in the broader fields, I am skeptical about whether this paper will hold strong interest for the ICLR audience.

questions:
Since the benchmark is generated from existing benchmarks, why is it described as the “first-ever comprehensive” benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674852311

thread_type:
official_review

<thread_object>
creator: reviewer_ydc1

<content>
summary:
The authors introduce MMAD—the first large-scale industrial anomaly detection dataset, specifically targeted at evaluating multimodal large language models. The dataset contains seven main subtasks: anomaly discrimination, defect classification, object classification, defect localization, object analysis, defect description, and defect analysis, which are all highly relevant to the IAD problem. The authors evaluate multiple out-of-the-box MLLMs on their dataset, as well as using RAG and expert model-based improvements. Finally, the authors present a very thorough evaluation of model performance by analyzing whether template images help in anomaly detection, how model scale contributes, and how the number of images affects the results.

strengths:
The proposed dataset is original, and seems relatively high-quality. Importantly, it seems to be filling an important gap in real-world MLLM evaluations, which I think makes for a strong claim of significance.

The paper is generally clear and well-written, and the figures are very well-constructed.

The thoroughness of the evaluations is impressive; I especially appreciate Table 3 and the corresponding analysis of template image usefulness. Additionally, I am surprised that RAG is this helpful—and I'm glad the evaluations are detailed enough to observe this effect! I also appreciate the mention of specific model versions in evaluation, such as "gpt-4o-mini-2024-07-18".

weaknesses:
The paper does not present any human baseline—while every example undergoes manual inspection for a theoretical 100% accuracy, it would be interesting to know what industrial experts achieve. Additionally, several frontier MLLMs are not evaluated, such as Claude Sonnet 3.6.

Some minor nitpicks/thoughts on style (feel free to disregard):

- Line 42: "represented by" is a somewhat strange term—maybe "such as" would be more accurate?
- Line 43: I would maybe rephrase this; "finish" is an unclear term
- Line 49: probably "mechanisms"
- Line 50: either the citations should not be parenthetical here, or it's unclear to me what the subject of the sentence is
- Line 79: I'm unsure what this sentence means; what is "unfair" about the comparison?
- Line 81: I guess you have already introduced it, so "committed" is a somewhat strange word to use here (e.g., why not just "We introduce")
- Line 93: would add a space before the parenthetical
- Line 346: this is probably not the usual definition of "autonomous agent"

questions:
I am not an expert in industrial anomaly detection; as such, I do not have a good intuition for what exactly the mechanism by which this dataset contributes to e.g. manufacturing actually is; the authors seem much more well-versed in this than I am, so it would be interesting to hear more detailed thoughts. Besides this, however, I do not have many questions, as the paper is relatively clear and self-contained.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714927050

thread_type:
official_review

<thread_object>
creator: reviewer_69ie

<content>
summary:
This paper introduces the first benchmark for industrial anomaly detection with multimodal LLMs. It defines 7 subtasks for industrial anomaly detection and creates a novel pipeline for generating semantic annotations of visual anomaly detection data. From the experiments, the best performing model GPT-4o reaches 74.9% accuracy, but it is still way below the industry needs and demonstrates the potential for future research.

strengths:
Overall, this paper is clear, well-motivated and provides a new benchmark for industrial anomaly detection with multimodal LLMs. The tasks selected from the benchmark are comprehensive for identifying industrial anomaly, including defect detection, classification, localization, description, analysis, etc. They also contribute to the diversity and novelty of benchmark tasks. In addition, the paper highlights a few explorational techniques that could improve model performance, such as RAG on domain knowledge, few shots learning and model collaboration. Those explorations are helpful for industrial practitioners using the benchmark in production.

weaknesses:
The paper could benefit from discussions with regards to the following points:
1. This paper fills a gap in the specific domain of industrial anomaly detection by providing a dedicated Multimodal LLM benchmark, and the benchmark has clear value in real-life industrial quality inspections. However, it is not clear how the industrial anomaly detection tasks are fundamentally different from other general visual Q&A tasks, and how this benchmark creates significant contribution to the general ML community beyond its specific application in industrial anomaly detection. There are many other venues that are great fit for industry-specific benchmarks, such as application-focused workshops and NeurIPS dataset and benchmark track.
2. The benchmark constructs the 7 key subtasks in the form of multiple choices. This format could be limiting or misguiding, as it is dependent on how the choices are created, the number of choices provided to the model, and the model could provide the correct answer without truly understanding the answer. For instance, in the defect classification, location, and analysis tasks, accurate open-text answers and locations are much harder to generate and evaluate than multiple choices, but those formats are directly useful in production settings. 
3. In addition, the benchmark only reports on accuracy based on multiple choices, but fails to report recall, F1 score, etc. which would be important in the anomaly detection setting.

questions:
1. It is interesting to see how RAG improves the performance with domain knowledge, as anomaly detection is a multimodal task, not a direct text retrieval task. How could prompting with domain knowledge affect the model performance, as context windows are now getting larger?
2. It is interesting to use an expert model to assist MLLMs - are there other ways of providing the expert information beyond the visualizations listed such as highlight, contour and BBox? It is intuitive to see how BBox can massively improve localization, but wonder if other direct text or other modal information can improve the performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731010602207

thread_type:
official_review

<thread_object>
creator: reviewer_nkbv

<content>
summary:
The authors introduce MMAD, a benchmark specifically designed for anomaly detection in industrial settings to evaluate the capabilities of multimodal large language models (MLLMs). The benchmark encompasses multiple subtasks, wirth 8,336 images and 39,672 questions. Evaluation of current state-of-the-art models demonstrates that performance levels (approximately 70% accuracy) are not yet acceptable by industry standards, highlighting significant gaps in current models’ capabilities. To address these shortcomings, the authors propose the use of Retrieval Augmented Generation for domain-specific knowledge and visual “Expert Agents” to enhance MLLMs’ perception of anomalies. MMAD highlights the current limitations of MLLMs in fine-grained industrial knowledge and provides a benchmark for evaluating future methods in this domain.

strengths:
Comprehensive Benchmark: MMAD is the first benchmark of its kind in industrial anomaly detection, specifically tailored for evaluating MLLMs. It covers a wide range of tasks related to industrial anomaly detection.

Diverse subtasks and Categorization: The benchmark includes numerous subtasks and defect types, ensuring fine-grained evaluations across different aspects of anomaly detection. 

The paper is well-written and easy to follow.

weaknesses:
**Lack of Details on Human Supervision:** Given that this is a dataset-centric paper, there is a lack of detailed information about the exact processes used for dataset filtering and the criteria applied during manual verification. More transparency in the data curation process would strengthen the credibility and reproducibility of the benchmark.

**Insufficient Diversity Analysis:** While the dataset is generated from multiple seed datasets and includes diverse subtasks, there is no systematic analysis of the dataset’s diversity (e.g., semantic similarity measures). Furthermore, there is a lack of analysis on the diversity of the questions generated by the LLM. It is unclear whether the questions follow only a few fixed templates or truly represent a wide range of scenarios, which could limit the benchmark’s effectiveness in evaluating models comprehensively.

**Lack of Comprehensive Analysis:** The paper generally lacks in-depth analyses, such as qualitative assessments, error analyses, or ablation studies, which are crucial for evaluating the quality and limitations of both the dataset and the proposed methods. For instance, although the authors note that models perform poorly on defect-related questions, they do not provide detailed insights into the specific types of errors or misconceptions the models exhibit.

questions:
See Weaknesses Above.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed a new benchmark for vision language models on the task of detecting industrial anomalies and defects. Reviewers generally find the contribution of the paper valid and valuable as the application of LLMs to industrial settings is important. The writing quality is good. The main weakness left unaddressed after the rebuttal is the novelty of the benchmark (compared to other image QA benchmarks) and its relevance to the ICLR community. Considering this benchmark has its own unique characteristics (e.g., defects detection requires fine-grained vision comparisons) and the broad interests in LLM applications in the industry, the AC is inclined to accept this paper.

The AC **strongly advises the authors to remove the word "first-ever"** from the title of this paper. Using this word is simply unprofessional in academic papers - every paper can claim their contribution as "first-ever" and this word is largely redundant and unnecessary.