INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance

## Abstract

Abstract Large Vision-Language Models (LVLMs) have demonstrated outstanding performance in various general multimodal applications such as image recognition and visual reasoning, and have also shown promising potential in specialized domains. However, the application potential of LVLMs in the insurance domain—characterized by rich application scenarios and abundant multimodal data—has not been effectively explored. There is no systematic review of multimodal tasks in the insurance domain, nor a benchmark specifically designed to evaluate the capabilities of LVLMs in insurance. This gap hinders the development of LVLMs within the insurance domain. In this paper, we systematically review and distill multimodal tasks for four representative types of insurance: auto insurance, property insurance, health insurance, and agricultural insurance. We propose INS-MMBench, the first comprehensive LVLMs benchmark tailored for the insurance domain. INS-MMBench comprises a total of 2.2K thoroughly designed multiple-choice questions, covering 12 meta-tasks and 22 fundamental tasks. Furthermore, we evaluate multiple representative LVLMs, including closed-source models such as GPT-4o and open-source models like BLIP-2. This evaluation not only validates the effectiveness of our benchmark but also provides an in-depth performance analysis of current LVLMs on various multimodal tasks in the insurance domain. We hope that INS-MMBench will facilitate the further application of LVLMs in the insurance domain and inspire interdisciplinary development. Our dataset and evaluation code are available at https://github.com/FDU-INS/INS-MMBench .

## 1 Introduction

In recent years, Large Language Models (LLMs) have demonstrated remarkably powerful semantic understanding and conversational capabilities [, , , , ], profoundly impacting human work and life. Building on this foundation, Large Visual Language Models (LVLMs) have taken a further step by mapping and aligning visual and textual features, enabling the processing and interaction with multimodal data [, , , ]. Researchers have found that LVLMs exhibit exceptional performance in general tasks such as image recognition, document parsing, and OCR processing [, , ]. Beyond exploring general capabilities, researchers have also begun to apply LVLMs to various specialized domains such as healthcare [, ], autonomous driving [, ] and social media content analysis [, ]. By exploring the capabilities of LVLMs in specialized domains through qualitative and quantitative methods, these studies have demonstrated various application potentials.

Insurance, as a discipline encompassing numerous multimodal application scenarios, involves extensive use of multimodal data and computer vision algorithms in its actual operations [, , , ]. This offers vast potential for the integration of LVLMs with the insurance industry. For instance, in auto insurance, analyzing images of damaged vehicles can enable quick assessments and accurate estimations of damage []. Similarly, in property insurance, analyzing images of buildings can help evaluate potential risks []. However, existing research [] has only qualitatively analyzed the application of LVLMs in the insurance domain, without systematically organizing related multimodal tasks or constructing domain-specific benchmarks. This has hindered the in-depth evaluation and promotion of LVLMs’ capabilities within the insurance domain.

To address this challenge, we introduce INS-MMBench, the first comprehensive LVLMs benchmark for the insurance domain (see Figure 1). For task design, we systematically organize and refine multimodal tasks across four representative types of insurance: auto, property, health, and agricultural insurance. Using a bottom-up hierarchical task definition methodology, we construct a total of 12 meta-tasks and 22 fundamental tasks, covering key insurance stages such as underwriting, risk monitoring, and claim processing. For data collection, we search and process datasets from multiple open-source channels, selecting datasets with high scenario relevance, task relevance, and data availability. For benchmark construction, INS-MMBench includes a total of 2.2K thoroughly designed multiple-choice visual questions. Such format facilitates convenient and objective analysis of evaluation results. These questions are formulated manually and the distractor options are generated with the help of GPT-4o.

Furthermore, we select 10 LVLMs for evaluation and conduct a comprehensive analysis of the results. The key findings from the evaluation are as follows: (1) GPT-4o performs the best among all models, scoring 72.91/100. It is also the only model to score over 70, reflecting the challenging nature of the INS-MMBench; (2) There are significant differences in LVLMs’ performance across different insurance types, with better results in auto insurance and health insurance compared to property insurance and agricultural insurance; (3) LVLMs exhibit marked differences in performance across different meta-tasks, closely related to the task type and the image type; (4) The gap between open-source and closed-source LVLMs is narrowing, with some open-source models now approaching or even surpassing the capabilities of closed-source models in some tasks; (5) The primary reasons for LVLMs’ errors on the INS-MMBench are lack of knowledge and understanding in the insurance field, as well as perception errors.

Overall, the contributions of our work are as follows:

- •
We propose INS-MMBench, the first LVLMs benchmark for the insurance domain, which includes a total of 2.2K multiple-choice visual questions covering four types of insurance (auto, property, health, and agricultural insurance), 12 meta-tasks and 22 fundamental tasks.
- •
We conduct an in-depth evaluation of 10 LVLMs, including 7 proprietary and 3 open-source models, representing the first quantitative assessment of LVLMs’ capabilities in the insurance domain.
- •
We conduct a further analysis of the evaluation results, providing insights into the potential applications of LVLMs in the insurance domain. This analysis also offers a reference for understanding the opportunities and challenges associated with LVLMs in this sector.

## 2 Related works

### 2.1 Large Vision-Language Models

With the rapid development of Large Language Models (LLMs) [, , ], researchers are leveraging the powerful generalization capabilities of these pre-trained LLMs for processing and understanding multimodal data [, , ]. A key area of focus is the use of Large Vision-Language Models (LVLMs) for visual inputs. LVLMs employ visual encoders and visual-to-language adapters to encode the visual features from image data and align these features with textual features. The combined features are then processed by pre-trained LLMs, leading to significant advancements in visual recognition and understanding [, ].

Various open-source and closed-source LVLMs are continuously emerging. In the realm of open-source models, notable examples include LLaMA-Adapter [], LLaVA [], BLIP-2 [], MiniGPT-4 [], and InternVL []. These models have successfully integrated visual and textual modalities, achieving commendable results. In the closed-source domain, representative models include GPT-4o [], GPT-4V [], GeminiProVision [], and Qwen-VL [], all of which have demonstrated outstanding performance in numerous tests and evaluations [, , ]. We intend to evaluate both open-source and closed-source LVLMs to verify the capability of different models in the insurance domain.

### 2.2 Benchmarks for Large Vision-Language Models

As research into LVLMs intensifies, an increasing number of researchers are proposing benchmarks to evaluate the capabilities of models [, , , ]. Based on the scope of capability evaluation, these studies can be categorized into three types: task-specific benchmarks, comprehensive benchmarks, and domain-specific benchmarks.

#### Comprehensive benchmarks

are characterized by their breadth and generality. Researchers construct these benchmarks by defining and categorizing the general capabilities and tasks of LVLMs, resulting in a comprehensive and wide-ranging evaluation. Representative studies include LVLM-eHub [], SEED-Bench [, ], MMBench [], MME, and MMT-Bench [].

#### Task-specific benchmarks

focus on particular tasks and types of visual data, providing detailed task definitions. Examples include SciFIBench [] for scientific images, MMC-Benchmark [] for charts, MVBench [] (using video frames as input) for videos and SEED-Bench-2-Plus [] for web pages, charts and maps.

#### Domain-specific benchmarks

are designed for visual tasks within specific professional domain. Due to the specialized knowledge and unique tasks of these domains, general benchmark cannot fully meet the needs of evaluating LVLMs in these areas. As a result, researchers have begun proposing specialized benchmarks for domains such as healthcare (OmniMedVQA []), mathematics [, ], autonomous driving (Talk2BEV-Bench []), and geography []. However, as mentioned previously, the insurance domain and even the finance domain currently lack corresponding domain-specific benchmarks for LVLMs [, , ]. Our work introduces INS-MMBench to address this gap, aiming for a significant advancements in the application of LVLMs in the insurance domain.

## 3 INS-MMBench

### 3.1 Tasks

Given the differences in workflows among various types of insurance in practical operations, we select four core types for building this benchmark: auto insurance, commercial/household property insurance, health insurance, and agricultural insurance. These categories cover both life and property insurance, which are the most prevalent in the insurance market and highly representative [, ].

Figure: Figure 2: An illustration of our bottom-up hierarchical task definition method. First, we identify and categorize different insurance stages. Next, we enumerate the key visual elements required at each stage. Based on these key visual elements, we define the fundamental tasks. Finally, we cluster the fundamental tasks to form meta-tasks.
Refer to caption: extracted/5664763/images/task_construction.png

To ensure that our evaluation tasks closely align with real-world applications in the insurance domain and fully demonstrate the capabilities of LVLMs in this context, we have developed a bottom-up hierarchical task definition methodology. Using this methodology, we construct a systematic visual task framework specifically tailored for the insurance sector. As an example, we discuss the detailed task construction process for auto insurance (see Figure 2).
Initially, based on the insurance value chain theory [, ], we select three key stages rich in multimodal data and tasks: vehicle underwriting, vehicle risk monitoring, and vehicle claim processing. At each stage, we identify the key visual elements that insurance operators need to extract. For instance, during the vehicle underwriting stage, operators must confirm elements such as license plate information, vehicle model, dashboard readings, and vehicle condition, which are crucial for information collection, condition verification, and underwriting decision-making. Further, based on these key visual elements, we define the fundamental tasks. For example, the need to extract license plate information led to the definition of the License Plate Recognition task, while the need to monitor risky driving behavior resulted in the In-car Driving Driving Behavior Detection task. By following this process, we define a total of nine fundamental tasks for auto insurance. Finally, we cluster these fundamental tasks based on their characteristics, forming four meta-tasks. Through this approach, we have constructed a comprehensive set of 12 meta-tasks and 22 fundamental tasks across the four types of insurance.

### 3.2 Dataset collection

Once the task definition is complete, we start collecting data and constructing the multi-choice visual questions. Our data collection and benchmark construction process (see Figure 3) is as follows:

Figure: Figure 3: An illustration of our data collection and benchmark construction process. First, we collect datasets from multiple public sources. Next, we perform manual filtering and random sampling of the datasets, followed by the necessary data processing. Finally, both manual effort and GPT-4o are utilized to construct task questions and multiple-choice options, creating a multi-choice visual question dataset.
Refer to caption: extracted/5664763/images/data_collecting_and_processing_pipeline.png

#### Data sources.

We search for datasets using keywords related to the fundamental tasks in several popular data sources, including Google, Kaggle, Github, and Roboflow. For tasks where multiple public datasets are available, we download and compare these datasets to perform an initial screening. We select datasets with high adaptability and usability for insurance scenarios, as detailed in Table 1.

#### Data processing.

To facilitate LVLMs evaluation, we set the number of images and questions for each fundamental task to 100. These 100 images are randomly sampled from our selected datasets, and considering the balance of test sample types, we perform balanced sampling on datasets with categorical labels. For example, in the vehicle damage severity detection task, we ensure that the number of labels - undamaged, minor damage, moderate damage and severe damage - is balanced to maintain the validity of the evaluation. Meanwhile, we process the annotation content, converting it to text-based labels, in preparation for subsequent question and answer generation.

#### Question and answer generation.

For each fundamental task, we set questions that are directly and unambiguously related to the task. For example, the question for the number plate recognition task is “What is the number plate of the vehicle in the picture?” The number of options for the questions is set to 2 to 4. For tasks with yes/no labels, we keep the yes/no labels as options. For other tasks, we generate distractor options using the GPT-4o model, and finally combine these options into a multi-choice visual question format. In each fundamental task we ensure a balanced distribution of correct option positions.

**Table 1: An overview of the datasets used in INS-MMBench.**
| Insurance type | Meta-tasks | Fundamental tasks | Dataset | Access |
| --- | --- | --- | --- | --- |
| Auto insurance | Vehicle information extraction | License plate recognition | CCPD [], mjdfodf-qmbuf [] | Open Access |
|  |  | Vehicle mileage reading | TRODO [] | Open Access |
|  |  | Vehicle warning indicator recognition | dataset_dashboard [] | Open Access |
|  | Vehicle appearance recognition | Vehicle make and model identification | Stanford Cars [] | Open Access |
|  |  | Vehicle modification detection | tuning-car-detection [] | Open Access |
|  | Driving behavior detection | Incar driving behavior detection | Driver-Distraction-Dataset [] | Open Access |
|  | Vehicle damage detection | Vehicle damage part detection | car_dent_scratch_detection-1 [] | Open Access |
|  |  | Vehicle damage type detection | Cardd [] | Open Access |
|  |  | Vehicle damage severity detection | car-crash-severity-detection [] | Open Access |
| Property insurance | Property risk assessment | Roof condition assessment | damages-svll3 [] | Open Access |
|  |  | Workplace risk assessment | worker-safety [] | Open Access |
|  | Property anomaly detection | House fire detection | fire-detection-cta61 [] | Open Access |
|  | Property damage detection | House damage type detection | damage-type [] | Open Access |
|  |  | House damage level detection | damage-level [] | Open Access |
| Health insurance | Health risk monitoring | Fall detection | Fall Detection Dataset [] | Open Access |
|  |  | Health device reading | blood-pressure-monitor-display [] | Open Access |
|  | Medical image recognition | Medical image organ recognition | VQA-Med 2019 [] | Open Access |
|  |  | Medical image abnormality recognition | VQA-Med 2019 [] | Open Access |
| Agricultural insurance | Crop type identification | Field image crop type identification | agricultural crop images [] | Open Access |
|  |  | Satellite image crop type identification | Drone Imagery Classification<br>Training Dataset for Crop Types<br>in Rwanda [] | Open Access |
|  | Crop growth status identification | crop growth stage recognition | wheat-growth-stage-challenge [] | Open Access |
|  | Farmland damage detection | Farmland damage type detection | agriculture-vision [] | Open Access |

## 4 Experiment

### 4.1 Experimental setting

#### Selected LVLMs.

We select a representative set of 10 LVLMs for our evaluation. This set includes seven closed-source LVLMs: GPT-4o, GPT-4V, GeminiProVision, QwenVLPlus, QwenVLMax, Claude3V_Sonnet, and Claude3V_Haiku as well as three open-source LVLMs including LLaVA, BLIP-2, and Qwen-VL-Chat.

#### Evaluation methods.

We employ VLMEvalKit, an open-source evaluation toolkit for LVLMs developed by OpenCompass [], to conduct our evaluations. This toolkit supports integrated testing of both closed-source and open-source LVLMs and is adaptable to custom benchmark datasets. VLMEvalKit provides two methods for evaluating responses to multi-choice visual questions: exact matching (finding "A", "B", "C", "D" in the output strings) and LLM-based answer extraction which analyzes the answer outputs using a Large Language Model (we use GPT-3.5 here). These methods help mitigate the issue of uncontrolled free-form content generation by LVLMs. The accuracy metric is used as the evaluation criterion.

### 4.2 Main results

Tables 2 and 3 present the evaluation results of LVLMS across various insurance types and meta-tasks, respectively, using random guessing as the baseline. The results are organized into two sections: the first seven rows feature proprietary LVLMs, while the subsequent rows cover open-source LVLMs. Overall, GPT-4o outperforms all other models, emerging as the top-performing LVLM on the INS-MMBench with a score of 72.91. This is the only model with an overall score exceeding 70, underscoring the challenging nature of the INS-MMBench. Most LVLMs scored below 60, and some even underperformed relative to a random guess baseline of 25 in certain insurance categories, indicating significant potential for improvement in applying LVLMs within the insurance domain. Based on the data in Tables 2 and 3, the following observations can be made.

Table: Table 2: Evaluation results of the LVLMs across different insurance types. The values in the table represent the average accuracy. The highest and second-highest results are highlighted in bold and underlined, respectively.

Table: Table 3: Evaluation results of the LVLMs across different meta-tasks. The values in the table represent the average accuracy. Specifically, VIE denotes vehicle information extraction, VAR denotes vehicle appearance recognition, DBD denotes driving behavior detection, VDD denotes vehicle damage detection, HPAD denotes household/commercial property anomaly detection, HPDD denotes household/commercial property damage detection, HPRA denotes household/commercial property risk assessment, HRM denotes health risk monitoring, MIR denotes medical image recognition, CGSI denotes crop growth stage identification, CTI denotes crop type identification, FDD denotes farmland damage detection. The highest and second-highest results are highlighted in bold and underlined, respectively.

LVLMs show significant variance across different types of insurance. Experimental results reveal that both open-source and proprietary LVLMs perform better in tasks related to auto insurance and health insurance compared to those involving property and agricultural insurance. For instance, GPT-4o, which exhibits the best performance, scores 85.33 and 82.00 in auto and health insurance tasks respectively; however, its scores drop to 65.00 and 45.50 in property and agricultural insurance tasks, indicating a gap from practical application. This discrepancy may stem from the availability of datasets. Our data collection process highlights that publicly available datasets are more plentiful and comprehensive in the automotive and medical fields. Based on these observations, we suggest that the future deployment of LVLMs in the insurance sector should be a progressive process, initially focusing on areas like auto and health insurance where they are most effective.

Figure: Figure 4: Performance of closed-source LVLMs on different meta-tasks.
Refer to caption: extracted/5664763/images/close_source_result_radar.png

LVLMs show significant variance across different meta-tasks. Experimental results reveal that LVLMs demonstrate considerable performance variability across various meta-tasks, likely influenced by the specific nature of each task and the characteristics of the images involved. Most models excel in tasks like vehicle information extraction (VAE), vehicle appearance recognition (VAR), and health risk monitoring (HRA), which primarily depend on visual element perception and object detection. In contrast, performance dips in more complex tasks such as household/commercial property damage detection (HPDD) and crop growth stage identification (CGSI), which demand additional domain-specific knowledge or reasoning abilities. Furthermore, LVLMs generally struggle with tasks involving satellite or drone aerial imagery, including household/commercial property risk assessment (HPRA), crop type identification (CTI), and farmland damage detection (FDD), where unique imaging perspectives and data complexities pose additional challenges.

Narrowing gap between open-source and closed-source LVLMs. A comparison of the overall performance of open-source and closed-source LVLMs on INS-MMBench indicates that, while there is still a notable gap between the two, some open-source LVLMs are nearing the performance levels of their closed-source counterparts. This trend suggests that as open-source models grow stronger and domain-specific data becomes more abundant, focusing on training high-performance, domain-specific LVLMs could become a key development strategy in the application of LVLMs within the insurance domain.

### 4.3 Error analysis

To provide further insights into the limitations of LVLMs in the insurance domain, we conduct an in-depth analysis of the errors made by selected models on the INS-MMBench. We examine the error patterns of three models: GPT-4o, GeminiProVision, and Qwen-VL-Max, categorizing the errors into four types: perception errors (where LVLMs do not recognize or detect objects or content within the image), lack of insurance knowledge or reasoning ability (where LVLMs can recognize and perceive visual content but lack the necessary insurance knowledge or reasoning skills to answer the question), refusal to answer (where LVLMs decline to respond to questions they deem sensitive or illegal), and failure to follow instructions (where LVLMs do not adhere to the provided instructions, resulting in irrelevant responses).

Figure: Figure 5: The distribution of error types for GPT-4o, GeminiProVision, and Qwen-VL-Chat.
Refer to caption: extracted/5664763/images/error_analysis.png

The error analysis results for these models are illustrated in Figure 5. The most common error type is the lack of insurance knowledge or reasoning ability, which accounts for 59.5%, 64.0%, and 57.2% of the errors in GPT-4o, GeminiProVision, and Qwen-VL-Max, respectively. Due to insufficient specialized knowledge and analytical skills in the insurance field, LVLMs struggle to accurately assess and judge factors such as risk conditions and the extent of damage. Therefore, optimizing LVLMs for the insurance domain should primarily focus on enriching domain-specific knowledge and enhancing professional capabilities. Perception errors are the second most significant error type. Limited by the capabilities of the visual encoder, LVLMs often fail to fully recognize and capture detailed content in images, leading to misinterpretations. For instance, GPT-4o misidentifies a damaged farmland image as ‘an abstract or close-up view of a textured surface with blue and purple hues’. This type of error is common across LVLMs. Additionally, due to built-in safety monitoring functions, GPT-4o and GeminiProVision sometimes incorrectly flag images as illegal and refuse to respond. Qwen-VL-Max, on the other hand, struggles with following instructions, occasionally outputting content in Chinese, which compromises result accuracy.

## 5 Discussions and conclusions

In this paper, we introduce INS-MMBench, a multimodal benchmark tailored for the insurance domain, designed to evaluate Large Vision-Language Models (LVLMs). To the best of our knowledge, this is the first initiative to systematically review multimodal tasks within this sector and establish a specialized benchmark specifically for it. INS-MMBench comprises 2.2K multiple-choice visual questions, covering four types of insurance, 12 meta-tasks, and 22 fundamental tasks, effectively supporting the assessment of LVLMs’ applications in insurance. Additionally, we evaluate several mainstream LVLMs and provide a detailed analysis of the results, offering an initial exploration into the feasibility of employing LVLMs in the insurance sector. We hope our benchmark and findings will guide future research in this field and enhance the integration of insurance academia with AI advancements, promoting interdisciplinary exchanges within the sector.

However, this study has limitations. A significant constraint is the lack of open-source image datasets specific to the insurance domain, largely due to privacy concerns. The data used in this study, sourced from publicly available datasets, has been manually curated but may still harbor biases that do not fully align with real-world insurance scenarios. This issue underscores the need for collaborative efforts between insurance companies and the academic community to develop dedicated open-source image datasets for the insurance domain.

Another limitation is that INS-MMBench disaggregates the tasks of LVLMs into various fundamental tasks, assessing LVLM performance from a micro perspective based on task-specific accuracy. In reality, visual tasks in insurance often entail complex integration of multiple capabilities and comprehensive analysis. Addressing this, our next objective is to construct a more complex, integrated application benchmark to enable a deeper evaluation of LVLM applications in the insurance domain.

## Appendix A Example Cases

To offer a detailed view of the task settings in INS-MMBench, we have selected sample cases for each core task and present responses from GPT-4o, GeminiProVision, and Qwen-VL-Max in this section.

Figure: Figure 6: A sample case of license plate recognition and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x1.png

Figure: Figure 7: A sample case of vehicle mileage reading and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x2.png

Figure: Figure 8: A sample case of vehicle warning indicator recognition and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x3.png

Figure: Figure 9: A sample case of vehicle make and model identification and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x4.png

Figure: Figure 10: A sample case of vehicle modification detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x5.png

Figure: Figure 11: A sample case of incar driving behavior detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x6.png

Figure: Figure 12: A sample case of vehicle damage part detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x7.png

Figure: Figure 13: A sample case of vehicle damage type detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x8.png

Figure: Figure 14: A sample case of vehicle damage severity detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x9.png

Figure: Figure 15: A sample case of roof condition assessment and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x10.png

Figure: Figure 16: A sample case of workplace risk assessment and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x11.png

Figure: Figure 17: A sample case of hosue fire detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x12.png

Figure: Figure 18: A sample case of hosue damage type detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x13.png

Figure: Figure 19: A sample case of hosue damage level detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x14.png

Figure: Figure 20: A sample case of fall detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x15.png

Figure: Figure 21: A sample case of health device reading and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x16.png

Figure: Figure 22: A sample case of medical image abnormality detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x17.png

Figure: Figure 23: A sample case of medical image part recognition and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x18.png

Figure: Figure 24: A sample case of field image crop type identification and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x19.png

Figure: Figure 25: A sample case of satellite image crop type identification and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x20.png

Figure: Figure 26: A sample case of crop growth stage recognition and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x21.png

Figure: Figure 27: A sample case of farmland damage detection and the responses of GPT-4o, GeminiProVision and Qwen-VL-Max.
Refer to caption: x22.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Check Weaknesses.
soundness: 3
strengths: 1. Comprehensive Benchmark: The paper presents INS-MMBench, which is the first comprehensive benchmark tailored for evaluating LVLMs in the insurance domain. This benchmark is extensive, covering 8,856 multiple-choice visual questions across 12 meta-tasks and 22 fundamental tasks, providing a robust framework for assessing LVLM capabilities in various insurance scenarios.

2. Systematic Framework: The authors have developed a systematic and hierarchical task definition that ensures the tasks are closely aligned with real-world applications in the insurance industry. This bottom-up approach to task construction enhances the benchmark's relevance and practicality, making it a valuable tool for both research and practical applications.

3. The paper also includes an extensive evaluation of multiple representative LVLMs, offering detailed performance analysis across different insurance types and meta-tasks. This analysis not only validates the effectiveness of the INS-MMBench benchmark but also provides actionable insights into the current capabilities and limitations of LVLMs in the insurance domain, guiding future research and development efforts.
summary: The paper introduces INS-MMBench, a comprehensive benchmark designed to evaluate the performance of LVLMs in the insurance domain. It is the first initiative to systematically review multimodal tasks within the insurance sector and establish a specialized benchmark for it.
weaknesses: 1. Multi-Choice Format Limitations: This benchmark follows a similar style to MMBench and MME in the general multimodal domain, all of which formulate their questions into multiple-choice formats. While this is an effective method for evaluating model performance, it has limitations that prevent generalization to open-ended question answering, which is more representative of real-world applications.

2. Static Benchmark and Data Leakage: The benchmark is static, which does not mitigate the data leakage problem. This will likely render the benchmark less effective in future developments.

3. Focus on US Insurance Law and Potential Bias: The benchmark primarily focuses on insurance, specifically insurance laws from the United States. This focus may introduce bias into the evaluation process, posing a risk for models developed in different country contexts.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Regarding the first limitation, could you share your perspective on how the current selective tasks directly align with the actual stages in the insurance process? For example, specific insurance stages like underwriting or claims processing?

I would consider slightly increasing the score if convinced that the benchmark specifically addresses key insurance stages, rather than being a collection of VQA tasks merely related to the selected insurance categories(auto, property, health, and agriculture).
soundness: 1
strengths: 1. The motivation behind establishing an insurance benchmark is worthwhile.  Evaluating LVLMs' capabilities on core insurance stages like underwriting and claims processing is practical and meaningful.
2. The benchmark covers a reasonable range of core insurance types relevant to key areas in everyday insurance applications.
3. The study provides an insightful error analysis, highlighting the current limitations of LVLMs in interpreting insurance-specific visual content.
summary: The potential for Large Vision Language Models (LVLMs) to impact the insurance industry is substantial, yet largely unexplored. This study establishes a benchmark to evaluate LVLM capabilities within the domain, focusing on four main insurance types: auto, property, health, and agriculture. To create the benchmark, the authors gathered multimodal data for each insurance category from public sources and converted it into multiple-choice questions using GPT-4o. They then evaluated popular LVLMs on this benchmark to provide an initial assessment of LVLM performance and reveal current limitations in handling insurance-related content by an error analysis. Finally, the authors try to address gaps in insurance knowledge and reasoning skills by adding insurance-related information to the prompt.
weaknesses: 1. **Misalignment between Intent and Implementation**: While the authors claim the benchmark includes 12 meta-tasks and 22 fundamental tasks across stages like underwriting and claims processing in the Introduction section, the tasks illustrated in the paper are only loosely related to these stages. For example, meta-tasks in auto insurance such as “vehicle information extraction” and “vehicle damage detection” focus heavily on general computer vision tasks rather than directly addressing insurance-specific stages. This makes the benchmark feel more like a vision task set than an insurance task set.    
2. **Limited Accessibility for Reproducibility**: Although the authors promise to release the code and dataset, the GitHub repository has not been updated in four months, containing only a readme and a few diagrams. This lack of resources limits my ability to further assess the benchmark’s true rationality and effectiveness.
3. **Limited Novelty**: Some conclusions, such as “performance of closed-source LVLMs varies by training data size and methods,” are too general and widely understood, offering little new insight. The paper would benefit from focusing on more specific findings directly related to the insurance domain.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you provide more details on how the benchmark tasks were selected?

It would be helpful to understand the criteria used to determine which tasks were included in INS-MMBench. Specifically, how did you ensure that the tasks accurately reflect real-world insurance challenges and not just general visual recognition problems?

2. Have you considered including more complex, multi-step reasoning tasks?

Given the importance of decision-making in insurance, would it be possible to expand the benchmark to include tasks that require multi-modal integration and reasoning (e.g., verifying a claim using images, text descriptions, and numerical data)? This could better showcase the strengths and weaknesses of LVLMs in handling real-world scenarios.

3. How do you envision improving the interpretability of model evaluations?

Since explainability is critical in insurance, have you considered adding tasks that require LVLMs to provide justifications or rationales for their answers? This could allow for a deeper evaluation of how well models understand and explain their decisions, which is crucial for real-world applications.

4. Do you have insights on the performance gap between open-source and closed-source models?

The results indicate a narrowing gap between open and closed-source models. Can you elaborate on specific factors contributing to this trend, and how future benchmarks might encourage more competitive open-source solutions?
soundness: 3
strengths: (1) Originality: INS-MMBench is the first benchmark tailored to evaluate LVLMs in the insurance domain. The authors' approach to defining tasks using a bottom-up hierarchical methodology is innovative and ensures that the benchmark aligns with real-world insurance scenarios, making it a pioneering effort in applying LVLMs to this new domain.

(2) Quality: The authors systematically identify and organize multimodal tasks across four types of insurance, and their comprehensive evaluation of ten LVLMs provides some insights. The inclusion of detailed error analysis and the exploration of prompt engineering techniques to mitigate common issues further strengthen the paper, offering practical suggestions for improving model performance.

(3) Clarity: The authors explain each step of their methodology in detail. 

(4) Significance: The introduction of INS-MMBench contributes to the field, as it enables a more nuanced evaluation of LVLMs in a domain with substantial practical applications. The benchmark could lead to improved automation in insurance-related tasks, such as claims processing and fraud detection, thus enhancing efficiency and accuracy in the industry. Moreover, by highlighting the narrowing performance gap between open-source and closed-source LVLMs, the paper encourages further research and development, potentially driving advancements in accessible and effective AI solutions for the insurance sector.
summary: The paper presents a benchmark in the systematic evaluation of LVLMs in this field by introducing INS-MMBench, a domain-specific benchmark designed to assess these models across various insurance-related tasks.

Key contributions:

(1) INS-MMBench is the first comprehensive benchmark tailored for the insurance domain. It covers four representative types of insurance: auto, property, health, and agricultural insurance, reflecting real-world insurance scenarios such as underwriting and claims processing.

(2) The authors used a bottom-up hierarchical task definition approach to identify and refine relevant tasks for each insurance type. They collected and processed datasets to create visual question-answer pairs, ensuring that the benchmark aligns with practical applications in the insurance industry.

(3) The paper evaluates different LVLMs using INS-MMBench. The results highlight the challenges these models face and give some insights.
weaknesses: 1. Benchmark Definition Lacks Depth in Insurance Scenarios

While INS-MMBench introduces tasks related to insurance, many are more aligned with general, common-sense VQA rather than specialized, nuanced scenarios seen in real-world insurance applications. To better reflect practical needs, the benchmark should include more complex tasks, such as multi-step reasoning or risk assessment based on a mix of visual and contextual data.

2. Overemphasis on Basic Tasks

Some tasks, like license plate recognition, are too basic and can be handled by smaller, specialized models. Evaluating LVLMs on these tasks does not showcase their strengths. Instead, the benchmark should focus on tasks requiring more advanced reasoning, such as verifying claims by cross-referencing multiple data points, to highlight the real capabilities of LVLMs.

3. Limited Emphasis on Reasoning and Higher-Order Tasks 

The benchmark lacks tasks that test higher-order reasoning, which is crucial for insurance scenarios. Tasks involving contextual understanding, complex decision-making, and multi-modal integration would better evaluate how well LVLMs can handle real insurance industry challenges.

4. Lack of Focus on Interpretability 

Insurance applications require transparency, yet INS-MMBench primarily uses multiple-choice questions, limiting the ability to assess whether models can explain their decisions. Future benchmarks should include tasks that require LVLMs to provide rationale, enabling evaluation of their interpretability, which is critical for building trust in automated systems.

5. Clarification Needed on Table 1
It appears there might be an error in the labeling of the last two rows in Table 1. Currently, "OmniMedVQA" is described as domain-specific for math, and "Mathvista" as domain-specific for medical. Given the names and typical use cases, it seems like these two may have been accidentally switched.

Recommendations:
1. Integrate more complex, domain-specific scenarios that mimic real-world tasks.
2. Replace basic tasks with challenges requiring higher-order reasoning and contextual analysis.
3. Add tasks that require models to explain their answers, enhancing interpretability.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - How do LVLMs perform on insurance tasks that related to temporal reasoning for example analyzing claim patterns over time? The work evaluates static image understanding, but many insurance tasks require understanding temporal relationships and changes over time, and it seems that those samples are missing from the current benchmark.
soundness: 3
strengths: - The paper is well-written and well-organized. 
- This is the first systematic benchmark specifically designed for the LVLMs evaluation in the insurance domain and fills a gap in the current benchmark that often overlooks domain-specific applications.
- The experiments are comprehensive, and thorough error analysis categorizing different types of model failures is provided in the paper.
summary: This work introduces INS-MMBench, the first comprehensive benchmark designed to evaluate LVLMs in the insurance domain. INS-MMBench includes four insurance types: auto, property, health, and agriculture, and includes 8856 multiple-choice questions across 12 meta-tasks and 22 fundamental tasks. It is designed to evaluate LVLMs in practical insurance tasks, such as vehicle damage detection and health risk monitoring, combining real-world visual information with insurance-specific questions. Through the experiments, the authors show the current limitations of LVLMs in insurance domain and suggests targeted data and domain knowledge for improving the performance.
weaknesses: - The human baseline experiments only involve 3 graduate students specialized in insurance, which is a small sample size.  This might not accurately represent the range of expertise and variability in the real-world insurance evaluations. I would suggest bringing in more experts from the industry to help perform the human evaluation. 
- This work does not discuss potential biases in the data sources or methods for mitigating them, which means that there is a risk that the benchmark may favor certain model behaviors or fail to generalize to different insurance scenarios.

Meta-review: The paper introduces a new benchmark for multimodal models in insurance, covering auto, property, health, and agriculture. Reviewers liked the domain-specific focus and thorough experiments. However, the tasks are mostly simple visual tasks, missing deeper insurance-specific workflows like multi-step or temporal reasoning. The data appear US-centric, and the approach relies heavily on MCQs which can be limited. These weaknesses reduce its potential impact as a specialized benchmark, and the final recommendation from the AC is reject.

Decision: Reject