Title: Unveiling the Tapestry of Consistency in Large Vision-Language Models
Summary: This paper introduces the ConBench benchmark to evaluate the consistency of LVLMs across prompts with varying solution spaces centered on the same knowledge point. The authors reveal key patterns in LVLM behavior and propose a trigger-based diagnostic refinement method to improve consistency and indirectly enhance captioning performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: consistency
Phenomenon defined by authors: Yes
Phenomenon definition: Although LVLMs can generate high-quality responses to task prompts, we discover that for correctly answered cases, simply modifying the prompt will result LVLMs in providing contradictory responses.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take an image and question, provide a short-form answer
Task item definition: <image> Question: How many real cats in the image. A) One B) Two C) Three D) Four. Answer: A
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 4000
Dataset metadata available: Yes
Dataset metadata detail: Hierarchical Core Capability, Question Type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Hierarchical Core Capability, Question Type
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability