Title: Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
Benchmark: R-Bench
Summary: This work introduces R-Bench, a new benchmark designed to evaluate hallucinations in inter-object relationships. It identifies three key sources of hallucination and reveals that LVLMs often ignore visual input, depend too heavily on language priors, and struggle with spatial reasoning due to long-tail distribution biases in training data.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: visual understanding, hallucination
Phenomenon defined by authors: Yes
Phenomenon definition: Existing LVLMs often tend to generate responses that are inconsistent with the content of the images. This issue is particularly critical for LVLMs, which are expected to accurately comprehend images and produce answers consistent with the content of the visual input. There exists a notable gap in addressing hallucinations related to inter-object relationships.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take a image-level and instance-level questions and provide the labels of "yes" or "no"
Task item definition: <image> question: Is there a man swinging a bat? answer: Yes
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 11651
Dataset metadata available: Yes
Dataset metadata detail: number of objects, number of relationship
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Response format detail: it's a classification task with "yes" and "no" labels
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Spatial Reasoning