Title: VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Benchmark: VRSBench
Summary: VRSBench is a versatile vision-language dataset and benchmark for remote sensing image understanding. This comprehensive dataset not only addresses the limitations of previous datasets that either ignore detailed object information or suffer from quality control issues but also enriches the field by providing a diverse range of annotations including detailed captions, object referring, and visual question answering with rich object information and verified by human annotators.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: remote-sensing image understanding
Phenomenon defined by authors: Yes
Phenomenon definition: Remote sensing models seek to understand the Earth’s surface using imagery captured from overhead, offering a unique perspective of our physical world. This technique is instrumental in various applications, such as land use mapping, urban planning, precision agriculture, disaster management, etc.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: They construct three distinct tasks for advancing remote sensing image under-
standing:
• VRSBench-Cap: This challenge requires the prediction of a comprehensive description for a
given remote sensing image, encapsulating intricate object details and contextual relevance.
• VRSBench-Ref: The task involves identifying and localizing specific objects from a given
remote sensing image based on textual descriptions.
• VRSBench-VQA: This task aims to answer questions related to visual content in a given
remote sensing image.
Task item definition: Each item in the dataset consists of an image, its caption, and a list of questions and answers based on the image. Each question is accompanied by metadata indicating its type, which specifies the aspect of the image it refers to (e.g., object coordinates, object position, object category, etc.).
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 37,408
Dataset size notes: Train: 85,813
Dataset metadata available: Yes
Dataset metadata detail: Question type: which specifies the aspect of the image the question refers to (e.g., object coordinates, object position, object category, etc.).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Intersection over Union (IoU)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding