Title: VLUE: A Multi-Task Multi-Dimension Benchmark for Evaluating Vision-Language Pre-training
Benchmark: VLUE
Summary: VLUE is a vision-language benchmark consisting of 4 representative VL tasks, each equipped with a private test set annotated on images from wild distribution. The authors evaluate the efficiency-performance trade-off of representative VLP models and build a Pareto SOTA landscape for current VLP research. Additionally they provide an extensive analysis of the generalization ability and the efficiency-performance trade-off of representative VLP models.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: - Generalization and transferability of VLP models
- Efficiency-performance trade-off
Phenomenon defined by authors: No
Phenomenon definition: The authors do not explicitly define the above phenomena. They only mention the following:
- Generalization: VLP models have already seen the images in the downstream datasets and their captions before fine-tuning and evaluating on them, overestimating their transfer and generalization abilities.
- Performance-efficiency: "We refer to the goal of this phenomenon as “Pareto SOTA” following, which means that there is no other model currently better than it on all the dimensions of interest such as performance and efficiency. Therefore, we believe it is necessary to measure and report performance-efficiency trade-off."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: VLUE covers a set of fundamental VL tasks, including image-text retrieval, visual question answering, visual reasoning, and visual grounding.
Task item definition: Depending on the task:
- retrieval: query and the relevant image
- reasoning: pair of images and a natural statement, True/False label
- grounding: image, target object, target location
- QA: image, question and answer
- captioning: image and caption
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 74,509
Dataset size notes: more than 2M
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Adaptability
New taxonomy: General Capability / Adaptability