Title: Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models
Benchmark: Multimodal ArXiv
Summary: Multimodal ArXiv consists of ArXivCap, a figure-caption dataset sourced from scientific papers, and ArXivQA, a QA dataset generated by prompting GPT-4V for QA pairs on ArXivCap entries. Results show that fine-tuning on these datasets boosts performance on the MathVista benchmark, and that evaluation results for various scientific plot comprehension subtasks are poor.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: comprehending scientific plots
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Purpose notes: The phenomenon is vaguely defined but the tasks are precisely defined
Task definition: Vision-to-text subtasks: caption a single (or multiple) scientific figure(s), including an in-context learning subtask, and generate paper titles given figures and captions.
Task item definition: A ground truth paper title and a list of scientific figures and corresponding captions
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 100,000
Dataset metadata available: Yes
Dataset metadata detail: arXiv domain, arXiv DOI
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "after training the model on QA pairs from each domain... Most domains hurt the Figure QA task. This suggests that synthetic Figure QA might not be the best benchmark for assessing realistic reasoning ability." (14373-4)
"our Multimodal ArXiv dataset sources from ArXiv papers due to their accessibility and open-source licenses. This approach may overlook the diversity of disciplines and data modalities present in the broader scientific literature." (14378)
Taxonomy: VQA / Understanding
New taxonomy: General Capability / Visual Understanding