Title: Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
Benchmark: Multi-modal Self-Instruct
Summary: The authors devise a synthetic data generation pipeline to generate a visual QA dataset on abstract images, like charts, dashboards, and 2D layouts. They find that LMMs struggle on basic QA tasks, like reading analog clocks. However, finetuning on their synthetic dataset yields minor improvements, including some transferred improvements to related benchmarks like ChartQA and MathVista.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning with abstract images
Phenomenon defined by authors: Yes
Phenomenon definition: "these capabilities, i.e., perceiving abstract images and reasoning about visual elements, are essential for LMMs if we deploy an LMM- driven agent in our daily lives" (19229)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The model is given an abstract image, like a chart, and a question about it. Answering the question may require more than just reporting visual elements, e.g. route-planning, comparing features, or adding and subtracting tabular figures.
Task item definition: An image, a question about the image, an answer (or answers), and rationale for the answer. All of the text is generated by GPT-4, and the image is produced with python visualisation code generated by GPT-4.
Task definition detail: There are eight tasks, assessing very different capabilities (not just image processing but planning, mathematical reasoning, abstract pattern matching like ARC-AGI, ...)
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 11,193
Dataset size notes: 62,476
Dataset metadata available: Yes
Dataset metadata detail: task, answer rationale
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: Route-planning is evaluated as a structured response, but the model is not instructed to adhere to any format. The model gives the answer in free response, and these are post-processed (presumably by an LLM but unclear)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Landmark Coverage Rate (LCR(%)) for route-planning
Metric aggregation: Averaged, but unclear if weighted
Metric statistics: simple mean/sum, percentage point improvements
Metric subscores: Yes
Metric subscores detail: Subtask (chart, table, map, etc.) plus results after finetuning on various subtasks
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning