Title: FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Benchmark: FIRE Bench
Summary: This paper introduces FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn conversations that are derived from 27 source datasets, empowering VLMs to spontaneously refine their responses based on user feedback across diverse tasks. The authors also develop the FIRELLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, and they show remarkable feedback-refining capability, outperforming untrained VLMs by 50%.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Feedback-refining capability of VLMs
Phenomenon defined by authors: Yes
Phenomenon definition: VLMs may sometimes produce undesirable outputs, possibly due to omitting important details in images or misunderstanding the instructions, which prompts the need for the feedback-refining capability beyond the normal instruction-following ability. This capability enables VLMs to spontaneously refine their responses based on user feedback, enhancing the efficiency and smoothness of interactions between users and visual assistants.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: FIRE dataset and FIRE benchmark consist of various datasets covering tasks including visual question answering, image captioning, complex reasoning, OCR, chart/table/document analysis, math problems, science question answering, etc.
Task item definition: Each sample consists of an image, a related question, the ground truth answer, and a multi-turn conversation spanning n turns. This conversation includes an initial response, textual feedback, and a refined answer generated by GPT-4o in response to the feedback.
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 11,000 samples
Dataset size notes: Train sets: 1M and 100K samples
Dataset metadata available: Yes
Dataset metadata detail: number of turns, responses lengths, score on the feedback
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: n-gram (BLEU, ROUGE, chrF), For dialogue assessment, they introduce four metrics: average turn (AT), average dialogue refinement (ADR), average turn refinement (ATR), and refinement ratio (RR).
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating