TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types
Authors: Jiankang Chen
Tianke Zhang
Changyi Liu
Haojie Ding
Yaya Shi
cheng.feng
Huihui Xiao
Bin Wen
Fan Yang
Tingting Gao
Di ZHANG
Venue: ICLR 2025 Poster (2025)
Keywords: instruction fine-tunning dataset
multimodal large language models
hierarchical task types
data generation pipeline
Primary area: datasets and benchmarks
TL;DR: We introduce TaskGalaxy, a large multi-modal instruction fine-tuning dataset containing over 19,000 hierarchical task types, which has demonstrated significant improvements in the performance of vision-language models across various benchmarks.

Abstract: Multimodal visual language models are gaining prominence in open-world applications, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task-specific data, leading to poor generalization and biased outputs. Existing efforts to increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only a few hundred task types. To address this, we propose TaskGalaxy, a large-scale multimodal instruction fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant question-answer pairs. Multiple models are employed to ensure sample quality. This automated process enhances both task diversity and data quality, reducing manual intervention. Incorporating TaskGalaxy into LLaVA-v1.5 and InternVL-Chat-v1.0 models shows substantial performance improvements across 16 benchmarks, demonstrating the critical importance of task diversity. TaskGalaxy is publicly released at https://github.com/Kwai-YuanQi/TaskGalaxy.

Published as a conference paper at ICLR 2025
TASKGALAXY:
SCALING MULTI-MODAL INSTRUCTION FINE-TUNING WITH TENS OF THOUSANDS VISION TASK TYPES
Jiankang Chen', Tianke Zhang, Changyi Liu, Haojic Ding, Yaya Shi, Feng Cheng,
Huihui Xiao, Bin Wen', Fan Yang, Tingting Gao, Di Zhang
Fan
Kuaishou Technology
(chenjiankang03, zhangtianke, 1 liuchangyi, yenbinn.ynngaar/kkuulshooo.
ABSTRACT
Multimodal visual language models are gaining prominence in open-world appli- cations, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task- specific data, leading to poor generalization and biased outputs. Existing efforts increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only few hundred task types. To address this, we propose TaskCalaxy, a large-scale multimodal instruc- tion fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant guestion-answer pairs. Multiple models are employed to ensure sample quality. This automated pro- cess enhances both task diversity and data quality, reducing manual interven- tion. Incorporating TaskGalaxy into LLaVA-v1.5 and Intern VL-Chat-- 1.0 models shows substantial performance improvements across 16 benchmarks, demonstrat- ing the critical importance of task diversity. TaskGalaxy is publicly released at bttps::/gge //ubbbb.. com cwm/Kwa -uaa--uuunnnnaaaaaaa...
models
INTRODUCTION
Recent breakthroughs artificial intelligence have been fueled with the development of a large
number of large multimodal models (LMMs) (Liu et al. [024bb Bai et al. 2023 Liu et al. 2024aa] Zhang et al. 2024c). These models are typically composed of a pre-trained visual encoder Radford al. 2021). pre-trained large language model (Touvron et al. 2023a). and a lightweight structure (Q-former for BLIP2 (Li et al. 2023c), two layers of MLP for LLa VA (Liu et al.] (024b)) etc.) connecting the above two, which have been adopted in various domains such as image captioning, object detection, visual question answering and other related fields. How to improve the model's performance in various mission scenarios is of great importance for deploying such model into an open-world system.
To enhance the performance of LMMs in specialized and general-purpose domains, mainstream ap- proaches focus on three key areas: optimizing model architectures (Alayrac et al. 2022 Wang et al.]
2023c, Liu et al. 2024a), improving training strategies (Wang et al. 2024 Sun et al. 2023 Banerjee et al. 2024 Zhu et al. 2024), and constructing high-quality data [uu et al. 2024 Chen et al.) 2023b Shi et al. 2024). While advances in model architectures
2023 Zhang et al. 2023a Li al.
and training strategies are crucial, our work focuses on addressing the critical challenges in the data domain. Current multimodal models typically undergo a biphasic training process: pre-training phase with image-text pairs for visual-textuse alignment, followed by supervised fine-tuning (SFT) phase with command- format data to refine multimodal abilities. However, the diversity of tasks in the pre-training phase is limited, affecting the generalization ability of visual language models (Xu let al.] 2003). Recent research (Hu et al.) 2024 Li et al.) 2024b] [Sh et al.) [2024) Xu et al.) 2023] Yin et al. 2024 Liu al. 2024b Zhao et al. 2024 has focused expanding task diversity in the
"This work was done while Jiankang Chen was interning at Kuaishou. *Corresponding author.

---

Published as a conference paper at ICLR 2025
NIVE
MNZ
M448endh
6345
1900
12.
NMMM
seMbarch
NMMU
MMMERA CN
SES0
POPE
53.1
Hilkdlocch
POPE
Bidd
Hy koleedd
LUNN Se
551
15.39
2504
Chinese Bench Lakkiii the
Chinese Bench
244
21.00
44.22
MMteet
Bonch
MMW
-20.7
O-Bench
1118
T0A
M2D
TIQA
AD0
Churtis
Nathiisa
MatiNiii
1- Sobrabay
baidialoyy
Figure I:
An illustration of the benefits of high task type coverage in TaskGalaxy for the SFT stage. We presented the performance of before and after integrating TaskGalaxy into the fine-tuning dataset.
and Intern Intern VL-Chat-v1.00-7 models, both
supervised fine-tuning phase to enhance instruction adherence and logical reasoning. Despite these efforts, instructional datasets still face limitations in task diversity, leading performance bottle- necks. For example, Vision-Flan (Xu et al., 2024) proposed comprehensive dataset with tasks like OCR and object detection but required extensive manual labeling and yielded only around 200 task types. Similarly, VisionLLM v2 (Wu et al. 2024) aggregated numerous task types but needed task-specific decoders, limiting the dataset's generalizability.
To address the limited task diversity in existing fine-tuning datasets and overcome current methodological constraints, we propose a novel pipeline for generating fine-tuning data. This pipeline enables the simultaneous construction of wide variety of task types using contemporary multimodal models (Hurst et al.) 2024] GLM et al. 2024] Chen et al. 2024b 2023c). We introduce an innovative supervised fine-tuming dataset, TaskGalaxy, which encompasses over 19,000 hierarchical task types, significantly enhancing task diversity in multimodal scenarios. Specially, to minimize manual involvement and effectively construct multi-level multimodal task type data for fine-tuning dataset generation, our data generation pipeline is divided into five main steps as follows: (1) initially designed several foundational multimodal graphic task types as seeds. These include tasks such as Optical Character Recognition (OCR) and Image Description for visual understanding, Logical Reasoning and Analysis for deep reasoning, well Multiple Choice Questions and Fill-in-He Blank Questions for satttion-and-nsseeeee formats and so on. We then design appropriate prompts to enable GPT-4o iteratively expand the range of task types both the same level and to generate new, more detailed task types at subsequent levels. Notably, the number of task types per level flexible and controllable, allowing for theoretically infinite generation. In our work, we have generated over 19,000 distinct types of tasks. (2) To provide image sources for all task types, we collected a diverse set of images from multiple open-source datasets. (3) We preliminarily select task types that are strongly correlated with each image using CLIP's (Radford) et al. 2021) similarity between image embeddings and text embeddings. Appropriate prompt for GPT is then designed further filter task types corresponding to each image.
(4) GPT-4o is utilized to generate question and answer pairs related to each task type for the selected images. (5) To ensure the reasonableness and alignment of the questions, task types, and images, we employ three robust open-source multimodal models GLM4v (GLM et al., 2024), Intern (Chen et al. 2024b) as judges for scoring and screening the
(Chen)
al., (202cc), and InterVL--Chat-v generated pairs. Finally. we generate a novel high-qualit fine-tuning dataset TaskGalaxy contain- ing 19,227 task types and around 410k visual Q&A samples. After fine-tuning L.LVAA-V..5-7B & 13B and data, the models showed an average improvement of 4.5 & 3.83, 3.0 & 3.64 points across fifteen benchmarks compared to the only original data fine-tuning, and Figure iillustrates the performance after incorporating TaskGalaxy during the SFT stage. Additionally, it demonstrated increase of 68 points for LLa 13B on the MME benchmark, indicating that our dataset enhances the model's generalization ability.
WL--Chat-v1.0-11
& 13B with the TaskGalaxy dataset along with raw fine-tuning
gains achieved by LLaVA-v1.5-13B and I Figure[illustrttes
2

---

Published as a conference paper at ICLR 2025
The contributions of this study are as follows.
We propose a novel multi-modal instruction fine-tuning dataset, TaskGalaxy, which con- tains tens of thousands+ of vision task types and approximately 413k samples, addressing the limitation of task diversity in existing datasets.
An almost fully automated pipeline for creating a comprehensive fine-tuning dataset of diverse task types was designed. This pipeline can be flexibly expanded by incorporating high-quality images, task types, and question-answer samples.
Incorporating TaskGalaxy into the fine-tuning of LLaVA-v1.5 and IntemVV-CCha-v 1.0 resulted in improvements across all 16 benchmarks compared to fine-tuning with the orig- inal data which proves that expanding the diversity of visual task types and high-quality question-answer pairs associated with these tasks significantly enhances the generalization capabilities of multimodal models.
2
TASKGALAXY DATASET
2.1 OVERVIEW
The TaskGalaxy dataset consists of 19,227 hierarchical task types, ranging from OCR and image description to fine-grained object recognition and complex logical reasoning and so on. These task types originate from a small set of manually defined seeds, further expanded by GPT-4o through prompt design. CLIP's graphical similarity is used to filter task types that best match specific im- ages. GPT-4o then generates corresponding question-answer pairs, which are evaluated by three open-source models to select the highest-quality samples. This process yields 413,648 high-quality question-answer pairs aligned with these hierarchical task types.
(o) Hierorchical Task Type Generation
(b) Image Collection
1-levelt
OCR
image description
]
[
Logical Rensoning
2-level:
{ our
webpagge
H
handantthe toxt ocr
H
activity busted
H
{
simple nostering
H
complex reasering
22 plle
besed
I
J
I
CFT-44
000
3-level
likk toxt extracticn
style resagaition
sports coere description
colar-ndated coption
spnticl reletioah
softwars and codine
geteration
Phaggg for b-ice takk hypc coparan 12/3.a
trange 2/5 leve
Froepe for 2/3 llvel tosh smeestice srhwet tyees
Shaneb
(4) Screening filtering
(d) Q&A Generation
(c) Match and Filter
Prompp for QSA Generation:
baak type. aucrtin, araversee
Your tork ic. for nech took iche. proposi qacition and crswer pair
Prompt for Screening Filtening:
dhoad
cua and
Yaar hade in fitnn on ingge and multimnds content peneenetrdrrggenaeet
mrnverr por comerpondig tark
teak lebel
tuok
libel determine whether the task libel and (uest on are satable for the image autable
eroaking tppn
Greee en max and maliple
Lobela Hovererr zone tual
201
ierge contart and this connot
Generets reltunet quectione....
Scan DiSoons
12
An overview of the task type and high-quality question-answer pairs generation pipeline for TaskGalaxy. We initially define the first level of visual task types, along with a small number of second and third level task types. Subsequently, we instruct GPT-4o extend these to broader range of task types. We then collect image modalities from existing publicly available datasets for matching task types with images, filtering, generating question answers related to task types, and utilizing the three referee models to obtain final high-quality visual quiz pairs for various task types strongly related to images.
Figure 2:
2.2 DATASET GENERATION PIPELINE
In this section, we outline the pipeline for creating the TaskGalaxy dataset, highlighting the reduc- tion of human intervention and leveraging multiple advanced L.Mss. The process encompasses task
3

---

Published as a conference paper at ICLR 2025
type expansion, precise matching between task types and images, automated question-answe answer gen- eration, and referee- filtering to ensure data quality, with the majority of these steps being fully automated. TaskGalax, stands out from other fine-tuning datasets due to its extensive visual task diversity, minimal human involvement, rich and high-quality task-related Q& A data, and scalability. The workflow is depicted in Figure
To ensure these key attributes of the GalxGa dataset, we have designed a five-step pipeline:
Prompt-task type Generation:
You are expert multimodal content understanding with extensive experience this field. I would like construct comprehensive task labeling system related to multimodal content understonding that includes only two modalities: images and text. In this system, the input consists of images and corresponding text, and the output in the form of text. wont to start constructing system from the first-lleel task categories, Currently, dozens first /leeel tosk categories have already been established by humans, which (rei(tas__type) Please expond and supplement with new first level categories that do not belong to the oforementioned categories scenarios. Output format: Each line should correspond to one category, without any other characters.
comprehensive coverage of categories multimodal content understonding
Figure 3: The prompt template used in GPT-4o API for first-level task type generation.
Hierarchical Task Type Generation. To address the challenge of expanding the range of visual task types in the multimodal domain, manual organization is impractical due to its time-intensiv nature. Instead, we leverage a robust multimodal model to automate the construction of diverse task types with minimal human intervention. Rather than randomly generating task types, we designed a hier- archical method. As shown in Figure 2 a), we began by manually defining a small set of first-level task types, such as OCR, Image Description, Logical Reasoning. Secondary and tertiary task types, such as OCR-webeage OCR, Logical Reasoning--Complee
Reasoning--softwarr and coding, and Detection~Anomaji Detection, were also specified. These initial task types served seeds, guid- ing GPT-44 to expand them at multiple hierarchical levels using specially designed prompts. The detailed prompt for first-level task type expansion illustrated in Figure 3] and descriptions of the prompts for secondary and tertiary tasks are included in the Appendix. These prompts have been carefully designed to mitigate the risk of overlap between different task types. Specifically, as illustrated in Figure 3. the instruction "Please expand and supplement with new primary-leve level cate- gories that do not belong to the aforementioned categories" refer to Table in the Appendix for additional prompts that generate non-overlapping task types) ensures this objective. This process resulted in the generation of 19,227 hierarchical task types, making Task the most diverse dataset of its kind. Further details on these task types provided in the Appendix.
Table 1: Summary of Image Sources for the Collection. We have curated a diverse array of image
sources, ranging from natural images (Lin et al. 2014 Kirillov al. 2023 Xu al. (024) to
specialized (Mathew et al., 2022; Lindström & Abraham 2022 Kafle et al. [018 Seo et al. 2015
Chang et al.] [202]] Lu et al. (202bb) and web-based images [Schuhmann et al. 2021 Ordonez
et al., 2211 2/hrrma et al. 2018). This selection aims to ensure the diversity and comprehensiveness
necessary to address various hierarchical task types effectively.
Name VLLaVA
Image Source
Samples
LAION, VFLAN
326K
Visual Genome
MSCOCO
108K
Math V360K
DocVQA, IconQA, UniGeo, CLEVR-Math, etc
31K
ShareGPT4V
COCO, SA, SBU,
358K
Image Collection. To align with the diversity and comprehensiveness of the hierarchical task types, as illustrated in Figure b). we collected approximately 800k images from variety of sources, as shown in Table I including ALLaVA (Chen et al. 2024a), Visual Genome (Krishna et al.) 2017). Math (Shi et al. 2024). and ShareGPT4V CCen et al. 2023b). These sources cover a range of tasks, such as object detection (COCO (Lin et al. 2014)), visual question answering (VFLAN (Xu al. 2024)), segmentation (SA (Kirillov al. 2023)), and document-related tasks
4

---

Published as a conference paper at ICLR 2025
(Doc (Mathew et al. 2022 Additionally, CLEVR-Math (Lindström & Abraham 2022) con-
tributes to mathematical -olving tasks, while LAION (Schuhmann et al., 20221 provides a broad array of web-sourced images, including artwork and watermarked content. All collected im- ages maintain their original resolution. Additionally, to provide further insight into the image data, the sample size statistics for all image source datasets are provided in the Appendix. Our highly scalable data pipeline can accommodate diverse image types, requiring only a single modality, thus highlighting its flexibility and adaptability.
Task Type: image description-location beseedddedddddeeeeeeeeeee cityscapes
Question: Con you provide a detailed description of the cityscape in the image?
The image shows panonamic of cityscape with prominent clock tower in the foreground. The city characterized by densely pocked buildings with red-tiled reofs and mix of modern and histonical architecture, In the background, a range of mountains is visible, indicating the city surrounded by natural scenery.
Answer:
Task Type: suggestions-sports safety
Question: Bosed on the image, what general sports safety suggestions can be recommended?
Answen: General sports safety recommendations include previding participants with proper,
fitted protective, gear such as helmets and gloves, enauning all children are visibly marked for idontification, maintaining clear and supervised skiing area, offering qaik access to first aid, and educating children skiigg safety and etiquctte before allowing them the slopes.
Tesk Type: storytelling based on image scenes-imag -mage descriptiot-histooicall context
Question: Create short story based on the image scene, incorporating historical context.
Answern In the bustling colorial city of Solgon, the main thoroughfore was alive with the clatter of rickshaws bearing their humon cargo. The grand colanial edificas stood as silent witnesses to the daily ebb and flow of life. Amid the sea of conical hats, Linh, ambitious young side the street symbolized power and authority, a place where Linh hoped to one day step inside not as servant but os a respected businessmon Life in the colonial city was a delicate dance between espiration and reality, shaped by the intricate topestry of cultural inf luences and the enduring human spirit.
dreamed of day
owning own rickshaw flect.
imposing government building on
Task Type: ocr-logo necognition-log text association
Question: What text is associcted with the loga?
Answer: The text associated with the logo is 'BC PB and '8380Л THCTMABOIMBBPOKOHHHRRPPPSS
Figure 4: Sample images, task types, and Q&& in TaskGalaxy The Task Type refers to the visual task related to the image. Question and Answers are generated by GPT-4o and subsequently filtered by three refereeing models.
Match and Filter. Given the complexity and labor-intensive nature of selecting images for a wide range of hierarchical task types, we utilize the vision language model CLIP (Radford et al. 2021) renowned for its robust text-to-image alignment capabilities, to streamline the initial screening pro- cess. CLIP is utilized to identify task type names that best correspond to each image, improving both efficiency and accuracy in matching. As illustrated in Figure 2c)). for any image Iy and task type text tj. we pass them through CLIP's image encoder and text encoder T to obtain their respective embeddings. The match between the image and the task type is then calculated using the following formula:
(1) where- denotes the cosine similarity between image embedding and text embedding, x; from image pool, ty from task type pool. To identify the most appropriate task type for given image, we initially select the & task types that exhibit the highest degree of similarity and & = 10 our pipeline. These selected task types form the preliminary sequence for that image.
$j = -(zz)) T(t,),
Given that task types with the highest l similarity have been initially assigned to each image, some mismatches may still persist. To further refine and filter the task types that are more relevant to the image, we use a specially designed prompt, detailed in the appendix, to guide GPT-4o to further analyze and filter the task types from the initial sequence, identifying those that most closely match the image. The resulting list of task types will serve as the final selection for subsequent processes. Examples of images and their corresponding task type lists are shown in Appendix.
Q&A Generation. A key factor in improving multimodal models for visual consstio-answweinng tasks is high-quality training data with diverse question- pairs. To address this, we developed
5

---

Published as a conference paper at ICLR 2025
high-quality image-matched matched task types, illustrated in Figure d). We designed specific prompts, detailed in the appendix, to guide GPT-4o in generating questions and answers based on image content and task-type text. This approach ensures diverse question-ansee answer pairs that cover broad spectrum visual Q&A scenarios. The resulting data provide a comprehensive training resource, enabling the model to learn from wide range of scenarios and enhance its capabilities.
Referee Screening. Despite GPT-4o's superior performance, may still encounter mismatches between task types, questions, and images. To improve alignment and manage the high costs of closed-source APIs, we used three high-perforning open-source multimodal models (Chen) al., 2023c 2024b GLM et al., 2024) to evaluate task types and questions for each image, as shown in Figure ee.. Following the principle that "three heads are better than one," these models assessed each task type and question relative to an image on binary scale, award- ing one point for match and zero otherwise. We integrated these scores, selecting only those with cumulative score of two or more for inclusion the TaskGalaxy dataset. This pro- cess enhances the accuracy of task type, question, and image matching. To ensure balance across task types, we randomly selected I-55 samples per task type from the final dataset.
Figure [5] illustrates the distri- bution of sample counts across different task types. To pro- vide more comprehensive derstanding of TaskGalaxy, we analyze only the proportion of samples in each task type but also the hierarchical dis- tribution of task types across layers. Specifically, the ratio of task types in the first three layers follows a 1:2:3 pattern, corresponding
41-55
(27.90%)
1-10
(50.1%)
21-40
(10.1%)
1-10
11-20
21-40
11-20
41-35
(11.1%)
115, 2796, and 14,370 task types, respec- tively. TaskGalaxy is a high- quality, supervised fine-tuning dataset covering a diverse range of tasks. Figure presents sam- ple images along with their asso- ciated task types, questions, and answers from TaskGalaxy.
Figure 5: Distribution of the number of images across the 19,227 task types in TaskGalaxy. The ranges 1-10, 21-40 and etc. indicate the number of samples associated with different task types in TaskGalaxy. The corresponding ratios represent the pro- portion of task types that fall within each specified sample range.
Our task types and image data are designed to be expandable. The TaskGalaxy dataset can be au- tomatically updated with additional images and task types through the fine-tume -un-ttning data pipeline de- scribed earlier. This scalability facilitates future iterations, offering opportunities for further updates. To assess the characteristics our dataset in relation to previous studies, we provide a comparative analysis between our dataset and previous work in the Appendix. And to comprehensively present properities TaskGalaxy, a detailed dataset card available at TaskGalaxy Data Card (PDF)
EXPERIMENT
3.1 EXPERIMENT SETUP
Model Architecture. In the matching and filtering stage, the CLIP-L/I RRadfood et al. (2021) model, developed by OpenAl, employed. This model utilizes a ViT-L/14 Transformer architecture as the image encoder and masked self self-attention Transformer as the text encoder. In the stage of evaluating the TaskGalaxy dataset, We use the LLaVA (Liu et al.) 2024b) and Intern VL-Chat-v 1.0 (Chen al., 2023cb models. Both models feature pre-trained visual encoder and a large language model, linked by a two-layer MLP projection layer. LA employs a two-stage training: initially, a subset of the CC3M (hhamna al. 2018) dataset pretrains only the projection layer for multimodal alignment. We use the model pre- traited this phase as the basis for fine-tuning in the subsequent phase. For validation, selected variants: L.L and L.L.AA-1... 1-11BBB. Similarly, Intern

---

Published as a conference paper at ICLR 2025
Chat-v1.0 undergoes two training phases: first, the MLP layers are trained with the 558K Liu et al., 2024b) dataset, followed by training the language model with the LLaVA-Mix LaaVA-MMx-666K Liu et al. 2024a) dataset.
Galxy Finetuning. We incorporated our TaskGalaxy dataset with the original supervised fine- tuning data organized by LLaVA and are gathered from publicly available academic taik-oriented datasets (Kazemzadeh et al. 2014, Krishna et al.; 2017 Marino et al., 2019 Mishra et al., (019)
during the second stage of model training for both models. In this stage, the visual encoder weights were frozen, while the projection layer and large language model weights were fine-tuned.
Benchmarks. To design a comprehensive instruction fine-tuning dataset for various vision tasks, we selected 16 benchmarks to evaluate model performance. These include MME-Perception (MME) Fu et al.] ) which measures perception abilities across 14 subtasks: MMBench (MMB) Liu et al. 2023)_ a multi-choice test covering all proficiency levels: MMBench CN 2.23), the Chinese version of MMBench; POPE <Li et al., 2023e), which assesses illusion using subsets of COCO (Lin et al. (0014) (random, normal, adversarial); VLa-A- the-wild (LLaVA ") (Liu et al. 2024b) and MM Vet Yu et al., 2023), evaluating visual conversation abilities and leveraging GPT-4o for response evaluation; VQA (TQA) (Singh et al., 2019), fo- cused on text-related visual question answering, ScienceQA (SQA) et (022a). with multiple- choice questions on scientific topies requiring visual answers; Math Vista (Lu et al. (023), assessing mathematical reasoning in visual contexts; ChartQA (Masry et al. 2022). evaluating visual and log- ical reasoning over charts; A121) (Kembhavi al. 2016), for diagram interpretation; Q-Bench (Wul et al. 2023). testing low-level visual perception and understanding: Chinese-Q-Bente (Q-Bench "N), the Chinese version of Q-Bench; HallusionBench (HalluBench) (Guan et al. 2024), focusing on vi- sual and knowledge illusions; SEED-Bench (SEED) (Li et al. (202bb). evaluating performance on both images and videos (with video accuracy assessed using sampled frames from VA); and MMMU Yue et al., (024), which tests multimodal models on large-scale multidisciplinary tasks requiring advanced subject matter knowledge and reasoning skills.
(MMB (Luu et al.
Implementation Details. For LLaVA-v1.5, ., we utilize the pre-trained projection layer weights from LL to fine-tune both the projection layer and the large language model (LLM). This fine- tuning is conducted using the 665k supervised fine-tuning data from LLaVA, supplemented by our TaskGalaxy dataset. We use two LLaVA architectures, Vicuna- 13B v v1.5 and Vicuna-7B v1.5 (Chi-) ang et al. 2023), combined with CLIP-VIT-L-3360
(Radford et al. 2021 and two layers of MLP as our visual-language models ( (VLMs). During instruction tuning. we fine-tune the MLP layer and LLM using 8 A800 GPUs, with an initial learning rate of 2e-5, batch size of 16 per device, for epoch, totaling approximately 9300 steps. For Intern VL-Chat-v 1.0, we first train the MIP layers with the L.GS-558K dataset using 8 A800 GPUs, with initial learning rate of 1e-3 and batch size of 32 per device. Subsequently, we fine-tune the MLP layers and the LLM with both the raw fine-tuning dataset and the TaskGalaxy dataset, using the same hyperparameters as for LLaVA. We employ two Intern VL--Cat-v1.. 1.0 architectures, Vicuna-7B 7B and Vicuna- 13B, along with Intern ViT- 6B (Chen et al. 2023c).
3.2
QUANTITATIVE COMPARISON
Table 2 presents a quantitative comparison of LLaVA-v1.5 and Intem 1.0 models trained on the original fine-tuned data versus those fine-tuned with TaskGalaxy. The new models show improvements of 4.5 and 3.83 points across all 15 benchmarks for LLaVA-vI.5-7 ----7B and 13B, re- spectively, excluding MME. For IntemVL-CChatvvll O. the improvements are 3.0 and 3.64 points. It worth noting that LLa VA-11.15--11 sees a performance increase of 68 points with TaskGalaxy on the MME Benchmark.
Taking LLaVA-v1.5-777 as an example, we observed a 3.35% and 3.1% improvement over the origi- nal baseline on MMBench and MMBench_CN, respectively. For LLLVA--inthhe-iilee we achieved a 3.3 points increase, demonstrating that the TaskGalaxy fine-tuning dataset enhances the model's 's per- formance in detailed description and complex reasoning tasks. Notably, incorporating TaskGalaxy resulted in improvements of 0.77, 1.75, 4.7, 5.48, and 12.94 points on TQA, SQA, Math Vista, ChartQA, and AI2D, respectively, highlighting the dataset's broad coverage. On hallucination mit- igation tasks, improvements of I to 2 points on POPE and HalluBench suggest that a diverse range of tasks helps address hallucination issues. Additionally, the SEED benchmark, which includes
7

---

Published as a conference paper at ICLR 2025
Table 2: Elucidate the advantages of incorporating our Taskya dataset into SFT phase. We present the performance comparison on existing representative benchmarks before and after in- corporating TaskGa The results in the Baseline represent our re-implementation of the officially provided checkpoint. All the numbers are presented in % except MME and the full score is 100%. The indicator of MME the perception score, the maximum value is 2000.
Model
Method
MME
MMB
MMB
POPE
Benchmarks
LLVAW
MMWa
MathNiiia
Riseling
150.
64.69
R39
BaselineeTaskkaaayy
1833
68.04
61.17
86.7
56.3
29.9
58.98
71.26
314
L1aVA-V1.--7
Biseline
Chungg
14.72
A220
25.32
26.08
U-Bench
13.55
Hillabench
50.05
SEED
58.39
MMMU
15.5
Average (odl MMIE
44,46
JuselineTTaskallan
20.20
38.26
43,58
34.8
51.74
60.28
21.8
48,96
MME
MMB
PUPE
LLaVA"
MMY
T0A
50A
MathVista
Reseline
1532
68.47
86.03
62.6
522
61.25
71.00
28.
LLAVA-v1.-118
Baseline+ Gasllaay
ChunQA
1600
A120
69.85
0 Bench
64.43
Q 0-Bench
86.30
Hallabench
63.1
SECD
34.4
MMMM
61.95
73.33
Average (sslo MMGG
33,3
Baseline
15356
21.13
19:05
20.5
40.84
6031
7.5
45.21
Baselingt Taskknne
23:44
29.16
23,06
53,21
61.22
17.8
49.04
MME
MMB
MMB"
POPE
LLAVA
MMN3
TQA
S0A
MathVista
Buseling
1500
65.29
57.28
86.07
516
26.2
56.65
66.4
27.7
lunenVL-Chiz-IIl 10-771
Basciine-TaskGalaxy
1532
67.10
60.91
86.97
54.8
30.6
57,51
70.93
30.8
ChanQA
A120
Q-Rench
O-Bench"
HailoBench
SEED
MMMU
Average (uolo MME)
Baseline
15:00
35.96
14.75
44.85
52.55
5929
27.0
47.79
Baselinee JaskGallayy
18:24
38,57
48.56
48.16
53.37
60.44
34.9
50.79
MMF
MMB
POPE
LIaVA
MMYa
TQ4
50A
MathVista
Basline
1525
63.64
60.31
86.00
51.7
25.7
$6.94
20.12
25.7
Iotem\L-CChu-v1.o 1-11BB
Baseline Jaskiilaay
1534
69.50
63.14
86.43
52,9
32.0
59.51
72.72
30.5
CanQA
A12D
Q-Rench
O-Rench
HalloRench
SEED
MMMU
Average (uil MMF)
Baseline
16.28
38.55
56.13
$1.86
50.05
59.35
25.1
19:90
Baseline Tasktinlaxy
17.04
52.60
58.60
58.36
$2,90
60.23
36.8
53.54
12 evaluation latitudes and 19k questions, there was modest 1.7 points improvement over the model trained solely on raw fine-tuned data. In low-level image evaluation tasks, Q-Bench and Chinese- Baresee---Beccie Q-Bench, models tnn-ttned with TaskGalaxy showed gains of 17.5 and 1.27 points, re- spectively. The improvements on the challenging MMMU benchmark, where the model showed a 5-point gain, are attributed to the inclusion of diverse task types, including math and humanities. We also observed 27-point improvement on the MME benchmark. Across the 15 benchmarks excluding MME, TaskGalaxy led to an average improvement of nearly points. Similar results were observed with the lowing Galaxy fine-tuning. These improvements were consistent across VL-ernV--Chattv To further demonstrate the effectiveness of TaskGalaxy, we provide additional comparative experi- ments which involve fine-tuning the aforementioned models, as well as more robust models, using TaskGalaxy alone and using other similar instruction-tuning datasets separately in the Appendix. The results are consistent with the findings presented earlier.
which achieved remarkable 68- 68-point gain on MME fol-
Our results demonstrate that TaskGalaxy, with its exceptionally broad coverage of task types, sig- nificantly enhances the performance of multimodal models across a wide range of tasks, offering valuable insights for the research community regarding task type diversity.
3.3 ABLATION STUDY
The number of task types. The primary objective of TaskGalaxy is to enhance the generalization capabilities of multi-modal models by encompassing broad array of visual-language task types. TaskGalaxy includes a diverse set of 19,227 distinct task types. In this subsection, we examine how the number of task types affects the performance of multimodal models. We selected task types in increments of 2k. 3k, 5k, 10k, 15k, 18k, and 19,227 from TaskGalaxy maintaining constant total of 100k images, and conducted ablation experiments using the LLaVA-v1.5-7B model. As shown in Figure Gleeft), benchmarks such as I.LaVA-in-the-will ChartQA, A12D, Q-Bench, and MMMU consistently improved with an increasing number of task types. The Averagge' performance across the 15 benchmarks, excluding MME, also shows clear trend of enhancement with more task types, which is corroborated by MME performance changes in Figure 7 These results highlight the critical role of task type diversity in enhancing the capabilities of modern multimodal models.
The number of samples. In addition to the impact of the number of tasks, it is well established that the amount of sample data in the instruction fine-tuning dataset also significantly affects model performance. To investigate this, we conducted ablation experiments to assess the effect of varying data volumes on model performance. As depicted in 6 we included all task types and
8

---

Published as a conference paper at ICLR 2025
Chanke
Becc
uto
Bood
an0
14
19227
Figure 6: Plot showing the change in baseline performance with variations in the number of tasks and the total number of samples. Left: The effect of varying the number of task types (ranging from 2k to 19,227) on model performance, while maintaining a constant total sample size of 100k. Right: The impact on model performance of varying the number of samples across all task types, ranging from a maximum of 5 to 55 per task type (resulting in a total sample size from 76k to 413k), while keeping the number of tasks constant at 19,227.
controlled the variation in total sample size by setting the maximum number of samples for each task type, ranging from 5 up to 55 which corresponds to the final TaskGalaxy dataset. The results show that for benchmarks such as MMVeT, ChartQA, AI2D, and Q-Bench, as well as the average perfor- mance across 15 benchmarks excluding MME, performance generally improves with the increase in sample size. However, for that the optimal sample size may vary depending the specific benchmark. Nonetheless, there is clear overall trend of increasing model performance with the increase in data volume.
performance peaks at 281k samples, suggesting
4 RELATED WORK
Large Multi-modal Models. With the rapid advancement of large language models (LLMs), such GPT-3 (Brown et al.
1550
1540
2020 LLama2 (Tovrron et al.
MME perception Score
2023b), InternLM (Team
1530
2023 and Baichuan 2 (Yang al. 2023), there has been growing focus on integrating visual knowledge into LLMs, exemplified by models like CLIP RRdfford et al.] (202I).
1520
1510
1500
1490
BLIP (Li et al. 2022). and et al. 2023c). While these models exhibit strong performance in graphic align- ment and image captioning. they continue to face significant challenges in handling more complex visual question an- swering tasks. To enhance the model's instruction adherence and content understanding vi- sual question answering (VQA), visual instruction fine-tuning strategies have garnered increasing attention in the training of large multi-modal models. For instance, models like LLaVA (Liu al. 2024b), MiniGPT-4 (Zhu al.) 2023), and InstructBLIP (Dai et al. 2023) leverage large language models from the GPT4 (Achiam al. 2023) family to generate fine-tuned instruction data, thereby enhancing performance in com-
BLIP2 et
1480
baseline
2ktask
aktask
sktask
10ktask
15ktask
1ektask
19227task
baseline
max
max 10
max 20
max 30
max 55
max
Figure 7: Plot illustrating the variation in MME Perception performance scores in relation the number of task types and the total number of samples. The left bar represents per- formance across different numbers task types, while the right bar represents performance across varying total sample sizes.

---

Published as a conference paper at ICLR 2025
plex VQA scenarios. Furthermore, to expand the range of VQA task scenarios, recent models such as LAMM (Yin et al. 2024) and MIMIC-IT (Li et al. 2023a). following the example of LLaVA, have extended their VQA capabiliities to encompass 3D scenarios, multi-graph tasks, videos, and other complex domains. Recently, a series of open-source large multi-modal models with enhanced performance have consistently outperformed existing benchmarks. Notable examples include (Zhang et al.) 2024b), and M2 (Zhang et al. 2024c), which are leading the field in various multimodal tasks. In addition to open-source models, recently developed closed-source models such as GPT-4v [OpenAI]20233 GPT-4o (Hurst et al. 2024). and Claude-3.5 (Anthropic) 2024) continue to lead the field, often matching or surpassing open-source models in various VQA tasks. To facilitate comprehensive improvements in the performance of open-source models across a wide range of Visual Question Answering (VQA) tasks, we construct fine-tuning datasets featuring diverse task types based on multiple LMMs, while also leveraging closed-source models. This approach aims to enhance the performance of open-source models in various tasks.
GLM-4v (GLM et al. (024). Qwen-VL (Bai et al. (2023),
Multi-modal Instruction-Tuninee Datasets. Data-driven instruction fine-tuning strategies have be- come increasingly crucial in the training of multimodal models. Recent works have introduced high-quality instruction fine-tuning datasets designed to enhance models' visual question-answering capabilities. Among these datasets. Multilnstruct (Xu et al.] 2023) is the first manually labeled multimodal instruction tuning benchmark dataset, encompassing 62 different multimodal tasks. Mini-GPT4 (Zhu et al., 2023) constructs an costruction-oolloiinn dataset by combining image-text datasets with manually crafted instruction templates. LLaVA (Liu et al., (024b) utilized the cap- tions from the COCO dataset and the contents of the bounding boxes, sending them to GPT-4 to construct an instruction fine-tuning dataset comprising approximately 150k samples. Similar to LLaVA, LAMM (Yin al. 2024) used the GPT API to generate command rrepoonse pairs for col- lected images and point clouds, resulting in approximately 190k graphic command-responss pairs and 10k point cloud command-response pairs. However, these instruction fine-tuning datasets do not emphasize the concept of task types specific to VQA scenarios and lack diversity in the range of task types. Considering the concept of task types, VisionLLM v2 (Wu et al.] (2024) aggregated hun- dreds of task types of multimodal data based on existing academic datasets but it requires the design of specific decoders for different tasks, which limits the generalizability of the dataset. Recent work VFLAN (Xu et al., 2024) enabled experts to construct 187 task types based existing datasets, resulting in a large fine-tuning instruction dataset containing approximately 1,664k samples. How- ever, this approach requires significant specialized manpower to annotate the extended tasks and generate the associated task structures, making it both time-consuming and labor-intensivy tionally, despite the effort, the dataset covers only around 200 task types. In contrast, we developed TasaCalaxx, high-quality instruction fine-tuning dataset, guided by the principles of maximiz- ing the coverage of hierarchical task types for VQA while minimizing manpower investment. We successfully generated around 20,000 hierarchical task types and 410k VQA samples. Integrating TaskGalaxy into multimodal architectures like LLaVA and InterV resulted in substantial performance improvements.
Addi-
CONCLUSION
In this study, we present TaskGalaxy, a multi-modal instruction fine-tuning dataset comprising ap- proximately 20,000 multi-modal task types and around 410k instruction Q&A samples. Addition- ally, we propose a pipeline for the systematic construction and generation of a diverse range of task types and corresponding high quality instruction Q&A samples. This approach addresses the limitations of existing multi-modal instruction fine-tuning datasets, particularly the narrow scope of task types and the excessive reliance on human intervention. TaskGa encompasses an exten- sive range of multimodal visual Q&& tasks and offers highly extensible pipeline that facilitates the addition of new task types and the generation of high-quality fine-tuned instructional data. We 1.0 models using TaskGalaxy, resulting in a signif-
fine-tuned the LLa and InternVL-Chai icant improvement compared to using only raw fine-tunning data, respectively. Adequate empirical evaluation confirms the effectiveness of our broader task type data in enhancing the performance of multimodal models, highlighting the critical importance of task type diversity in the instruction fine-tuning dataset. We hope that our approach of constructing a dataset with a broad range of task types and reduced manual labor will guide future development of multi-modal instruction fine-tuning datasets and we plan to make the dataset publicly available for community research.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
This study upholds rigorous ethical standards to ensure the credibility, confidentiality, and respon- sible usage TaskGalaxy, a large-scale multimodal dataset designed for fine-tuning multimodal models. While the dataset does not directly involve human subjects, we developed it with care- ful attention to ethical considerations in data collection, processing, and dissemination. All data used in TaskGalaxy underwent thorough de-identification procedures protect privacy and main- tain anonymity. The images are sourced from publicly available, open-source datasets that comply with relevant copyright and usage laws, and we have clearly documented the licenses for these im- ages. Additionally, we have provided a comprehensive license for TaskGalaxy itself, ensuring proper usage and distribution. The associated text generated by GPT-40, a model designed with safety guidelines to produce content that aligns with ethical standards. We maintained transparency and accountability throughout the dataset creation process, full compliance with legal frameworks.
As TaskGalaxy is generated using automated models, we acknowiedge the potential for biases in these models, such as GPT-4o and CLIP. To address these concerns, we employed multiple models and advanced filtering techniques, significantly mitigating the impact of these biases. Additionally, we encourage users to remain mindful of potential bias when applying the dataset in real-world scenarios. We are committed to continuously refining the dataset over time further reduce bias and promote fairness. We have been transparent in documenting the creation and methodology of TaskGalaxy, sharing detailed information about the dataset's s construction, filtering process, and quality control measures with the open-source community. This transparency allows for community feedback, collaborative improvements, and ensures the dataset used in ways that align.
Throughout the development of TaskGalaxy, we adhered to rigorous ethical guidelines, prioritizing the rights and autonomy of contributors, and ensuring no academic misconduct occurred. We affirm our commitment to upholding ethical research practices and addressing any unintentional errors or oversights promptly. By releasing TaskGalaxy, we aim advance research in multimodal learn- ing while carefully considering its potential ethical and social impact, encouraging the academic community to use the dataset responsibly and aware of the broader consequences in society.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 23Xv:2303.08774, 2023.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Anthropic. Claude 3.5 Sonnet, 2024. Available at -laude-3-S-sonnee (accessed on February 14, 2025).
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arxiv:2308. /2966, 2023.
Oishi Banerjee, Hong-Yt Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, and Pranav Rajpurkar. Direct preference optimization for suppressing hallucinated prior exams in radiology report gen- cration. arXiv preprint arXiv:2406.0644 2024.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering choropleth maps. arXiv preprint arXiv:2211.08544 2022.
11

---

Published as a conference paper at ICLR 2025
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpttv-synthesized data for a lite issoo-language model. arXiv preprint
2tt.:2222...6888 2024a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint 1rXiv:2311.12793, 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. 14238, 2023c.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404.168// 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%% a chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purposs vision-language models with instruction tuning, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Team GL.,, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatgim: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.11 12793, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: simple multimodal I1m for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2256-2264, 2024.
Aaron Hurst, Adam I Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5656 2018
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pp. 787-798, 2014.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Visi 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part IV 14, pp. 235-251. Springer, 2016.
12

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International joternatiionaJuuraalll of computer vision, 123:32-73, 2017.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.0555 05425, 2023a.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal IIms with generative comprehension. arXiv preprimt arXiv:2307.16125. 2023b.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interlleae: Tackling multi-image, video, and 3d large multimodal models. arXiv preprint urXiv:2407.07895, 2024b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19742. PMLR, 2023c.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M it: A large-scale dataset towards multi-modal multilingual 2023d.
struction tuning. arXiv preprint arXiv:22306.0433
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 2023e.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208. 05358, 2022.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 26296-26382 -6306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.066 06201, 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song- Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-252. 2022a.
13

---

Published as a conference paper at ICLR 2025
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv 14610, 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310002255, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emest Valveny, and CV Jawahar. Infographievga. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 20/9 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
Chancharik Mitra, Brandon Huang, Trevor Darrlll, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420-14431, 2024.
OpenAl. GPT-4V(ision) System Card, 2023. Available at gpt 1ytt-4v-50 10-4--yyye syy-syste---o card (accessed on February 14, 2025).
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02112 2021.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466-1476, 1476, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymee, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-220 2565, 2018
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.1722 17294, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rihf. arXiv preprint arXiv:2309. 14525, 2023.
14

---

Published as a conference paper at ICLR 2025
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer. Xichen Pan, et al. Cambrian-1: fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 19,,,,,,199,,222222
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- 2023b.
tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922:
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arxiv:2406. / 1839, 2024.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for large language models. 2023.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, al. Q-bench: A benchmark for general-purpor foundation models on low-level vision. arXiv preprint arXiv:2309. 14181, 2023.
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.088: 2024.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct Improving multi-modal zero-shot learning via instruction tuning, 2023.
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint 24X2 1690, 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprini arXiv:2309.10305 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 2023.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang. Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction- tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.024999 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Daoan Zhang, Jumming Yang. Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of- of-thoughtt prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401 025882, 2024a.
15

---

Published as a conference paper at ICLR 2025
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang. Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internim Internlm-ccomp xcomposer. A vision- language large model for advanced text-image comprchension and composition. arXiv preprint arXiv:2309.1511 15002. 12, 2023a.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang. Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang. Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. mneemmmm-ccomppser-25555 versatile large vision language model supporting Cong-contextual input and output, 2024b.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm- wwwplll-xxooooerr-222 A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.0332 2024c.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal Shain-of-thougeee reasoning in language models. arXiv preprint arXiv:2302.0092.. 2023b.
Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.004::: 2307.00077 2023.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei
Wu. Retrieval-augmeente mixture of lora experts for uploadable machine learning. arXiv preprint arXiv:2406. 16989, 2024.
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168 36:5168-5191, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint M.Xi:23304.1092. 2023.
Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. ICML 2024, 2024.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 OVERALL REVIEW OF IMAGE SOURCES
Considering the accessibility of data sources and the task-related nature the image data we aim mine, we have opted for open-source image data. The approximate data sources and their corre- sponding sample sizes are presented in Table of the main text. To provide further insight into the image data, the Table A-1 presents the statistics of the sample sizes for the different data sources collected.
Table A-1: Statistics of the sample sizes for the different data sources collected.
Dataset
Sample Size
Dataset
Sample Size
UniGeo
1,507
GeoQA+
2, 2,146
GEOS
64
VizWiz
749
CLEVR-Math
590
PlotQA
612
VAQ2.0
2,327
TQA
1,137
Doc
2,672
FigureQA
1,970
Ge
1,182
MapQA
590
VQA-AS
658
A-OKVQA
3,391
VQA-RAD
242
PMC-VQA
4,208
TabMWP
2,538
Super-CLEVE
971
IconQA
2,560
DVQA
1,660
allava_laion
145,359
allava_vflan
181,393
Visual Genome
108,249
wikiart _images
62
oer_vqa_images
12,614
coco 1oco_train2017
118,326
web-celebrity images
67
web-landmark images
57
3,485
sam sam_images
223,720
share textvqa.imaae
55
A.2 MORE DETAILS ON THE DENERATION PIPELINE PROCESS OF TASKGALAXY
About Prompt: the TaskGalaxy dataset pipeline, the first step involves using GPT-4o to continu- ously expand new task types based set of human-defined task type seeds. This process requires designing distinct prompts for different levels of task types and determining whether lower-level task types exist within each hierarchy, allowing GPT-4o to systematically expand and populate the dataset. For level-1 task types, we focus solely on extending the existing prompt, as detailed in the main text. For generating two-level task types, the approach varies depending on whether the one-leveltask types have corresponding two-level tasks; prompts are designed accordingly to either continue expansion or generate new prompts. The same methodology applies three-leval tasks. The detailed prompts are provided in Table
After generating large number of hierarchical task types and collecting substantial amount of open-source image data, the third step in our pipeline involves matching and filtering. Following the image-text cosine similarity matching conducted by CLIP, we proceed to further refine the selection of task types that are most compatible with specific images. In this stage, we employ a specially designed prompt for GPT-40, denoted as p_ f_iitter Table A-3 to filter and select the task types that best match each particular image.
After completing the matching and filtering of task types and images in the third stage, the fourth stage involves generating question-answer pairs related to all task types matched with the images. The prompt templates guiding GPT-4o to generate these task-type-related answer texts are denoted as p- A in Table
After generating all the Q&A samples related to the task types, to further refine the selection of task types and questions that best match the images and ensure higher quality, we employ three open-source models in the final stage. This step focuses on filtering the images, task types, and their corresponding questions to identify the most suitable and coherent matches, while also considering
17

---

Published as a conference paper at ICLR 2025
cost-effectiveness. The prompt templates used for this filtering process by the three open- models are listed as p-openfilter in Table A-3
In addition to the above prompt templates, in the ablation experiments, aimed to verify the effectiveness of Chain-of-Thouu Thought (CoT) under multi-task type conditions. For a dataset with a total size of 76k, we prompted GPT-4o to articulate the reasoning process, guiding it generate CoT answers. The prompt templates used for this purpose are denoted as p. P_CoT in Table A-3
Table A-2: Different prompt templates were generated for the layered task types in the first stage. Here, p2.w denotes the continuation of expansion for a one-level task type that includes a two- level task, while p2.w/o represents the direct generation of two-level task for one-level task type without an existing two-level task. Similarly, p3 w indicates the continuation of expansion for a two-level task that includes three-level task, and p3 w/o represents the direct generation of three- level task for a two-level task type without an existing three-level task. The placeholder {-..) denotes the corresponding task type field be populated.
Type
Prompt
You are expert in multimodal content understanding with extensive experience in this field. want to construct comprehensive lask label system related to multimodal content understanding. which only includes image and text modalities. In this system, the input image and the corresponding text, and the output is the form of text. The primary and secondary categories 'n'. The task name of the primary category that needs to detailed currently is {....,, and multi- ple secondary categories have already been established manually for this primary category, which are: {...}- Please supplement other categories that do not belong to the aforementioned secondary categories to cover all task scenarios under the primary category of multi-modal content understand- ing. Output format: Each line corresponds to one task category, without any other characters, and different levels of task categories are connected by '~'".
connected by
p2.w
You are expert in multimodal content understanding with extensive experience in this field. want to construct comprehensive task label system related to multimodal content understanding, which only includes image and text modalities. In this system, the input image and the corresponding text, and the output in the form of by 'N"". The task name the primary category that needs be detailed currently is {...). Please expand the secondary task categories under this primary task category to cover all tasks included in this primary task category in the context multimodal content understanding. Output format: Each line corresponds to one task category, without any other characters, and different levels of task categories are connected by '~',
The primary and secondary categories are connected
p2 w/o
You are an expert multimodal content understanding with extensive experience in this field. want construct comprehensive task taxonomy for multimodal content understanding that in- cludes only two modalities: images and text. The input to this taxonomy will he images and their corresponding text, and the output will he in text form. The primary, secondary, and tertiary cate- gories this laxonomy are connected by '~". Currently, the task name for the secondary category that needs to be detailed is {...}. and several tertiary categories have already been established man- ually for this secondary category, which are: {...). Please supplement additional categories that do not belong to the aforementioned tertiary categories to cover all tasks under the secondary category in the context of multimodal content understanding, Output format: Each line should correspond to one task category, without any other characters. Different levels of task categories should be connected by '~"
p3 w
You are an expert multimodal content understanding with extensive experience this field. want to construct comprehensive task taxonomy for multimodal content understanding that in- cludes only two modalities: images and text. The imput to this taxonomy will be images and their corresponding text, and the output will be in text form. The primary, secondary, and tertiary cate- gories in this taxonomy are connected by '~-., Currently, the task name for the secondary category that needs to be detailed is {...). Please expand the tertiary task categories under this secondary task category to cover all tasks included in this secondary task category in the context of multimodal content underslanding. Output format: Each line should correspond to one lask category, without any other characters. Different levels of task categories should be connected by "~'".
p3. w/o
About task types corresponding to the image: In the third step of the data generation pipeline, the top 10 task types with the highest CLIP's s image-tex cosine similarity are assigned to all images. These task types then further refined using GPT-40 to identify multiple task types corresponding to each image, which are subsequently used to generate question-answw pairs. Figure A-I the list of images along with their associated task types.
18

---

task Published as a conference paper at ICLR 2025
Table A-3: The prompt templates used throughout the other stages of the TaskGa pipeline: further filtered in the third stage based on image and task type compatibility, employed in the fourth stage to generate question-answer pairs, and finally refined in the fifth stage through multi-referee scoring to ensure optimal matching and high-qualin alignment between task types, image content, and question question-answer answer pairs, and the prompt template for Chain-of- Chain-of-Thought (CoT) answers generation in the ablation study.
Type
Prompt
You are multimodal content understanding expert. Given image and multiple task labels related multimodal content understanding, with task labels init_rakk_lype I would like to generate some question-unswer pairs related to these task labels based on the image content. However, some task labels might not be relevant to the image content and thus cannot generate relevant questions. Please fully understand the image content and the meanings of the task labels, and select all the task labels that are appropriate for this task. Ensure the task labels are the same as the original. The output format should be: [task labels], without any other characters, If there are no matches, output [None].
p.filter
You are multi multi-modal content understanding expert, very skilled in handling visual question an- swering tasks. Your task is: Given an image and task lahels that highly relevant to the content of this image, the task labels are: {.... Please fully understand the content of the image and, for cach task label. propose a question and answer pair related to the image content. Please try to propose some complex questions and provide answers to these questions. Please strietly follow the ISON format for the output. Each line should represent question and answer puir corresponding label in the following JSON format: ("task type": "question": "answer":)) )
P.Q&A
You are an expert in mulimmoda content understanding, particularly skilled in handling visual ques- tion answering tasks. Your task is: Given an image and a multimodal content understanding-rett reelaad task label, along with question related to that task label, the task label is "(task_ty (task.type"" the ques- tion is "[question]" J", please fully understand the image content, the task label, and the question, and determine whether the task label and question are suitable for the image. If suitable, score as 1: otherwise, score it Please only output your final seore without any other characters.
p_filter_ referee
You are a multimodal content understanding expert and you are very good at solving visual question answering tasks, I will give you image and a question related to this image, the question is: {question). J. your task is to fully understand the content of the image along with the question, please fully think about how the question should be answered, please give step by step thought process of how you solved the question and finally output the answer to the question.
P.CoT
Images
Task types related to image content
Images
Tosk types related to image content
Limage descriptiorrculture
1image description-activity baseit---tssdsstttt event description
based-architecta e and lanciarkss
2. 2.content creation-virtul tour norrationncity
anallytie-cchletic skills techriques-skil
landmark tours
clessifico ication
3 visuol question logpennng-location-aee based
imege content understanding-conten
question enswering-ident iddettiff specific
recognition-see sioi--noootttrmeesesee recognition
landmark in image
Le-conmerce ethancmment -ppodcctt destcription generation
Z. commerce enkkancmennt-prode product recommendation 3e commerce onhoncement-ppoe pradocccttt-tt tiile optinization 4.content creaflor-e- commeroe contintt creation- product fecture listing
viviuual data mising-visual commonsanse
reosoning--oe copttning-spptiol relationship reasoning
2 visiul dato mining ressoring-lleiced Gressnning
Sy-commerce edhen:tment-ingg--eee -ottdd product tegging Bu--ommerce cohemcement--rooe and up-sell
suggestions
1. Leciinnc-rellaee-vviie
text generation based on image-news article generction-contexxuuu bockground generation
reasoning aestooomnnittooooiioolloo image interpretation
S.ccience relored--nnwieddge
2 summar12atiormmaus article sumvmarization 3 www.ddeniffitiionneeeeeee ocr
populorzzstion-an pettiin-sstrrnmmee nelsted education
3. topic classificatior-ssooooooo tssorrtrroyyyy
Figure A-1: Examples of images along with their corresponding task list pairs filtered by GPT-4o.
A.3 COMPARISION WITH EXISTING DATASETS
Table A-4 presents a comparision between existing visual instruction tunning datasets and TaskGalaxy. For existing visual instruction tunning datasets, we directly adopt the numbers of tasks and instances reported in their original papers. VL-Qwen Bai et al. (2023) is a newly intro- duced large-scale dataset with human annotations; however, it is not publicly accessible. In contrast, MultiInstruct Xu et al. (2023) is constructed from publicly available datasets but primarily empha- sizes visual grounding tasks, containing only 29 tasks that exclude region-specific information. To
19

---

Published as a conference paper at ICLR 2025
further expand the number of task types, the recent Vision-Flan constructed a dataset comprising 196 task types through expert collection and curation. Although this approach increases the num- ber of task types compared to previous datasets, it heavily relies expert involvement, making the creation process highly time-consuming, labor-intensive, and significantly limiting its scalabil- ity. In contrast, TaskGalaxy employs almost fully automated process construct high-quality instruction-tunae dataset encompassing nearly 20,000 task hundress of times more than previous efforts.
Table A-4: Comparison between TaskGalaxy and existing visual instruction tuning datasets.
Dataset
Instances
# Tasks
Source
LLaVA Liu et al.
2024b
150K
3
Synthetic
LAMM Yin et 2024)
196K
8
Synthetic
VL-Qwen Bai et (2023
350K
Unknown
Synthetic
M3IT Li et al. 2023d
2.4M
40
Private
mPlug- Owl Ye et al. 2023
150K
3
Synthetic
Shikra Chen et al. (022a)
156K
4
Synthetic
SVIT
Zhao et (023)
4.2M
4
Synthetic
Multilnstruct Xu et al. (2023)
510K
62
Synthetic
VISION-FLAN Xu al. (2024)
1.6M
196
Public
TaskGalaxy (Ours)
431K
19227
Public+Synthetic
A.4 ILLUSTRATIONS AND ANALYSIS OF SAMPLES FILTERED DURING THE DATA GENERATION PIPELINE
In TaskGalaxy generation pipeline, there are two parts involved in matching and screening, and in the first part, we employ a two-phase process for matching and screening task types and images. In the first phase, task types are generated and open-source image data is collected. We use CLIP to perform an initial screening, matching task types with images based on their textual and visual similarity. However, the performance of CLIP's image-text matching is inherently limited. This sometimes leads to overestimation of similarity scores, resulting in mismatches where task types paired with images that do not accurately represent their content. To address this limitation, the second phase involves leveraging GPT-4o with carefully designed prompts to refine the matches. This step effectively filters out task types that not contextually related to the content of the images. Below, we provide a comparison illustrating the image-task pairings before and after the second-phase refinement by GPT-4o, demonstrating the improvement in alignment between task types and image content.
Match and Filter by CLIP:
image content understonding--onten recognition-public facilities recognition
safety and risk assessmen--hygieee sefety and risk assessment-sanitthiie protocol adherence
content creation-travel blog content creation-sustainbbee travel accommadations
image hatpppioon-inn-grrnnd---reeeee detailed description
recommennaation
suggestions--mart home solutions
safety and assessment -hygeent safety
ossessment-wrrkllcc cleonliness
wuggestions-intev/e lighting ideas
content creation"virtual tour nerration-mseumm exhibit explonations
content creation-travel content craation-acoommddation reviews
Match and Filter by GPT-4o:
image Sseriition-fine-grrine----
detailed description
safety and risk dosessment-hygieee safety and risk assessentt-onitatioon protocol adherence
content creation-travel content cneation--ustainable travel accommadations content creation-travel blog content creation-accommmddaiion reviews
Figure A-2: Example of Filtered-Out Samples.
In Figure A-2 depicting the bathroom of this hotel, the initial CLIP matching may generate terms like "public facilities", "restaurants", "smart home", "workplaces", and "museums" which not
20

---

Published as conference paper at ICLR 2025
well-aligned with the actual image content. However, after applying corrective filtering with GPT- 4o, the results are more compatible with the image content. This filtering refines the matches to include task types such detailed descriptions of the room, hygiene-related information, tourist accommodation, and accommodation reviews, all of which align well with our expectations.
In Figure A-3 featuring the iconic building, the initial CLIP matching generated numerous task types related to communities, which were clearly not aligned with the content of the image. However, after GPT-4o filtering, only relevant task types were retained, such those related landmarks and weather background, ensuring a better match with the image content.
Match and Filter CLIP:
disaster response plaming--oommuity engegement plamiing-faclitatinn public forums
disaster response planning communitty engagement planeing-monitoring community
feedback
image geolocation idettiiiattoonllannnatt recognition
safety and risk geessmnttweather-eelteee risk essessment-severe weather detection
disaster response plamiin--community engagement planningridentifying offected
communities
safety and risk innessmnt-infrrstruutue safety and risk aseesmat-comunniation
tower safety inspection
response plamning-damage assessent-utillyy infrestructure damage detection
visual question naaeering-lccctio--beeee visual question onswering-identify specific
landmark in image
disaster response planning--commnic engagement planning-building community
disoster response plaming-communit engegement planning-assessing needs affected communities
Match and Filter 6PT-40:
image geolocation ddntiffiationnlanndark recognition
visual question onswering- based visual question onswering-identify specific landmark image
safety and risk assesmment-wectherrredtte risk assessment-severe weather detection
Figure A-3: Example 2 of Filtered-Out Samples.
In Figure A-4 showing the baseball sports, the initial CLIP matching generated numerous task types related to social media analysis. This may be influenced by the coexistence of social media content and sports within CLIP's training data. However, the images are more specifically aligned with the theme of sports. After GPT-4o filtering, the retained task types focus on sports recognition, player analysis, and other topics more relevant to the image content, providing a better contextual match.
Match and Filter by CLIP:
media anolysie-hashtop performance onolysis-hashtog historical performance tracking
social media analysis-ad performance onclysis-competitor od performance analysis
anddrstonding-contenn recognition-epore recognition content creation-sports commentary-player comparison articles
image
content creation-sports
insights
anclysis-uports events-pleyer performance anelysis
social media enalysis-heshteg performance anolysis-hashtog platform-specific
performance analysis
text generation based image-image report generation-sport image report generation
media analysis-ad performance onelysis-cros--pett performance analysis
social media enalysis ed performance melysisselick-throom rate prediction
Match and Filter by GPT-4o:
image content understanding-conten recognition-sports recognition
content creation--sparts commentary-ployer comparison articles
content creation-sports hammeentrr-behin--theeeeee thees-eeee-ss- insights
onalysie-sports events"ployer performance analysis
text generation based image-image neport generation-sports image report generation
Figure A-4: Example 3 of Filtered-Out Samples.
In Figure A-5 showcasing restaurant food, the initial CLIP matching assigned task types such as restaurant OCR and related information. This could because restaurant food often associated with menus in the training data. However, after GPT-4o filtering, only food-related tasks, such as food recognition and other more relevant task types, were retained, ensuring a closer alignment with the image content.
Analysis:In the third part of the data generation pipeline, we utilize CLIP to initially match task types for each image. The primary purpose of using CL.IP is identify and screen the ten images
21

---

Published as a conference paper at ICLR 2025
Match and Filter by CLIP:
creation--rovel blog creation-local cuisine recommendations
image geolocation identifiiation-lcaal cuisine recognition
image descriptionnc/lure--baa food and cuisine
ocr"restourant review ocr-menu extraction
image content understanding-conter
recogrition-foed recognition
fine-grained entity recognitior-food recognition
ocr"restourant review ecr-foood item idontification
ocrmimage preprocessing ocr"restourant review ocr-review classification
oor"restourant
Match and Filter by 6PT-4o:
content creation--rovel blog content creation-lacal cuisine recommendations
image description-cuhure-baaee bosddd--oodddd cuisine
image content understanding-conte reconition-food recognition
fine-grained entity recognition-foed recognition
Figure A-5: Example 4 of Filtered-Out Samples.
most similar to the content of given image. Additionally, this step helps avoid excessively long text prompts, which could exceed the input token limit during the subsequent GPT-4o screening stage. However, the effectiveness of the initial matching is influenced by the training data of CLIP, which can lead to the generation of illusory task types. For instance, during CLIP's training, food images may often co-occur with menus, sports images with media reports, natural scenery with tourism-related content, and pet images with pet products. These associations can cause CL.IP to in- accurately match task types that not directly relevant the actual content of the image, resulting in imperfect matching in this first stage. To address these issues, the second stage involves GPT-40, which refines the candidate task type list based on the actual content of the image. As demonstrated in the examples above, the filtered results effectively retain task types strongly aligned with the im- age content. This two-stage process ensures a higher degree of relevance and accuracy, achieving the intended purpose of the pipeline.
In the final step the data generation pipeline. task-type-related question-answer pairs generated by GPT-4o are evaluated and screened using three open-source multimodal models. Samples are rejected in this step if at least two of the three models determine that the task type, questions, and answers are not well-aligned. The primary objective of this process is to eliminate low-quality samples that may arise from GPT-4o's hallucinations or incorrect responses. Below, we present several examples of samples that received total score of less than 2 during the final scoring by the open- source models. These examples highlight cases where the task type, question, and answer alignment failed meet quality expectations.
B
C
Task Type: complex mathematical calculations Question: If AO units, OB 12 units, and OC = 13 units, find the perimeter of quadrilateral ABOC.
the quadrilateral ABOC is AO OB BO OC 5 + 12 + 12 + 13 = 42 units.
Answer: The perimeter
Tyeoo.co
"interv12 2__e__22B_______ score":
2 "O"
@glmmv_score"" "O" "internvl_score!: "O"
O
Figure A-6: Example 5 of Filtered-Out Samples.
For the geometric content-related questions in Figure A-6 the task type is categorized as complex mathematical operations. However, based on the task type and the picture content, GPT-4o generates unreasonable values for the lengths of OA, OB, and OC in the question. Furthermore, the calculated perimeter in the answer is incorrect. As result. the MLLM (Multimodal Language Model) both for- mulates and answers the question incorrectly. Open-source models uniformly scored these samples 0 points, leading their exclusion.
22

---

Published as a conference paper at ICLR 2025
Task Type: vehicle recognition~make and model identificatior-muiiinddd data fusion
BUILDING A SCALABLE DATA WAREHOUSE WITH VAULT. 2.0
Question: What is the make and model of the vehicle shown in the image, and how does it relate to the concept of multimodal data fusion?
Answer: The vehicle shown in the image is Ferrari F8 Tributo, high-performance sports car known for its cutting-edge design and aerodynamics. The image demonstrates the concept of multimodal data fusion by combining the vehicle's 's advanced engineering data with urban infrastructure data, enabling real-time optimization for autonomous driving in high-speed scenarios.
"intervi2_2e" 2aar22222___see_ 26B_scoree: "O" "glmmv_scored" "O" "internul_score:: "O"
MK
Daniel Linstedt Olschimke
Figure A-7: Example of Filtered-Out Samples.
The Figure A-7 features an artistically designed virtual vehicle, a specific real-world brand. The response incorrectly identifies the vehicle as a Ferrari F8 Tributo. Additionally, there is no direct information about multimodal fusion embedded within the image. This aspect of the response is entirely speculative and fictional, likely influenced by the task type "data fusion," which has led to associating unrelated information with the image. The response attempts from the task type (e.g.. multimodal data fusion) to content that is absent in the image. As result, diverges from the core task of identifying the vehicle's make and model. Instead, focuses on explaining the concept of "multimodal data fusion." producing answer that is misaligned with the task requirements. Due to this disconnection, the response was filtered out by the open-source model.
connect keywords
Task Type: image
understanding-content
recognitoo--miiallsannnd color recognition
Question: What is the predominant color of the cat in the image?
Answer: The predominantly grey with white markings. _______B_eeer "O"
"intervi2_26e Leervil___B_____11
@olm4v_score":
"internvi_score"" "O"
Figure A-8: Example 7 of Filtered-Out Samples.
In Figure A-8 the answer states that "The cat is predominantly gray with some white markings," but based the content of the image, the animal in the picture is actually dog sitting by the window, not a cat. This represents a classic case of hallucination, where the model incorrectly categorizes the animal image clearly shows the dog as predominantly dark brown or black, not gray. The task was to identify the "predominant color' of the animal, but the mention of "white markings' in the answer is not supported by the image content. This indicates that the model likely introduced assumptions or biases unrelated to the image itself when generating the answer, resulting in response that does not align with the factual visual content. The open-source model received score of O points due to its failure to correctly identify the animal species (mistakenly identifying a dog as a cat), incorrect description of the animal's predominant color, and its generation of an answer based entirely hallucinations unrelated to the factual content of the image.
cat. Additionally, the answer describes the animal as "mainly gray," whereas the
In Figure A-9 the answer avoids addressing the core of the question, which concerns "the former owners of the painting and whether it is part prestigious collection." Instead, it merely states that detailed research into art records and auction histories is needed, failing provide specific in- formation directly related to the image content. Furthermore, the response entirely disregards key features of the image, such as the figure depicted and the style of the painting. It made attempt to
23

---

Published as a conference paper at ICLR 2025
Task Type: interactive art creation-hiistrical art recreation with text guidance-provenance tracking
Question: Who were the previous owners of this painting, and has it been part of any famous collections in the past?
Answer: This question would require detailed research into art records and auction histories to find out about the painting's s ownership and past collections.
LeB_sccre": "O" "glm4v_scorr": "O" " "internvi_score" "O"
Figure A-9: Example 8 of Filtered-Out Samples.
analyze or extract relevant information from the image itself. The task type, "provenance tracking," explicitly requires specific details regarding the painting's provenance. However, the response devi- ates significantly from the task's objective by offering a generalized and non-informative statement, thereby failing to meet the expectations of the task. The open-source model received score of points due to its failure to extract useful information, its lack of analysis of the image content, and its inability to fulfill the task objectives.
Hoseele
Larson Hosteller
Task Type: logical reasoning~ccomplx reasoning=ccomple mathematical calculations-eagebraiic operations
ALGEBRA
Question: If y. and are real numbers such that x y+ and the sum of the squares of the roots of the quadratic equation 2x 4xz + 32 0 is equal to 10, what is the value of z?
ALGEBRA
AND TRIG METRY
RIGONOMETR
Answer: 5/3
EDITIN
"intervi2_2631 26B__coree: "O"
"gim4v_score" "0"
"internvi_score:: "1"
Figure A-10: Example 9 of Filtered-Out Samples.
In Figure A-10 the answer provided 5 as the final solution, but through logical reasoning and algebraic calculations, can be correctly determined 222 or - Thus, the answer is incorrect. The task requires logical reasoning and complex algebraic calculations to solve the problem, but the answer does not meet the expected level of complexity required for this task type. Additionally, the image content appears to be more relevant to OCR recognition of the book cover rather than the mathematical problem presented. This mismatch indicates that the problem itself may be irrelevant or disconnected from the graphical content, making the task type unsuitable. As a result, two of the three open-source models correctly judged the answer with 0 points. The Internvi model, however, provided a score, which may reflect a lack of mathematical reasoning capabilities, leading to an incorrect evaluation of the task's requirements and performance.
3
In Figure A-II the model depicts a scenario in which "flooding inundates the marina and affects nearby buildings and facilities". The image shows no sign flooding all, only a normal marina scene with moored boats. The model's responses were generated entirely on the basis of fictional content. The model may have assumed flood damage the image by default due to the task type "damage assessment" and ignored the normal conditions of the actual image. The task asks for an assessment of "infrastructure damage due to flooding" and there is no flooding in the image, the model should recognize this and answer with something realistic such "no flooding or damage observed".. This mismatch between the task type and the image content led to the creation of subsequent illusory responses, which were filtered with total score of less than two.
24

---

Published as a conference paper at ICLR 2025
Task Type: damage assessment
Question: What kind of infrastructura damage can be observed in the image due to the flood?
Answer: The image shows high level of water submerging the docks and possibly affecting the buildings and facilities near the waterfront, indicating significant infrastructural damage.
"intervl2_266 26B__coree "O"
@glm4v_scoret "O"
"internv[_score: "O"
Figure A-l1: Example 10 of Filtered-Out Samples.
Task Type: ocr*environmental text ocrapublic information boards reading
Question: What public information is provided by the board attached the yellow structure?
Answer:
The board attached to the yellow structure provides
information from NYC regarding collection dates and categories of
refuse collection.
"intervi2_26B__r 2et____r__ccoe" "O"
: "O"
"internvi_score"" "1"
Figure A-12: Example 11 of Filtered-Out Samples.
In Figure A-12 the yellow structure the picture appears to be a discarded refrigerator, and there visible notice board attached to it. The model likely generated plausible but fabricated response based on the cues "yellow structure" and "notice board." However, the information in the response is entirely fictitious. Moreover, the yellow structure on the left clearly displays the text "New York Post," indicating it is box for distributing newspapers. This has no connection to recyeling dates or garbage sorting. The model incorrectly associated the task type (public message board) with garbage sorting information, likely due to a lack of OCR capability and proper semantic understanding, resulting in a response that is detached from the actual content of the image. Two open-source models correctly scored the response 0 points, recognizing that the content was illusory and inaccurate. However, one model mistakenly scored the response point, possibly because the answer aligned loosely with the task type in superficial manner.
Analysis: In the final stage open-source model scoring and judging to screen samples, an anal- ysis and summary of the above examples reveal several key issues. Some samples are unqualified due to errors in the questions or answers generated by the MLLM (Multimodal Large Language Model) itself. Others fail because the model's answers contain hallucinations incorrect informa- tion, while in some cases, the task type does not align with the content of the image, resulting in the model's 's question-answer pair being invalid. Additionally, certain issues arise when the model generates hallucinatory information that, while related to the task type, is not connected to the actual content of the image. Moreover, the model's inherent capability limitations (e.g.. OCR capability, mathematical reasoning ability) also contribute to errors in the answers. These challenges highlight the problems faced by MI.LMs when generating answers. To mitigate the inclusion of such hallu- cinatory and erroneous samples in the final dataset, we incorporated three open-source MLLMs to evaluate and score samples. This multi-model evaluation approach helps identify unqualified sam- ples more effectively. However, observed in the examples above, even open-source models can sometimes misjudge samples. To address this, we adhere to the principle "three ignorant cobblers working together outdo a Zhuge Liang." allowing multiple models evaluate and score samples collectively. This strategy maximizes the chances of filtering out unqualified samples and ensures the construction of high-quality TaskGalaxy dataset.
25

---

Published as a conference paper at ICLR 2025
A.5 LICENCES DATA
The licensing information for the image sources listed in Table i follows: ALLaVA (Apache License 2.0), Visual Genome (CC BY 4.0), M6th (Apache License 2.0), and ShareGPT4V (CC BY-NC 4.0). The proposed dataset, upon its open-source release, will be licensed under CC BY-NC 4.0.
We acknowledge our responsibility for ensuring legal compliance in data usage. The dataset licenses have been carefully reviewed, and our release under CC BY-NC 4.0 aligns with the restrictions of certain sources. Steps have been taken mitigate potential legal risks and ensure adherence the respective terms.
A.6 EVALUATION REGULATIONS
For the evaluated benchmarks MME, MMBench, MMBench CN,, MM-VeT, POPE, SEED, SQA, and TextVQA, we utilized the official evaluation code provided by LLaVA. For AI2D, ChartQA, HallusionBench,, LLa ALaVA-in-the-wild, MMMU, Q-Bench, and Chinese- Q-nncch we referred to the evaluation code that follows the official evaluation protocol of InternLM XComposer, The Math- Vista evaluation bascline was conducted using the official evaluation code of Vista. In bench- marks such as MM-VeT, LLaVA-in -----in--ttll-t and Math Vista, we replaced the original GPT-4 API with GPT-4o, which offers more stringent criteria and improved performance for scoring, answer extraction, answer matching, and related tasks.
AttemLM-XCooppse
A.7 EXPERIMENTAL EVALUATION USING TASKGALAXY, BASELINE DATASET, AND OTHER INSTRUCTION-TUNEE DATASETS INDIVIDUALLY
To demonstrate the impact TaskGalaxy's task diversity on model performance, we compare the baseline fine-tuning data with TaskGalaxy fine-tuning data, as well as several other instruction- tuning datasets, including ShareGPT-4V [Cen et al.) (2023b).
V.aVA-OOeVisiin [i et al, (2024a), VA et al. (2024a), and Cambrian- [Tng et al. 2024). For fair comparison, domly sampled the same number samples from each dataset as in TaskGalaxy for fine-tuning. The results, summarized in Table A-5 demonstrate that TaskGalaxy consistently achieves the highest performance on most benchmarks across multiple model architectures, validating its effectiveness.
ran-
Table A-5: Experimental Evaluation of TaskGalaxy, Baseline Dataset, and Other Instruction-Tuniiee Datasets Separately, All the numbers are presented in % except MME and the full score is 100%. The indicator MME the perception score, the maximum value 2000. The best results are highlighted in bold.
Model
Method
MMF
MMR
MMB
POPF
Benchmarks
ITVVA
MMNN
TQA
SQA
MatiVVita
1476
ShareGiPT-4
LLaVA- One Vision
1251
59.79
52.84
83.90
51.20
29.60
52.99
73.19
28:20
ALLaVA-IV
1474
60.13
55.39
84.21
38.00
27.00
53.77
70.05
29:20
Cambrinn-1
1494
61.08
54.46
85.46
52.00
25.70
55.17
71.03
29
IaskGalaxy
1520
66.62
59.43
86.40
$2.30
58.08
71.06
29.30
LLaVA-v15--B
Q-Rench
Q-Rench
HalluRench
Average
MIME)
Baseline
14:40
25.29
21.89
31.26
47.95
58.62
19.70
43.62
ShareGPT-44V
17.84
27.08
26.22
32.51
48. 79
$9.26
15.60
44,48
VlaVA-OneVision
18.72
27.95
24.48
33.51
47.74
30.06
17.00
1.66
ALLaVA-IV
17.00
23.73
23.95
33.18
48.73
40.52
18.10
41.53
Cambrian-1
TackGalaay
19.90
50.95
20.70
MME
MM3
MIM
POPE
LLEVA
MIN
TQA
SQJA
MailiVista
Baseline
1458
64.35
36.41
45.20
25.70
35.29
05.03
27700
ShazGPT--P 4Y
1191
47.08
40.63
82.62
30.00
17.90
44.21
64.55
27.20
LLaVA-
ALLaVA.4V
Cambrian-1
1481
60.22
53.01
84.17
43.70
26.80
52.61
67:21
33,00
InternV1 Whh.--Chhhlllli De78
TaskGalaxy
1512
65.03
$7.91
86.23
$2.30
30.10
56.15
68.88
10
CharQAA
AI2D
Q Bench
D-BBench"
HalluBench
SEED
MMMMU
Averrup
XIMEST
Baseline
14.12
S1
ShareGPP-44
14.52
LLaVA- OneVissio
13.76
22.75
10.08
40.87
24.60
40 20)
ALLaVA
12.90
28.28
42.87
44.15
51.41
48.36
27.30
42.3 22.94
Cambrian-1
16,00
36.69
48.00
41.33
54.63
56.24
30.60
66
Tasaaalayy
15.16
37.69
48.21
$3.00
60.44
32,80
49.63
26

---

Published as a conference paper at ICLR 2025
A.8 MORE ADVANCED MODEL ARCHITECTURE
For more advanced models, we utilize the second-stage instruction fine-tuning data publicly available. For the comparison, randomly sam- ple the same number of samples as TaskGalaxy from the officially disclosed instruction fine-tuning dataset.
model, which has made its
We fine-tune InternV--Chan WL-CCat--2.00-83 using both the original instruction fine-tuning dataset and the TaskGalaxy instruction fine-tuning dataset, ensuring that the number of samples for each is consistent with TalkG...xy. The Table/A-Ojshows the performance comparison between the original instruction fine-ture ane-tuning dataset and the TaskGalaxy instruction fine-tuning dataset on Intern VL-Chat- V2.0-8B.
Table A-6: Experimental Evaluation of Fine-tuning Intern VL-Chat-v -iaaatvv1.... 1.0-8B Using TaskGalaxy and Baseline Dataset Individually. All the numbers are presented in % except MME and the full score 100%. The indicator of MME is the perception score, the maximum value is 2000. The best results are highlighted in bold.
Model
Method
Benchmrkks
MIM
MM3
MMB
FOPE
LLaVA
MMMV
10A
S0A
Math
Baseline
1536
68.52
66.46
36.30
63.20
46,17
66.24
90.58
S0,10
TiskCallayy
73,88
70.49
52.31
InterWVI.Chha-Va
CharQA
A1213
Q-Reach
Q-Bench"
HalluBench
SF33D
MMML
Averge (w/o MME:
Baseline
76.64
75.88
57.79
36.98
57.51
62.72
40.50
65.86
TaskCialary
64.25
As shown in the Table after fine-tuning the model using the TaskGalaxy dataset and the original fine-tuned dataset, TaskGalaxy outperforms the original baseline dataset
Intern WL-ChL-t-V222 on 14 out of 16 benchmarks. For the remaining two benchmarks, ChartQA and LLaVA-in-the- wild, the performance difference compared to the baseline dataset is minimal (less than 0.5). This demonstrates that TaskGalaxy's s enhancement of task diversity is also effective for more advanced models, providing significant boost in overall performance.
A.9 THE BENEFITS OF CHAIN-OF-THOUGHTTCCN)
Numerous studies (Zhang et al. 2023b Zheng et al. 2023 Zhang et al. 2024a Mitra et al. 2024) have highlighted the significant impact of Chain-of- hought (CoT) prompting on enhancing MLLM performance, In this section, we investigate whether CoT prompting improves performance with in- creased task types. We constrained the maximum number of samples per task type 5, resulting in total of 76k samples (as indicated by max in Table A-7). We compared the performance using original TaskGalaxy Q& data with CoT-generated answers from GPT-4o, designed through specific prompts detailed in the Appendix. The results show significant improvements for the CoT versions TaskGalaxy (max 5) in benchmarks such as MME, -L-----i-thh--ill and Q-Bench. Additionally, the average performance across 15 benchmarks, excluding MME, increased by approx- imately 1.3 points with COT. These findings underscore the value of incorporating CoT prompting into multimodal models.
Table A-7: Performance comparison of CoT validity verification. +max 5 refers 19,227 task types, each containing no more than 5 samples, totaling approximately 76k samples. In contrast, 5max_5 (CoT) represents the version where Chain-of-Thought (CoT) answers were generated for all the aforementioned samples.
Beachmarks ChurQAA
Model
Method
MME
MMB
LLeVAW
MathVisia
Q-Bench
MMMU
Averageí MME)
Bascline
1506
64.69
53,0
26.7
14.72
26.08
16.6
44,46
LL
Baselimetmux. 5
1506
65.80
53.4
27.3
20.20
36.48
17.4
46.61
Raseline-mmax.5
1523
66.72
64.7
27.9
20.96
43.27
19.3
47,92
A. TASK TYPES IN TASKGALAXY
One of the key challenges addressed by the TaskGalaxy instruction fine-tuning dataset is the sub- stantial expansion of task type diversity. Initially, we manually defined small set of task types,
27

---

Published as a conference paper at ICLR 2025
which was later expanded to 19,227 hierarchical task types using GPT-4o. Given the large number of task types and space constraints, we present only a selection of these hierarchical task types here. A more comprehensive list will be available in the full dataset upon its release.
Table A-8: Comprehensive Task Type Table
Level-1 Task Type
Level-2, etc. Task Types
bill ocr ~ medical bill recognition; bill ocr travel expense recognition; book oer metadata extraction; book ocr ~ text sum- marization; business card ocr text extrac- tion; business card ocr ~> entity recognition; captcha ocr multi-font text recognition; chart bar chart ocr; chart ~ violin plot ocr; comic strip ocr speech bubble detec- tion; comic strip ocr text translation; dia- gram ocr ~ flowchart recognition; document form ocr; document full text litera- ture ~ research paper ocr; font recognition font type identification; graffiti hancement; grocery list ocr ~ item price ex- traction: handwritten text oer word recog- nition; infographic ocr ~ caption analysis; invoice ocr data validation; invoice signature detection; label ocr ~ word recog- nition; lecture notes oer speaker identi- fication and attribution; logo recognition brand identification; logo recognition ~ logo location detection; musical notes ocr time signature detection;
OCR
text
abstract-conneet-based symbolic repre- sentation; abstractt-onncept-aaeed ~ aes- thetic judgment; accessibility-based descrip- tion highlight key elements; accessibility- based description emotionally aware de- scription; action-based ~ contextual action description; activity-based
sports event description; activity-based performance or event description; advertisement-Be based testimonial integration; art-style-based surrealist description; attribute- based ~ pattern recognition; autobiographical-ban based daily activities; bias-mitigation-baall ~ disability bias detection; coarse- coarse-grained highlight extraction; coarse coarse-grained alt text generation; context- aware description correction ~ factual accuracy verification; contrast-based emphasis on unique as- pects; cross-cultural adaptability identify cultural context; culture-based ~ architec- ture and landmarks; culture-based ~ social norms and values; educational-conter based step-by-ste by-sepp tutorial; emotion-based emotion-driven storytelling: event-based ~ event sentiment analysis;
Image Description
Continued on next page
28

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
object detection ~ single object detection ~ firefighting equipment detection; object de- tection ~ single object detection vehi- cle interior detection; target detection scene detection ~ campus scene detection; target detection ~ scene detection ~ traffic scene detection; signature detection; out of stock detection; anomaly detection ~ vehi- cle anomaly detection; anomaly detection behavior anomaly detection; object detection multiple object detection quantity de- tection; object detection ~ single object de- tection road traffic signal detection; object detection single object detection pub- lic facility detection; target detection scene detection ~ pond scene detection; signature detection;
Detection
color analysis color contrast analysis; color analysis ~ dominant color detection; complex scenes attribute extraction; con- tent accessibility analysis ~ automatic sum- marization for accessibility; content person- alization ~ user interest profiling; content prudence analysis source reliability eval- uation: design concepts and intentions evaluate layout and composition; emotional analysis mood classification; emotional analysis emotion trajectory analysis; fash- ion analysis season identification; fash- ion analysis ~ color analysis; game rules and strategies goal determination; gender representation analysis ~ gender stereotype identification; gender representation analy- sis gender representation in advertising analysis; language use analysis language complexity analysis; language use analysis speech act recognition; review analysis review summarization; review analysis authenticity verification; political sentiment analysis polarization detection; political sentiment analysis ~ emotion detection;
Analysis
fine-grained image classification; seman- tic segmentation; caption generation; mul- timodal trend analysis: image inpainting: region-based image captioning; object at- tribute extraction; attribute recognition; con- tent summarization; visual relationship de- tection:.....
Image-based Knowledge Distillation
Continued on next page
29

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
FAQ creation ~ Extract information from images create detailed FAQ answers; FAQ creation ~ Determine ambiguity in text and images for FAQ refinement; advertising con- tent creation ~ brochure content creation; advertising content creation ~ flyer content creation; annual report writing industry benchmarking; anmual report writing ~ risk assessment; art critique technique and brushwork examination; art critique ~ ges- ture and movement evaluation; artistic inspi- ration writing ~ mood setting; artistic in- spiration writing genre-specific style im- plementation; augmented reality content cre- ation virtual staging and design; aug- mented reality content creation ~ 3D ob- ject placement; brainstorming design concepts; brainstorming product ideation; brainstorming headline gener- ation; children's book creation conflict resolution; children's book creation ~ nar- rative voice consistency; content curation content diversification; corporate train- ing content creation employee onboard- ing material creation; diversity and inclu- sion content creation celebration of cul- tural events; e-commerce content creation ~ purchase decision support content creation; e-commerce content creation content cre- ation for limited-time promotions; educa- tional game content creation image cap- tioning games; interactive content creation immersive simulations; letter writing closing statement formulation; log writing ~ daily summary, log writing task manage- ment:
Content Creation
graphic
home decor ideas; furniture design; wed- ding planning; recipe suggestions; seasonal decorations; party themes; workspace er- gonomics; hair styling; book recommenda- tions; interior lighting ideas; eco-friendly products; movie suggestions; fashion advice; life hacks; meditation practices; time man- agement techniques; TV show recommen- dations; coding resources; online learning platforms; memory improvement exereises; volunteer opportunities; language transla- tion aids; entertainment activities; speech writing: content creation tools; study tech- niques.......
Suggestions
Continued on next page
30

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
mathematics; transportaiionlloiisiic. religion: language& literature; law; agricul-
ture: architecture; psychology; geography; environmental studies; political science; entertainment; economics; philosophy: education; history; media&communication. arts&katss~music studies; arts&
studies; arts@karts-accaemii theoret-
Subject Question and Answer
ical studies; busnness-manggemmnt business--finace stud-
studies;
ies; business--accoounting studies;
heath--medicine--elinicll medical stud-
ies: humanities&ssocial sciences-<hssory
studies; science-geographhy stud-
ies:
maathee--tt studies; engineering cechoolggkeeegiieern--eeeeeeeree energy studies;.....
studies;
sports event summarization; pop culture summarization; policy summarization, news article summarization; book overview sum- marization; technical document summariza- tion; agricultural data summarization; recipe summarization; presentation summarization; environmental impact summarization; ge- nealogy summarization; customer journey summarization; financial report summariza- tion; advertisement summarization: legal document summarization; travel itinerary summarization; instructional summarization; customer preferences summarization; mar- ket analysis summarization; political debate summarization; comparative summarization: multimodal dataset summarization; multilin- gual content summarization; event summa- rization; healthcare report summarization; wildlife monitoring summarization; cus- tomer service conversation summarization; event outcome summarization: climate data summarization; historical document sum- marization; educational content summariza- tion; radio interview summarization; product description summarization; brand sentiment summarization; thematic summarization; vi- sual trend summarization; image- based con- tent summarization; product review summa- rization:.....
Summarization
Continued on next page
31

---

analysis-high-dimeen Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
abductive reasoning- hypoth- esis generation; analogy reasoning-ssenne relationship analogy reasoning; causal reasoning-couuttrfaccull
reasoning: complex reasoning-thrre-dimenii spatial relationship reasoning: complex reasoning--passengee flow analysis rea- soning; complex reasoning--market analy- sis reasoning; complex reasoning~-tak relationship reasoning:
complex reasoninge-ethical dilemma reasoning: complex reasoning--role
reasoning: complex reasoning--state reasoning; complex reasoning--chart understand- ing and analysis-mmllt-layer pie chart logical understanding and analysis; com- plex reasoning--chart understanding and data scatter plot understanding and analysis; com- plex reasoning--omplees mathematical salculations-gcomettii mathematical op- erations; complex Frasoning~complee mathematical caleulations-ccoppute sci- ence algorithms operations; deductive reasoning--attribute deduction; deductive reasoning-iinerrence of missing infor- mation; simple sooooiinn-goalddireetee reasoning: simple reasoning--emotion reasoning; simple reasoning--sequence
Logical Reasoning
ordering: simple reasoning--basic
mathematics~simplle numerical calcula-
tion reasoning: simple reasoning-~basic
mathematiics-qquatiiyy reasoning; spatial reasoning--topological reasoning; spatial reasoning~33D scene reconstruction; spatial reasoning--spatial layout recognition; tem- poral reasoning--contexxuu time inference: temporal reasoning--event ordering; seasonal and holiday recommendations; per- sonalized shopping guides; context-based travel suggestions; context-specific per- sonal assistant; visual product recommenda- tions; real-time event notifications; location- based activity suggestions; dynamic content customization; context-sensitive educational content; contextual news delivery; adaptive learning resources; contextual target audi- ence analysis: lifestyle-based content cura- tion; interest-based content filtering: mood- based content suggestions; context-specific social media posts; personalized content sug- gestions;
Context-Aware Recommendations
Continued on next page
32

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
due to animal cruelty--idenification of animal types in images; due animal cmuelty~image and text align- ment for cruelty evidence; due to child exploiation-iidenfficatono
of minors; due to child explotation--ongaeemmnn with law enforcement; due to conspiracy theories--bimmdaal sentiment analysis re- lated to conspiracies; due to conspiracy theories--detect repetitive conspiracy mo- tifs; due to dangerous stunts--compare depicted stunts with known dangerous activities: due dangerous stunts<-analyze risk levels of described actions; due to deepfake content-cclasiffcationn due document
Refusal
human traffickigg-deetecinng forgery, due incitement of panie-ccooss- verify with trusted sources; due to political propagandar-identiiy exaggeration polit- ical claims; due to solicitationn-deectinng refusal language; refusal due to illegal activities--refusal due tribution; refusal due to pornographic conient-ccharacterizaiion of suggestive poses......
illegal dis-
esimattoon-Sel--uurrvvsd depth estimation; Material and texture recognition-MMatriill texture segmen- tation images; Material and texture recognition-Mattrial texture correla- tion between images and text; Object attribute extraction--Color detection; Ob- ject detection-RRal-timee object detection; Object CaaeciionnCCaeegoy--bbbase object de- tection; Object interaction modeling--Object
Depth
spatial relationships; Object matching and retrieval=-Aliningg textual descrip- tions with visual object attributes; Object matching and retrievll=-Multivvinnn retrieval from textual input; Object part recognition-PPat-bassee object localization; Object part segmentation relationship analysis; Object pose estimation--Instanee level pose estimation; Object recognition in context-Seemantic segmentation of objects in context; Object registration--Mulii--idn registration; Object tracking~ -bjeect local- ization; Occlusion handling--Multi-vieer fusion for occluded objects; Occlusion Sandiing-Self-supervis occlusion robustness:....
3D Object Recognition
image
learning for
Continued on next page
33

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
aviation safety and risk assessmentt safety monitoring: aviation safety and risk assessment~aiir traffic control com- munication analysis; biological hazard risk asessmentt-ooddhonee pathogen identification; biological hazard risk assessment-watee quality assessment; biological hazard risk assessment--isease outbreak identification; chemical hazard risk assessment-incident analysis and reporting; child safety and risk assessment-~age- appropriate content detection; child safety and risk assessment-explicit content filtering; construction site safety and risk assessment--saett signage compli- ance; construction site safety and risk assesmment-emergenny response prepared- ness; consumer product safety and risk assessment-label product hazards; cultural heritage safety and risk assessment-damage assessment; cultural heritage safety and risk assesmenn-eeniioommentll identification; elderly care safety and risk assessment~~flll risk detection; entertain- ment venue safety and risk assessment-ffrst aid station location identification; environ- mental safety and risk assessment-willlitt impact assessment; infrastructure safety and risk assessment-bridge stabil- ity analysis; wildlife safety and risk assessment-monitorigg health,.....
Safety and Risk Assessment
hazard
of wildlife
image siquence storytelling: im- age sequence-nnex image prediction: image sequence~sstryy board genera- tion: multiple images~-evnt chronology; multiple images--relationship extraction; multiple images--collage interpretation;
Image-Tern Text Matching
single image~visual reasoning: multi- ple images-visual consistency; multiple images-reelationship extraction; question answer selection~-visue entailment; ques- tion answer selection--scene-tss based question answering; question answer sslecton-iimge--apiion--sss question answering:.....
Continued on next page
34

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
anomaly detection in scientific data-ddetecting unexpected changes medi- cal imaging and diagnostic reports; chart and diagram interpretationiidenify underlying assumptions in data; chart and diagram imferpretation-compace different data sets; chart and diagram interpretation--explane scientific concepts; citations and influence analysis--citation context analysis; citations and influence analysis--historcaal citation trends analysis; common knowledge ques- tion and answer~correlate text to scientific imagery: common knowledge question and answer--classify scientific categories in images; common knowledge question and answer-ccoppaee scientific phenomena in different images; common knowledge question and answer-deduce outcomes based on visual experiments; concept drifts detection--alleration in hypothesis testing; concept drifts detection~shift in research focus; concept drifts detection-updates scientific vocabularies; conclusion extractionn-pattem recognition; content paraphrasing--xxplaattoy paraphrasing; data correlation analysis--metadata extrac- tion: data correlation moulysis~informaate synthesis; data correlation anatesis--oontett- aware filtering; error detection and correction~-table data correctness; ex- periment hypothesis generatioon-interactive multimodal hypothesis testing; scientific argumentation analysis--confounding fac- tor detection; visual reasoning--flowchan analysis; visual reasoning--equation visuil correlation:.....
Olence-Relater
attribute extraction--texture recognition; attribute extraction-sspatial geometry deter- mination: attribute extraction-affordaate
recognition; attribute extraction--ojeet
functionality identification; caption generationn-gcograaphiee context cap-
tioning; contextual similarity--onnextuaal
relationship extraction: contextual
Concept Extraction
similarity-cross-modaa context expansion;
image-text localization--catiinn region
sociation; keyphrase eestraction-hheaarchhh
keyphrase extraction; keyphrase
extraction--frqquenny-bb
keyphrase
extraction; relationship extraction~aggent
action relationship identification; relation-
ship extraction~~causal relationship iden- tification; relationship extraction-object. action relationship identification; summary -aneration--bllaneeed summary,.... .............
Continued on next page
35

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
3D model generation from text~3D object reconstruction; 3D model gen- eration from text=~Image-buee texture generation; adaptive theme-based art expansion using text~cooteettawwrt embellishment of art based on text; adap- tive theme-based art expansion using text~<coherence evaluation across multi- modal elements; adaptive theme-based art expansion using text~hhierarchicl theme structuring from text; critique and suggestion-~color analysis; art critique and suggestion--comtextuul relevance; cri- tique and suggestion--technical proficiency critique; artistic scene composition--color palette matching; collaborative art cre- ation with text=<crowdsoureed art projects; conceptual visualization from narrative text--potrr-innpired illustrations; context- aware adaptation-oobeet removal:.....
Interactive Art Creation
document summarization; anomalous region detection-lesion detection; anomalous region detection-ddeoomatioo detection; anomalous region detection--obstruectie de- tection; biomarker identification-preiiiie biomarker
biomarker discovery;
identification-radiinii feature extrac- tion; clinical trial matching--Imaging Biomarker Identification; clinical trial matching-~Clinical Trial Summarization; clinical trial matching--Patient Profile Con- struction; data annotation-diisasee classifi- cation; data annotation-~report generation; disease diagnosis~eye disease detection; disease diagnosis-ttmoor classification;
disease diagnosis-autoinmmun disease
Medical Imaging Analysis
detection; disease diagnosis-diabetes. related imaging analysis; functional mapping- identification; image classificationn-oraann segmentation; image classification-tiisuee type classification;
image clasification-inagee quality assess- ment; medical captioning--anatomiiall
structure captioning:
medical
captioning-diagnostie summary captioning: medical captioning--proeeduuee description
captioning: medical captioning--lesion
detection captioning: patient outcome prediction-comppication risk assess- ment; patient outcome prediction-disease progression prediction; assistance--training and simulation; surgical assistance-auggmente reality visualization; treatment planning--tumor localization; treatment planning--risk assessment.....
surgical
Continued on next page
36

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
text-based image description; image to mo- tivational quote generation; image to narra- tive generation; image to multilingual article generation; dynamic image caption genera- tion; fine-grained image understanding and text generation; image to paragraph genera- tion; image-based dialogue generation; con- textual image description generation:.....
Multimodal Translation
text-based; image-based; combined media-based; image-text based~image
focus; image-text based=bbalanced focus;
Multiple Choice Questions
image-text based~text focus; image-text
based-<contextuall inference; image-text based-ttemporal understanding:
activity recognition~~sleep behavior
analysis; activity recognition~aaniaa
behavior recognition;
activity
recognition--gesuree recognition; activ- ity recognition--group activity recognition;
aption gencration--tyll-speiie captioning: context reasoning--cultural context detec- tion; event detection~evvent duration esti- mation; expression recognition-eepressioo detection in group images; expression recognition--ose estimation for ex- pressions; gesture recognition--gesture pose estimation; human-object in- teraction detection-iinteraciion dy- namies modeling: human-object in- teraction detection-iiteraation lo- calization; human-object interaction detection--interactive object detection inter- action intection--rllationsii extraction;
Scene Understanding
scene muderstanding--human-ooo
relationship detection-humman-hume
relationship: relationship detection--human-
environment relationship: relationship
detection--human-bbee relationship: scene classification--uraan scene classification;
scene clas- sification; scene classification--reeee
scene classificttonn-commnee
scene classification;
description--ojeet identification; scene description--scene composition evaluation; scene description--backgrundd element identification; visual reasoning-sspatial reasoning: visual reasoning--contextuaal reasoning:.....
Continued on next page
37

---

Published as a conference paper at ICLR 2025
Continued from previous page
Level-1 Task Type
Level-2, etc. Task Types
target recognition in paintings; target recog- nition in sketches; target recognition in clip art; target recognition in doodles; target recognition low-resolution images; target recognition in photographs; target recogni- tion in cartoons; target recognition in info- graphics; target recognition in X-ray images; target recognition in ct images; target recog- nition in 3d rendered images; target recogni- tion low exposure images,.....
Target Recognition Special Image Domains
business--eadership besinsss-
changes; and acquisi- report-
business-ffnancial ing: business--ooppettiive analysis; business--corporate social responsibility;
tions;
education--student proficiency assess-
ment; education--educctionn resource
identification: education--dducation level identification; emotion detection-anger
detection; entertainmennt-event clas- sification; entertainmenn--comiic book identification: health--exereise and fit- ness tracking; health--treatment rec- ommendation; health~-mental health assessment: lifestyle-heelth and well- ness; lifestyle-hhobies and crafts; politics--electin event detection; politics--politcall sentiment analy- sis; politics-political stance detection; scienco--gcoooy;: science--computer
Topic Classification
science; sports>-matc predictions; sports--transfer news; technology-softtware
categorization; technology~induutry trends; travel~-travel safety information extraction; travel~travel tips extraction:.....
38

---

Published as a conference paper at ICLR 2025
A.11 SAMPLES IN TASKGALAXY
In the main paper we provide a small number of sample question-ansser pairs corresponding to task types, and in this section we provide examples.
Task Type: ocr-bill ocr-medical bill recognition
Question: What is the smallest charge among the listed providers, and which provider charged Answer: $2,230.00 by St. David's Rehabilitation Hospital.
Figure A-13: Task Type: ocr~bill mer-mel bill recognition
Dovid Cleston Carral
Task Type: ocr-book ocr-metadata extraction
The Complete QDRO Handbook
Question: Who is the author of the book mentioned on the cover?
Answer: The author of the book is David Clayton Carrad.
Dhviding ESIDA Malterr, Collectnn CURLE Exgpkon
Figure A-14: Task Type: ocr~<book ocr-mmttadata extraction
sawp-aaaaaiiiiiii-----eeeee event description Question:ls the child participating in sports activity, and how would you describe the context?
Task Type: image
Answer: Yes, the child is engaged in sports activity, specifically skatehoarding. The context is likely casual practice indoors, suggesting the child learning playing at skateboarding rather than participating in formal sports event.
Figure A-15; Task Type: image descippionnacctityy-beee--eeeeed event description
39

---

Published as a conference paper at ICLR 2025
Task Type: image pnterpppoonaatoooogapppilllallld
activities Question:Whn daily activity the person in the image engaged in? Answer: The person in the image is engaged in the activity of pouring wine into cup, which could be part of social personal relaxation activity.
Figure A-16: Task Type: image dececipdoonaauoooiirrppiaaaaaaed activities
Task Type: image description-cuuluree seallure-based-arccheeeee and landmarks Question: you describe architectural style and design elements of the building shown the image?
Answer: The building in the image exhibits blend of Gothic, Renaissance, and Baroque architectural styles. It features a tall, slender tower with decorative elements and windows, typical of Gothic and Renaissance architecture. The façade adorned with classical Baroque details, including omamental balconies and intricately designed cornices.
Figure A-17: Task Type: image deccippioo-alttn---aaddaaaaeeeee and landmarks
Task Type: image deserpptin-cccuciiioooooooommoss tutorial
Question Provide step-by-step tutorial bascd the images for educational content setting.
Answer: Step I: Usc rolling pin to roll out the dough to even thickness (image 6). Step 2: Take pizza cutter and begin cutting the dough into long, even strips (image 7). Step 3: Rotate your cutting direction 90 degrees and cut again to form grid pattern (image 8). uniform baking.
Ensure the dough is evenly
Figure A-18: Task Type: image http://iinnccccciooooooooooooeed
tutorial
Task Type: image deserrptii---mmtiii-----ddd-oood storytelling
lagpy endings. Endings are the saddest part, So jus give me a happy middle And very happy stoet.
Question: Descrrbe the image as part of an emotion-drre -eationn-drienn story. Answer: In the realm of literature, where Shel Silverstein's 's words emotions into the fabric existence, people didn't 't fear the end, for they celebrated the beginnings with radiant smiles and lived the middles with unguarded joy, creating memories made every end soem but a brief intermission in the ongoing narrative of happiness.
quote serves as reminder. The
Figure A-19: Task Type: image dacepptorn-cmmiioooooccccceeoooi storr storytelling
40

---

Published as a conference paper at ICLR 2025
Task Lype:detection-oojeed detection-singlt object detection-firfiighing equipment detection
Question What firefighting equipment shown in the image?
Answer: The image shows fire hydrant.
Figure A-20: Task Type: detection-oojeet detection--single object detection deteection-firefighhtig equip-
ment detection
Task Type:dltetionnoobjed detection-ssigll object detection- vetiiio--vveiiiinnttrrriio dctection Question: Which specific vehicle brand is detected in this image based on its interior components?
Answer: vehicle brand detected based its interior components, such as the steering wheel logo, Mercedes-Bent
Figure A-21: Task Type: detection--ohject detection--single object delection--vehccle-iintiioo de- tection
Task Type:dttection-targe detection -cene detection-train traffic scene detection
Question:Whns What kind of traffic scene is depicted in the image?
Answer: The image depicts night-time urban street with ongoing roadwork on the left side and eyclist the distance.
Figure A-22: Task Type: detection--target detectionn~scene detection--traffic scene detection
Task Type:amalysi--color analysis-color contrast analysis Question: What aspects of color contrast are significant in this image? Answer: The significant aspect is the contrast between the warm, bright colors of the autumn foliage and the cool, neutral tones of the rocks and water. This contrast creates a visually appealing composition, drawing attention the leaves while maintaining natural, balanced setting.
Figure A-23: Task Type: analysis analysis contrast analysis
41

---

Published as a conference paper at ICLR 2025
Task Typeeanalysis-design concepts and intentions-evallute layout and composition
Question:Wine is the evaluation of the layout and composition of the furniture in the image?
Answer: The layout efficient and cohesive, with L-shaped sectional maximizing seating and the coffee table centrally placed for convenience.
Figure A-24: Task Type: analysis-desiin concepts and intentions--evallute layout and composition
Task Type:analysis- fashhoo analysis identification Question:For which scason is this jacket most suitable? Answer: This jacket is most suitable for winter or late fall, given its thick material and protective design.
Figure A-25: Task Type: analysis-ffastion analysis--season identification
MEMPROW
Task Type:analysis-geenn gender representation malysis-gender representation in advertising analysis
Question:How does the image represent gender the advertising?
Answer: The image represents gender in advertising by focusing on women's empowerment and the importance of femalc leadership.
Sustaining women's leadership
Figure A-26: Task Type: analysis-gender representation analysis--gnnder representation in adver- tising analysis
Task eyperanalysis-politiee sentiment auuesionnanallsii-poiiiii sentiment analysis- polarization detection Answer:Yes the image depicts politically polarized environment, evidenced by a large group of protesters holding signs and flags, indicating strong stance media freedom which suggests significant division in opinion.
detection
Figure A-27: Task Type: analysis-polltical sentiment analysis- detection
42

---

Published as a conference paper at ICLR 2025
Task Type:content crcation- advieriiidn content rreation-brochure content creation
livfun
Question:Whie What type of imagery and content from the image would be
effective a promotional brochure?
Answer:Images of scenic travel destinations like Alaska and articles on wellness and personal empowerment would be engaging. These elements can visually appeal to prospects looking for enriching and adventurous experiences, reflecting Leisure Care's offerings.
STRENGTH
WISDOM
Figure A-28: Task Type: content creation content creation--brochun content creation
Task Type:content creation-ar art riitiue-teehniqqu and brushwork examination
Question: What can be inferred about the artist's brushwork and technique in this painting?
Answer:Thh The artist employs loose and expressive brushwork technique that captures the texture and form of the architectural elements. The thick and energetic strokes suggest quick and confident application of paint, particularly noticeable in the depiction of the stone surface and the balustrades. This approach adds dynamism and sense of movement to the otherwise static architectural subject.
Figure A-29; Task Type: content creation-art crtt--utciitttue-eeechhiiiuee and brushwork examination
Task Type:content creation--diveritty and inclusion content crattonn-olebration of cultural events
Question:Hov does this image exemplify diversity and inclusion in the celebration cultural events?
Answer: The image exemplifies diversity and inclusion by showcasing group of people from various backgrounds coming together celebrate a cultural event, sharing in traditional practices, attire, and communal activities like smoking hookah and enjoying food.
Figure A-30: Task Type: content creation--diverityy and inclusion content creatton--celebrattin of cultural events
Task Type:content creation-letter writing-closing statement formulation Question: What would be an effective closing statement to conclude the letter of recommendation for Denise Ball?
Answer:l strongly believe that Denise Ball would be tremendous contribution to the Department of Education. She definitely has the knowledge, leadership qualities, and interpersonal skills that needed to be successful.
Figure A-31: Task Type: content creation~-leteer wliting-closing statement formulation
43

---

Published as a conference paper at ICLR 2025
Task Typessugeetions-weddig planning
Question:How does the image provide inspiration wedding planning? Answer:The image inspires wedding planning by displaying a beautifully dressed bride and elegantly dressed groom, set in scenic outdoor locations, suggesting theme an outdoor or garden wedding.
Figure A-32: Task Type: suggestions-wwedding planning
HOLMIES
Task sypessuggestion--mvvi suggestions
Question: What movies would be good to watch you enjoy 'Holmes & Watson"?
Answer:lf you enjoy 'Holmes & Watson', you might also enjoy movies like 'Sherlock Holmes' (2009), 'Sherlock Holmes: A Game Shadows' (201 'The Great Mouse Detective', and 'Enola Holmes'.
Figure A-33: Task Type: suggestions- suggestions
Task rypessummarizaiion-spprr event summarization the sports event displayed in the image. Answer:The image shows congratulatory moment where athletes from various countries celebrate their participation in a Winter Youth Olympic Games event, holding mascots and posing in front of Olympic symbols.
Figure A-34: Task Type: summarization-ssorts event summarization
Task Type:logical reasoning-analogy reusoning-scene relationship analogy reasoning
AMARELÃO Ancilostomose
Question:How are the
the comic strip related by analogy?
Answer:Both scenes are visual analogies representing symptoms ancylostomiasis. The first scene with the scale indicates extreme weight loss, while the second scene with the man sitting and holding his head depicts weakness and fatigue.
Figure A-35: Task Type: logical anasonyg reasoning relationship analogy reason- ing
44

---

Published as a conference paper at ICLR 2025
Task Type:logical reasoning-complex reasoning-complex mathematical calculations goometric peluuations-ggommtrri mathematical operations Question:IfO is the center of the circle and angle AOB given as x degrees, what is the measure of angle ACB?
0
Answer:Angle ACB is half of angle AOB because an angle at center of the circle is twice the angle the circumference the same arc. Therefore, ACB x/2 degrees.
C
Figure A-36: Task Type: logical reasoning--complex reasoning--complex mathematical calculations mathematical operations
Task Type:logical reasoning-complee reasoning- reasoning
Question: What can be inferred about the importance of the military in the context of this historical event?
Answer:The presence multiple high-ranking military officials suggests that the military holds significant importance in the context of this historical event, possibly indicating ceremony announcement with major national security implications.
Figure A-37: Task Type: logical reasoning--complex reasonigg-miliarr--eeted reasoning
Task Type:logical messoing--copllx reasoning-complex mathematical calculations geometrri mathematical operations
Question:If If the circle's radius is extended to 10, how will this affect the
12
7
length of segment WV, given that length of UW remains 12?
V
Answer: the circle's 's radius VU is extended to 10 while UW remains 12, Pythagorean theorem again. UW*2 WV*2 VU*2.
W
X
Substituting the new values: 12 = x + 10"2. So, 144 = 100.
Therefore, x 144 100 = 44. Thus, the length of WV when the circle's radius extended to sqrt(44).
Figure A-38: Task Type: logical reasoning--complex reasoning--complex mathematical calculations-ggommeefe mathematical operations
Task Type:logical reasoning- simple reasoning- basic mathematics-simple numerical calculation reasoning Question:If and b- 4, what is the value of c? Answer:Using the Pythagorean theorem: hence c 5.
4 9 25,
b c²= a b²
Figure A-39: Task Type: logical reasoning-ssimpl reasoning~-basie mathematics--simple numeri- cal calculation reasoning
45

---

Published as a conference paper at ICLR 2025
Task Type:logical leasoning-complles manooninn-miiitryyrelleeee reasoning
Question: What can be inferred about the relationship between the two people based on the individual's attire and their interaction with each other?
Answer: individual's military attire suggests that they likely in military service, and the casual attire and hand-holding suggest
personal, likely romantic relationship.
Figure A-40: Task Type: logical ceasoningg-cmmlee reasnnng--mlittrr--rleeed reasoning
Task Type:logical acasoning-spatinnn rcasoning-33 scene reconstruction :uestion:How are the objects spatially arranged with respect to cach other the image?
Answer: The objects are spatially arranged as follows: The yellow cube is the foreground the left, the blue cube the largest and is centrally placed the right of the yellow cube, the red sphere is to the right of the blue cube, the gray cylinder is in the background to the far left, and two smaller cylinders (purple and turquoise) are between the gray cylinder and the blue cubc, with the purple one bcing slightly farther back.
Figure A-41: Task Type: logical reasoning--spatial reasoning-~3D scene reconstruction
Task Typereefual--refsut to drug-related content-recoonnzingg drug
names and slang
Question:Can recognize any drug
slang in this image?
Answer: image does not contain text or slang, but the visual elements suggest heroin or similar injectable drugs.
Figure A-42: Task Type: refusal-refusal due to drug-related content--recognizing drug names and slang
Task Type:safety and risk assessment-ffe safety and risk assessment--ost-iniddee assessment post-incident damage assessment Question:Wins What evidence from the image can be used to assess the extent of damage caused by the wildfire?
Answer: The charred and blackened areas of the ground and vegetation indicate significant burning, suggesting considerable extent of damage affected area.
Figure A-43: Task Type:
safety and risk assessment-fire safety and risk assesmment-poxt-inddee
damage assessment
46

---

Published as a conference paper at ICLR 2025
Task appessiinne-rclate--img---- alg.rmmaee-eaiiiiaiiaaaaeeeeee aligament-labeling diagrams with text explanations
Question:Label the key components and forces shown in both diagrams. Answer:In both diagrams: The 'Earth' labeled at the center. The 'Ocean' surrounds the Earth, represented by the light blue areas. The 'Moon' is positioncd to the right, exerting gravitational pull. Arrows indicate the direction of water movement: (a), toward and away from the Moon, showing high tides; in (b), perpendicular to the Moon-Eate -arth line, indicating low tides.
Figure A-44: Task Type: scienc-relatedd-imggeeee alignment--laellng diagrams with text expla- nations
Task Iypesseience-related--e description cencration-biologeen process illustration
Trplerteierr
Quscrion.Describe the biological process that this image illustrates. Answer: This image illustrates a blastocyst, an early stage embryonic development. The blastocyst contains outer layer of cells called the trophectoderm, inner cell mass that will develop into the embryo, and fluid-filled cavity known as the blastocoel.
Men
auy
Figure A-45: Task Type: science-relatdd--maag description generation--bologicaa process illustra- tion
Task r visualization interpretation bar interpretation
Question: What can be inferred about relationship between service exports and imports of Nicaragua from 2006 to 201 1?
Answer:From 2006 to 201 both service exports and imports of Nicaragua show an upward trend. However, exports consistently surpass imports throughout the period.
Figure A-46: Task Type: science-related--dtta visualization interpretation--erro bar interpretation
Task Type: seience-related-viiee reasoning- anatomical diagram labeling Question: Label the major muscle groups shown the figures. Answer: In the male and female figures, the major muscle groups include the pectoralis mujor (chest), rectus abdominis (ubs), external oblique (sides), quadriceps (front thigh), deltoid (shoulder), biceps brachii (urm), and gastrocnemius (calf muscles).
Figure A-47: Task Type: science-relateed-vvisud reaaoooiiinsirrrraareeeeeeeeeeee reasoning--anatomiicc diagram labeling
47

---

Published as a conference paper at ICLR 2025
Task Type:subject question and pereee--ccccnceeggggapphh studies Question: how soil creep affects the landscape.
What creep it a extremely slow movement of material down slope
Answer:Soil creep affects the landscupe by causing gradual downhill movement of soil and rock matcrial. This process lead curved tree trunks, tilted fence posis, und distorted roadways as the ground slowly shifts over time.
Figure A-48: Task Type: subject question and answer-sciennee--gggaaly studies
Task Type: subject question and answer-hummiliss & social sciences-history studies
Question: Can you name famous historical figure associated with the institution shown the image?
Answer: A famous historical figure associated with Trinity College Dublin is Jonathan Swift, the author of Gulliver''s Travels', who student there the late 17th century.
Figure A-49: Task Type: subject question and answer--humanttis social sciences-history stud- ies
Task Type:subject question and musser-scicccc--hyiic studies Question:How does the spacing of charges differ between configurations (a) and (b) in image?
Churges on (and inj conductor
ando infemme of snaue
diages cnagee thet
Answer:In configuration (a), the charges are more concentrated because the spherical shape minimizes surface area for given volume.
pow the wore
configuration (b), the charges are farther apart to cover a larger surface area, which helps to distribute the charges evenly and minimize electrostatic repulsion.
epaeerice por jnd
adic cnd
acnde aersec?
Figure A-50: Task Type: subject question and answer-ssiince--phyiiis studies
Task Type: subject question and answcr-arts & arts arts-music studies Question: In the context of an orchestra, what rolc docs the clarinct scction typically play?
Answer: an orchestra, clarinet section often provides rich and melodic mid-range tones, capable of versatile dynamics and expression, bridging the higher flutes and lower brass and strings.
Figure A-51: Task Type: subject question and answer>aarss & arts~musiic studies
48

---

Published as a conference paper at ICLR 2025
Task Type:3D object recognition-Mattiin and texture aocognition-Materin and damage recognition from images Question:Whan What the material of sculpture image? Answer: The sculpture is made of ceramic.
2022
2024
Figure A-52: Task Type: 3D object recognition-Mattrial and texture recognition- wear and damage recognition from images
DEPOSITION VS. ROOT CANAL
Task Type: concept extraction-texx classification humor detection Question: What the main theme the image based on the text visual cues?
Answer: The main theme of the image is comparing the discomfort of legal deposition to root canal, with humorous tone focusing on minimizing the unpleasantness of dental work.
Pat Getter
Figure A-53: Task Type: concept extraction--text classificttonn-hmmrr detection
Task Type:context-awant recommeendaaioos--esssn and holiday recommendations
Question: What recommendations would you provide someone visiting this seasonal market for the first time?
Answer:For first-time visitors, it's recommended explore the market during evening to fully experience the festive lights. Start walk through the market to get feel of the different stalls, try some local holiday treats, take plenty of photos of the lit decorations and historic buildings, and consider purchasing unique handcrafted gifts.
Figure A-54: Task Type: context-aware recommendations--scaaane and holiday recommendations
Task Type: context-aware recommendations-conteee contextuul chatbot responses Question: What should chatbot response be if a user asks about what makes Badoo different from other dating apps?
badoo
Dating Free Chat
than just dating app; it's community where you can make real connections. It offers a unique blend of features that help you meet new friends, date, and even network. Plus, with our robust safety measures and diverse user base, you're safe hands.
Answer: Badoo stands out by being
Dadio
Figure A-55: Task Type: context-wwar www.://wwwerreommmnnnnnnnnnoonno chatbot responses
49

---

Published as a conference paper at ICLR 2025
Task happeiimage-baseee knowledge distillation- recognition Question:Identtin the objects present the image and their primary function.
The image shows two coffee grinders and sign reading 'KASIR' which typically indicates cashier's area. The primary function of the coffee grinders is to grind coffee beans, and the 'KASIR' sign indicates where customers puy for their purchases.
KASIR
Figure A-56: Task Type: image-basee based knowledge distillationnoobeet recognition
Task Type: image-text matching- single image-visual reasoning Question: Based on the visual evidence, how would you describe the transfomation that occurs in the image?
P
Answer: The letter on the left undergoes 90-degree clockwise rolation to become a rotated form that no longer serves its original function as a letter but instead becomes an abstract shape.
Figure A-57: Task Type: image-text matching--snngle image visual reasoning
Task Type:interactive art creation-environment design based on textual lore-narrative coherence verification
Question How can the stained glass art used create an environment inspired by fantasy lore?
Answer:Tho stained glass be used in the design of grand hall temple within fantasy lore setting, where the shifting sunlight streaming through the colored glass creates magical, otherworldly atmosphere.
Figure A-58: Task Type: interactive art creation--nnvrromment design based on textual narratvae coherence verification
Task Type: medical imaging analysis-contee content extraction-diseat identification
Question: Which specific lung condition can be identified from chest X-ray image?
Answer: image indicates the presence of pneumonia, characterized by the consolidations and opacities seen the lungs.
Figure A-59: Task Type: medical imaging cnatesis--content extraction--diseaee identification
50

---

Published as a conference paper at ICLR 2025
Task Typermultimodal translation -ext-based image description Question : Provide a detailed explanation of the signboard text and its visual features.
Answer: The image features two illuminated vertical signboards displaying bilingual text. The Chincse characters A X 'x' are in large white font blue background. Below the Chinese text, the English translation GUANGDONG, HONG KONG. AND MACAO GREATER BAY AREA' is also displayed in white font. The signboards positioned within modern architectural environment.
GUNN
BOND KONGLAND
MACAO GREATER
BAY
mudo
Figure A-60: Task Type: multimodal translation--eet-bbase image description
Type: multiple choice questions-image-ten based-image focus If the diameter of the circle is 20 units, what is the length of AO? A) units B) units C) 15 units D) 20 units
Question:
O
Answer: B) 10 units
2024
200
20
B
Figure A-61: Task Type:multiple choice questions-immae-texx based~image focus
51


Reviews:
[{"review_id": "JjfFaZcqYb", "reviewer": "Reviewer_U9mU", "summary": "This paper proposed a new instruction tuning dataset generation pipeline and a new dataset consist of ~19K tasks and ~400K samples. Through a hierarchical task defination, it aims to make diverse task types and further enhance diversity of dataset. By instruction tuned on proposed TaskGalaxy dataset, MLLM show improved performance on various benchmarks.", "questions": "See in Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The pipeline proposed by the paper is diverse and scalable. It aims to increase sample diversity of instruction tuning, which is an key problem in relative field. Tuning with proposed dataset brings further improvements of MLLMs.", "weaknesses": "1. As the task type generation is a hierarchical process, provide some examples or statistics of level 2/3 tasks can be more clear. For example, provide a list of tasks with the most/least sample, analysis of image resolutions, distribution of tasks across levels.\n2. The diverse task types actually can improve image diversity. However, image sources are still similar with existing datasets, whose diversity maybe limited. Overall review of images sources, such as the proportion of samples from different data sources, is also necessary to understand the dataset.\n3. I think the experiments are not convincing enough. As baseline is instruction tuning with LLaVA-665K, TaskGalaxy finetuning actually increase the amount of data, which is shown to be a key factor for model performance. Comparing with baseline LLaVA-Instruct only and TaskGalaxy only with equal amount would be better.\n4. Apart from comparison with baseline, there are several high-quality instruction tuning dataset such as LLaVA-OneVision[1], ShareGPT4V[2] and etc. Meanwhile, I think works like ShareGPT4V should be included in related work. Comparison between those works are more important to show effectiveness of TaskGalaxy, performance of models tuned with the same amount of data samples from TaskGalaxy, LLaVA-665K, LLaVA-OneVision and ShareGPT4V is more convincing.\n5. To comprehensively present properties of TaskGalaxy, a detailed dataset card is preferred to show more about statistics or usage. Examples can be found at [this link](https://arxiv.org/abs/2204.01075).\n\n[1] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).\n\n[2] Chen, Lin, et al. \"Sharegpt4v: Improving large multi-modal models with better captions.\" arXiv preprint arXiv:2311.12793 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zijFoCDxYN", "reviewer": "Reviewer_EeP4", "summary": "This manuscript introduces a novel multi-modal instruction fine-tuning dataset called TaskGalaxy. TaskGalaxy proposes an automated pipeline comprising five main steps, which leverages robust multimodal model and reduces manual intervention. The proposed TaskGalaxy consists of tens of thousands vision task types and 413k samples to address the limitation of task diversity in existing datasets. The performance of LLaVA-V1.5 and InternVL-Chat-V1.0 model incorporating TaskGalaxy fine-tuning is improved across 16 benchmarks.", "questions": "1.\tTypos: In the caption of Figure 1, The spelling of \" llustration\" is wrong. \n\nPlease address the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is easy to follow. The details of data construction and experiments are clear.\n\n2.\tTaskGalaxy contains a rich set of task types, alleviating the impact of limited task types on model generalization ability.\n\n3.\tThe pipeline of TaskGalaxy can be flexibly expanded to create more high-quality data.\n\n4.\tIncorporating this training data, the performances of LLaVA-V1.5-13b and InternVL-Chat-V1.0-7b have been improved.", "weaknesses": "1.\tIn Hierarchical Task Type Generation, this process resulted in the generation of 19,227 hierarchical task types. But the task types are completely GPT-4o generated, how to ensure that 19,227 task types do not overlap with each other? The diversity of the tasks and data are neither thoroughly nor rigorously studied. There is no in-depth comparison to previous tasks and data. \n\n2.\tSince the improvements are obtained by training on baseline data + TaskGalaxy data, it is not possible to tell whether the improvements are due to the larger data amount or the larger task number. \n\n3.\tIn Match and Filter section, images and task types are encoded by the off-the-shelf CLIP respectively to get the embeddings. As CLIP is trained mostly on data of natural concepts, filtering images with abstract phrases such as task type may be not as effective as expected. \n\n4.\tThe performance of the baseline is too weak to prove the validity of the dataset. For example, InternVL-Chat-V1.0-7b only achieved the performance of 35.96% on AI2D and 15.2% on ChartQA. More advanced models, such as LLaVA-V1.6 and InternVL-Chat-V2.0-8B, should be evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BD6J0AWNtE", "reviewer": "Reviewer_VWt8", "summary": "The paper introduces a finetuning dataset for Vision-Language finetuning called TaskGalaxy, consisting of 19k hierarchical task types. The data is synthetically generated starting from seed tasks and contains 413k samples.\n- First two levels of hierarchy are defined manually, and GPT-4o is prompted to expand the hierarchy\n- The task type names are used to match images from multiple open source datasets using CLIP to match the text to images\n- Given the top 10 matched task types to a given image, GPT-4o is used to filter out the irrelevant task types\n- Based on the image and task types, GPT-4o is prompted to generate question-answer pairs\n- Finally, 3 open source models (GLM-4v-9B, InternVL-Chat-V1.5 and InternVL2-26B) are prompted to score the generated question, task type and image with a score of 1 if the question and task type match relative to the image, and 0 otherwise. Samples are filtered through majority voting.\n- The generated dataset is used to finetune models on downstream tasks, showing improved performance.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a scalable synthetic data generation pipeline across multiple task types from publicly available datasets.\n2. Models finetuned with the generated dataset in addition to the original datasets show improved performance, which shows the effectiveness of the proposed dataset.\n3. The dataset is ablated well to check the effect of varying the number of task types and number of samples for each task type, with higher number of task types and samples showing better performance, ~5% improvement on average (Table 2 and Figure 6).\n4. The paper proposes a multi-stage pipeline for filtering the genearated samples, which can reduce the noise on the generated data.", "weaknesses": "1. **Check for hallucinations and incorrect responses**\n- The paper does not have an explicit check for hallucinations and incorrect responses. Prior LLM generated datasets have had problems with hallucinations, leading to subpar performance on downstream tasks. Upon resolving the problems, the performance on downstream tasks typically increases [1]. While the multi-stage pipeline can potentially eliminate hallucinations, especially when 3 models are used as validators, the method and dataset will be more trustworthy if an explicit check is performed.\n2. **Scope and impact**\n- In the current form, the proposed dataset and generation pipeline is limited to building a good finetuning/visual instruction tuning dataset.\n- The paper does not provide more information about the samples that are rejected during each stage of the filtering pipeline. The rejected samples might provide more insights into the behavior of the generator model and could be potentially valuable for future research into synthetic data generation. Studying this aspect might help broaden the impact of the work.\n- For example, the authors could analyze type of question-answer pairs are typically rejected by the three filtering models, or if there is a pattern to which task types are rejected the most.\n- This might also partially address Point 1 about hallucinations.\n\n---\n\n## References\n[1] Gene Ruebsame, \"Cleaned Alpaca Dataset: https://github.com/gururise/AlpacaDataCleaned\", 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emWQ5Tzaup", "reviewer": "Reviewer_EFVj", "summary": "This paper presents TaskGalaxy, a multi-modal instruction fine-tuning dataset. A pipeline for the systematic construction and generation of a diverse range of task types and corresponding high quality instruction Q&A samples is proposed. The fintuning results of LLaVA-V1.5 and InternVL-Chat-V1.0 validate the effectiveness of the proposed dataset.", "questions": "1. Compared with other existing similar datasets,  what are the advantages of the proposed dataset?\n2. The comparisons of SFT results on various architectures with other similar datasets are suggested to supplement.\n3. In the stage of match and filter, which CLIP model is adopted?\n4. I wonder how the accuracy changes under the same architecture but with various sizes trained with the proposed dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dataset construction pipeline is elegant and makes sense.\n2. 19,227 hierarchical task types and 413,648 associated samples are a lot, I believe it can make contributions to the community.", "weaknesses": "1. As a submission on the area of datasets and benchmarks, the access to data is not provided to reviewers, which is obviously unreasonable.\n2. The experiments are insufficient and lack comparison with other similar datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)