Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models (2024)

Abstract: Large vision-language models (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap, we introduce ArXivQA, a questionanswering dataset generated by prompting GPT4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs’ mathematical reasoning capabilities, achieving a 10.4% absolute accuracy gain on a multimodal mathematical reasoning benchmark. Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs. Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domainspecific training yields substantial performance gains. Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.1

heading: 1 Introduction
text: Large vision-language models (LVLMs), which integrate large language models (LLMs) (Brown et al., 2020a; Touvron et al., 2023) with pre-trained vision encoders through cross-modal alignment training (Madureira, 2021; Liu et al., 2023b; Li et al., 2023d), have demonstrated remarkable perceptual and cognitive capabilities in processing
*Equal Contribution. 1Datasets and models are released at our project page:
https://mm-arxiv.github.io.
concrete images from everyday scenes (OpenAI, 2023; Fu et al., 2023; Yang et al., 2023a; Reka, 2024). However, recent studies have shown that open-source LVLMs struggle to understand abstract figures, such as geometric shapes in multimodal mathematical reasoning (Lu et al., 2023; Zhang et al., 2024b) and scientific plots (Yue et al., 2023). The inadequacy of training datasets in scientific domains that involve complex reasoning with abstract figures is the main underlying cause.
To address this, we construct Multimodal ArXiv by utilizing the rich resources in preprints hosted on arXiv to improve the ability to understand scientific literature in LVLMs. We first curate ArXivCap, a diverse scientific figure-caption dataset. In contrast to previous scientific figure datasets, which consist of synthesized figures (Chen et al., 2020) or are restricted to simple captioning scenarios in the computer science domain (Hsu et al., 2021), our dataset is composed of figures extracted from academic papers across a range of domains. ArXivCap has 6.4M images and 3.9M captions from 572K papers. We also keep the subfigure structure, and titles of original papers, thereby supporting diverse evaluation tasks. We further instruct GPT-4V to generate 100K multiple-choice question-answering (QA) pairs for the figures in ArXivCap. The resulting ArXivQA dataset could naturally serve as a pivotal resource for improving the scientific reasoning abilities of LVLMs.
We validate the effectiveness of our Multimodal ArXiv dataset from two dimensions: reasoning ability measured by QA accuracy and generation performance through novel vision-to-text tasks. Our experiments demonstrate that ArXivQA brings a significant 10.4% absolute accuracy boost for Qwen-VL-Chat (Bai et al., 2023b), on the MathVista (Lu et al., 2023), a challenging benchmark for multimodal mathematical reasoning. Additionally, detailed analysis uncovers the relationship between paper domains and fine-grained task performance.
14369
Moreover, using ArXivCap, we define four generation tasks of varying complexity to benchmark the ability of LVLMs to comprehend scientific plots: (1) captioning a single academic figure, (2) generating overall summaries for multiple sub-figures, (3) in-context figure captioning given previous figurecaption pairs, and (4) generating paper titles from figure-caption pairs. We examine various LVLMs, including open-source models as well as proprietary models including GPT-4V (OpenAI, 2023) and Gemini 1.0 Pro Vision (Gemini Team, 2023). Evaluation results reveal that despite that current LVLMs still face challenges generating faithful captions for scientific figures, in-domain training on our dataset yields substantial performance improvements across all four tasks. Manual error analysis underscores that LVLMs still suffer from misinterpretation of the visual context, recognition errors, and overly simplified captions, paving the way for future studies.

heading: 2 Related Work
text: Recent advancements in LVLMs have seen notable progress in model architecture, training paradigms, and dataset creation (Zhang et al., 2024a).
Model Architecture LVLMs typically comprise three core modules: (i) a vision encoder for image feature extraction, (ii) a modality alignment module to integrate visual features into the language model embedding space, and (iii) an LLM backbone for decoding multimodal context. CLIP (Radford et al., 2021) is widely used for image encoding, while LLaMA (Touvron et al., 2023) and Vicuna (Chiang et al., 2023) serve as popular choices for LLMs. The alignment module varies from simple linear projections (Liu et al., 2023b; Zhu et al., 2023) to more complex architectures like gated cross-attention layers substantiated by Flamingo and IDEFICS (Alayrac et al., 2022; Awadalla et al., 2023). Innovations such as the Q-Former module in BLIP2 (Li et al., 2023b) and instruction integration in InstructBLIP (Dai et al., 2023) further enhance alignment capabilities. Additionally, Fuyu-8B (Bavishi et al., 2023) introduces a novel framework mapping raw image pixels directly to the LLM embedding space.
Training Paradigms Regarding the training recipes, PaLI-X (Chen et al., 2023b) investigates the scaling effects of both vision encoders and language models, highlighting the advantages of scal-
ing both components. Qwen-VL (Bai et al., 2023b) increases input image resolution and explores different module unfreezing strategies. Alignment methodologies such as RLHF training (Ouyang et al., 2022), e.g., LLaVA-RLHF (Sun et al., 2023), and preference optimization through AI feedback (Li et al., 2023c) demonstrate effectiveness in aligning LVLMs with human preferences.
Dataset Curation Dataset quality significantly impacts LVLM performance. Modality alignment training often utilizes web-scale image-caption pairs such as Laion-400M (Schuhmann et al., 2021), with recent studies favoring cleaned captions (Chen et al., 2023a; Yu et al., 2023). Instruction fine-tuning (IFT) helps LVLMs respond according to user queries, triggering the exploration of high-quality IFT datasets. Efforts include multimodal instruction collections such as MultiInstruct (Xu et al., 2023) and M3IT (Li et al., 2023d), dialog-style datasets such as LLaVA (Liu et al., 2023b) and domain-specific datasets for medical (Li et al., 2023a) and text-rich images (Zhang et al., 2023). In the scientific domain, FigCAP (Chen et al., 2019) and FigureQA (Kahou et al., 2017) are created based on synthetic figures. DVQA (Kafle et al., 2018) creates heuristicbased questions for bar charts only. SciCap (Hsu et al., 2021), SciCap+ (Yang et al., 2023b), and M-Paper (Hu et al., 2023) collect figure-caption pairs from specific domains such as computer science. Compared with these datasets, our ArXivCap is sourced from diverse scientific domains with a much larger scale, enabling more comprehensive improvements and evaluations. Besides, we employ GPT-4V for creating ArXivQA with challenging questions, showcasing its effectiveness in boosting the mathematical reasoning ability of LVLMs.

heading: 3 Multimodal ArXiv
text: This section presents a detailed construction process of our Multimodal ArXiv dataset, consisting of two sets: ArXivCap (§3.1) and ArXivQA (§3.2).

heading: 3.1 ArXivCap
text: Construction Process We outline the creation process of ArXivCap below and Figure 1 gives an overview. Paper Filtering with Publication Type: ArXivCap is extracted from ArXiv (Clement et al., 2019), which is under CC-0 licence for modification and distribution. The raw files of papers posted on
ArXiv tar files before June 2023 are downloaded. To ensure the quality of our dataset, we employ a rigorous selection process to filter potentially low-quality papers that might influence the figurecaption pair quality. Firstly, we retrieve metainformation for papers from Semantic Scholar (Kinney et al., 2023), which contains the publication record for each paper. Papers with publication types JournalArticle, Conference, or Review are kept as we assume the peer-review process could ensure the overall figure-caption quality is satisfactory. We further exclude papers with titles exceeding 100 words or abstracts longer than 300 words, in alignment with common submission requirements. Figure-Caption Pair Extraction: Images and captions are extracted from the original LaTeX files by matching the syntax. We further use a robust tool ImageMagisk (ImageMagick Studio LLC) to convert images into JPEG format for easy processing. The extracted images and captions are stored in a designed chunk structure, which consists of either a single figure-caption pair or multiple figures with their respective sub-captions and a main caption for the overall description. This format is more consistent with the layout of academic papers, and Figure 2 illustrates the chunk structure. Caption Cleaning and Image Filtering: After a manual inspection of the initially collected dataset, we design several transformations to clean the cap-
tions and filter the images. Caption Cleaning: (i) Chunks with captions shorter than 5 words are removed; (ii) For captions with LaTeX expressions such as math formulas and references, we apply the pylatexenc2 to transform the LaTeX to text with math formulas retained, citations to a special symbol <cit.>, references to <ref>. An illustration of caption cleaning can be found in Appendix A.1. Image Filtering: We remove images that are deemed to be problematic according to the following rules: (i) Images with an aspect ratio larger than 100; (ii) Images with the shortest edge shorter than 224 pixels; and (iii) Images with pixel numbers larger than the decompression bombs threshold.
After these processes, 100 pairs are sampled to perform an additional manual inspection, where we found all of these pairs contained clear images and correct caption descriptions. We provide visualized figure-caption pairs in Appendix A.2.
Statistics of ArXivCap Table 1 lists the dataset statistics. ArXivCap consists of 572K papers, containing 6.4M high-quality images in total with 193M words. A word cloud illustration of captions can be found in the Appendix A.3. Figure 3 demonstrates the paper domain distribution extracted from ArXiv, where we find that our ArXivCap covers 32 domains, such as computer science, mathematics,
2https://github.com/phfaist/pylatexenc
physics, and economics. As shown in Table 2, compared with previous scientific figure datasets, our ArXivCap is the largest figure-caption dataset collected from real papers and covers a wide range of scientific domains, serving as a valuable resource for improving and benchmarking LVLMs.

heading: 3.2 ArXivQA
text: As our ArXivCap contains diverse images from scientific domains, we assume that learning to answer questions about these figures could boost scientific reasoning ability. Following the successful practice of LLaVA (Liu et al., 2023b), we adopt GPT-4V to generate instruction-tuning datasets for generating the QA pairs based on the figures extracted from scientific papers. Specifically, we design a prompting template to query GPT-4V for generating QA pairs based on 35K images randomly sampled from our ArXivCap. Table 11 in Appendix A.4 provides the template we used for the prompt. The generated pairs are parsed according to the format requirement and we discard the samples without options
and rationales. There are 100K QA pairs after filtering the invalid samples. The dataset comprises questions with an average word count of 16.98 for the question text. On average, there are 4.20 options per question and the average length of the text for a single option is 7.59 words. Appendix A.2 provides samples from the ArXivQA dataset.
As a preliminary study, we sample 1,000 samples from ArXivQA and prompt open-sourced LVLMs to predict answers given the questions and options. A simple prompt is designed to employ GPT-4 for extracting the answer label from the model generations. For human performance, we ask four authors to perform predictions on a 100-sample subset (where 17 samples are from the CS domain). Each of them is asked to answer 50 samples and the accuracy scores are obtained by averaging two annotators. As shown in Table 3, most models struggle to perform satisfactorily on the ArXivQA dataset, falling far behind human performance. This verifies our premise that current open-sourced LVLMs fail to understand scientific figures. We also notice that simply increasing the model scale from 7B (LLaVa-1.5-7B) to 13B (LLaVa-1.5-13B) does not yield a significant boost, which indicates that the ability for multimodal mathematical reasoning cannot be simply acquired from the LLM-side only.

heading: 4 Experiments
text: We conduct experiments to (i) validate the effectiveness of ArXivQA for boosting multimodal scientific reasoning for open-source LVLMs (§4.1) and (ii) benchmark LVLMs capability to comprehend scientific figures with ArXivCap (§4.2).

heading: 4.1 Boosting LVLMs with ArXivQA

heading: 4.1.1 Experimental Settings
text: We adopt Qwen-VL-Chat-7B (Bai et al., 2023b) as the backbone due to its support for interleaved image-text input formats and high-resolution images. We fine-tune it on our ArXivCap (Qwen-VL-Chat-7BArXivCap), ArXivQA (QwenVL-Chat-7BArXivQA) and combination of these two datasets (Qwen-VL-Chat-7BArXivCap + ArXivQA) for three epochs with a learning rate of 1e-5 following the original paper. We combine the answer and the rationale in ArXivQA to form the target output during training. Models are evaluated on MathVista (Lu et al., 2023), a benchmark that requires fine-grained, deep visual understanding and compositional reasoning. MathVista contains 6,141 examples, consisting of five multimodal tasks Figure QA, Geometry Problem Solving, Math word problem, Text Book QA, and Visual QA. We select 478 multiple-choice questions in the testmini split to avoid the inconsistency of answer parsing. We compute the accuracy scores and adopt the provided prediction files for calculating the baseline performance.

heading: 4.1.2 Results
text: As shown in Table 4, fine-tuning on our Multimodal ArXiv, especially on the ArXivQA dataset, consistently boosts the performance, helping the open-sourced Qwen-VL-Chat achieve a comparable overall MathVista reasoning performance. Due
to the wide coverage of the scientific figures, the performance gain mainly comes from significantly improved Geometry Problem Solving, Textbook QA, and Visual QA tasks. For example, after finetuning on the ArXivQA dataset, the accuracy is increased from 19.1% to 34.0% and from 46.7% to 70.0% on Geometry Problem Solving and Textbook QA tasks, respectively. The improvement on Math Word Problem is marginal, where we think the domain-specific data augmentation can be further explored with a curated filtering dataset on our dataset (Gao et al., 2023). On the contrary, the accuracy of Figure QA deteriorates slightly compared with the original backbone model, which we attribute to the fact that most of the plots in the Figure QA evaluation are sampled from synthesized datasets such as DVQA (Kafle et al., 2018), exhibiting great gaps between real-world paper figures.

heading: 4.1.3 Analysis
text: We investigate how different subject domains affect mathematical reasoning ability using pairs of questions and answers (QA). We focus on six domains with more than 5K samples each. From each domain, we randomly choose a subset of 5K samples to ensure fairness in comparison. We then fine-tune the Qwen-VL-Chat base model using QA pairs from each domain and observe how it affects the model’s accuracy compared to its original state. Figure 4 demonstrates the relative accuracy changes (i.e., Accuracy after Fine-tuningOriginal Accuracy − 1) after training the model on QA pairs from each domain. Our findings reveal several key points: (i) QA pairs from the Computer Science (CS) domain are highly effective for improving mathematical reasoning ability, achieving a notable 27.09% relative improvement. We attribute this to the compositional nature of the CS area. (ii) The most beneficial do-
main varies depending on the specific task. For instance, QA pairs from astrophysics domains enhance geometry problem-solving, while those from Condensed Matter improve performance in math word problems. (iii) Most domains hurt the Figure QA task. This suggests that synthetic Figure QA might not be the best benchmark for assessing realistic reasoning ability. These findings underscore the efficacy of generated QA pairs and offer valuable insights for future research, such as adjusting task-specific weights in the dataset accordingly.

heading: 4.2 Benchmarking LVLMs on ArXivCap

heading: 4.2.1 Evaluated Tasks
text: Four vision-to-text tasks to benchmark LVLMs’ ability to comprehend scientific figures.
Single-Figure Captioning Similar to the traditional image captioning setup (Lin et al., 2014), single-figure captioning requires the model to generate a caption for the given single figure. The captions generated by the model are expected to encapsulate the nuanced details within these figures, including numbers and mathematical formulas, presenting a unique challenge for models to identify and articulate these elements accurately. Formally, given an image-caption pair (I, C), the LVLM M is asked to generate the caption given an instruction prompt Ps to hint the goal of scientific captioning:
Ĉ = M(I, Ps),
where Ĉ would be evaluated according to the ground-truth C.
Multiple-Figure Captioning We introduce a more intricate challenge involving applying reasoning across multiple images. This task, termed Multiple-Figure Captioning, necessitates the model to craft a comprehensive summary caption for subfigures. As exemplified in Figure 2, the model is tasked with generating an overarching caption for two or more subfigures, leveraging visual clues to draw comparisons and formulate summary captions. Formally, given a list of figures L = (I1, . . . , In), the model is asked to generate the ground-truth main caption C by considering all the semantics in the figures with a task prompt Pm:
Ĉ = M(L,Pm) = M(I1, . . . , In, Pm).
Contextualized Captioning Inspired by the evolving in-context learning capabilities of LLMs (Brown et al., 2020b; Dong et al., 2022),
we introduce a contextualized captioning task to examine the in-context learning ability of LVLMs. In this task, the model is presented with a set of figure-caption pairs, and its goal is to generate a caption for a given image based on the provided demonstrations. Given a sequential image-captions pairs S = {(Ii, Ci)}ni=1 consisting of n pairs of image Ii and the corresponding Ci, the contextualized image captioning task can be formalized as follows:
Ĉn = M(I1, C1, . . . , In−1, Cn−1, In, Pc).
The model is supposed to leverage the context history to enhance the accuracy and coherence of the generated caption.
Title Generation This task requires a nuanced understanding of figures and captions to distill essential observations into a high-level summary of the presented results for LVLMs. Specifically, instead of producing the captions for the figures, this task requires the model to connect different figures and corresponding captions to infer the paper title. Let S = {(Ii, Ci)}mi=1 be a sequence of m figurecaption pairs in the extracted paper. Note that Ii could be a single figure or a multiple-figure, and we reuse Ii for simplicity here. The title generation asks M to generate the title for the paper given a task prompt Pt:
T̂ = M(I1, C1, . . . , Im, Cm, Pt).
The prediction T̂ is evaluated by comparing it to the original title T .

heading: 4.2.2 Experimental Settings
text: Dataset We divide ArXivCap into training and test sets with a 9:1 ratio for evaluation. The test set includes: 161.3K samples for single-figure captioning, 12.8K samples for multiple-figure captioning, 57.2K samples for contextualized captioning, and 57.2K samples for title generation.
Evaluated Models We select various LVLMs covering different architectures. (1) LVLMs designed for dealing with a single image, BLIP2-OPT-6.7B (Li et al., 2023b) , InstructBLIPVicuna7B (Dai et al., 2023), LLaVA-1.57B/13B (Liu et al., 2023a). Due to the ability limitation, we only benchmark these models on the single image captioning task; (2) LVLMs capable of handling interleaved text-image inputs, such as OpenFlamingo-9B (Alayrac et al., 2022; Awadalla
et al., 2023), IDEFICS-Instruct-9B (Laurençon et al., 2023), Qwen-VL-Chat-7B (Bai et al., 2023b). These models are evaluated on all the tasks we proposed; (3) Proprietary models such as Gemini 1.0 Pro Vision and GPT-4V. Due to the large scale of our test set, we randomly sample a subset consisting of 500 instances for evaluating these two models to reduce costs, with corresponding scores colored in grey. Details of evaluated models and the task prompts used are provided in Appendix B.
Training Settings To investigate whether indomain training can enhance the model’s capabilities, we train the Qwen-VL-Chat-7B on ArXivCap using the same setting as in §4.1.1. To fit the input length limit, we set the maximum number of figures per sample to four. The training process takes 70 hours with 8 NVIDIA A100s.
Metrics BLEU-2 (Papineni et al., 2002), ROUGE-L (Lin, 2004) and BERT-Score (Zhang et al., 2020) are adopted as the automatic evaluation metrics. We also explore using GPT-4 to assist in caption evaluation. Our findings in Appendix B.3 indicate that ROUGE-L and BLEU-2 scores are highly correlated with GPT-4’s annotations. We primarily use these three metrics due to their convenience. A manual error analysis is conducted to supplement the automatic metrics (§4.3).

heading: 4.2.3 Results
text: Results of Single-Figure Captioning The evaluation results for the single-figure captioning task are presented in Table 5. Despite achieving nearperfect performance on conventional image captioning tasks like MSCOCO (Lin et al., 2014), open-source LVLMs, such as LLaVA models, face challenges when applied to academic figures. For closed models, GPT-4V performs comparably with
Gemini 1.0 Pro Vision. Furthermore, continuous training on our dataset yields a significant performance boost for this task. For instance, finetuning results in a notable increase in the BLEU-2 score from 4.4 to 8.9, indicating a promising avenue for enhancing academic figure comprehension through domain-specific training. We also investigate whether providing additional context information, such as the paper title and abstract, could help models generate better figure captions. As shown in Table 6, adding the title is beneficial evidenced by the boosted scores, while providing abstracts brings negligible gains.

heading: Results of Multiple-Figure Captioning As
text: shown in the first block of Table 7, similar to singlefigure captioning, multiple-image captioning poses a challenge for current open-source LVLMs. For instance, Qwen-VL-Chat achieves only a 3.0 BLEU2 and a 7.2 ROUGE-L score on this task, considerably lower than its performance in singlefigure captioning. In contrast, GPT-4V consistently demonstrates proficiency in both tasks, suggesting a balanced ability to capture semantics across multiple images. Notably, training on our ArXivCap dataset yields more pronounced improvements
for this task, culminating in Qwen-VL-Chat even surpassing the performance of the GPT-4V model. This enhancement underscores the pivotal role of our dataset in facilitating LVLMs to enhance reasoning capabilities over multiple images, leading to more effective summarization of scientific figures.
Results of Contextualized Captioning In the middle block of Table 7, we find that IDEFICSInstruct-9B achieves the best performance on this task. This achievement is largely attributed to its remarkable proficiency in leveraging contextual cues, stemming from its extensive pre-training involving interleaved image-text pairs (Laurençon et al., 2023). Interestingly, fine-tuning on ArXivCap results in marginal performance declines across all metrics, with GPT-4V achieving the lowest scores as well. This phenomenon can be attributed to the tendency of sequential captions to exhibit similar patterns, thereby favoring models that effectively leverage contextual cues. We perform two more challenging evaluations by (i) providing context pairs from another paper and (ii) randomly shuffling the order of figure-caption pairs in the context. As shown in Table 8, the performance with random contexts degrades significantly, validating our previous hypothesis. Instead, the fine-tuned model demonstrates more robust captioning results under these settings, evidenced by the slight 8% drop on ROUGE-L compared to the 31% of the original model with shuffled context orders.

heading: Type
text: Results of Title Generation The results are presented in the last block of Table 7. Notably, the title generation task poses a formidable challenge, evident in the significantly lower overall BLEU2 score compared to the captioning tasks. This suggests the inherent difficulty in generating precise predictions for paper titles. A contrasting picture emerges when considering the ROUGE-L and BERT-Score metrics, which either closely align or surpass the performance on captioning tasks. This underscores the model’s proficiency in producing semantic-related results given the presented figures. Consistent with the previous two tasks, fine-tuning the model on our dataset yields substantial enhancements for the title generation task. The BLEU-2 score jumps impressively from 2.6 to 6.7, while the ROUGE-L score sees a commendable increase from 15.8 to 23.5. These findings highlight the challenge of title generation for current LVLMs and the effectiveness of our dataset in improving the model’s capability to generate accurate titles.

heading: 4.3 Analysis
text: Manual Evaluation of Generated Captions We conduct a manual inspection for single-figure captioning results. To ensure a more informed evaluation, we focus on a paper from the CS domain, leveraging our domain knowledge to assess caption quality better. The quality of generated captions
is assessed by scrutinizing the figure, the groundtruth caption, the paper title, and the abstract. We categorize captions into the following quality types according to our preliminary inspection: (1) Acceptable, where captions accurately encapsulate the scientific figure’s essence, aligning with the intended information of the ground-truth; (2) Over Simplification, instances where the model oversimplifies content, offering a broad overview while neglecting specific details and nuances present in the ground truth; (3) Recognition Error, where the model inaccurately recognizes and describes key visual and textual elements in the scientific figure, such as colors, numerical values, or textual context; and (4) Contextual Misinterpretation, where the model misinterprets the specific context of the scientific figure, resulting in captions relevant in a generic sense but inaccurate for the given figure. Visualized generated captions of different types are shown in Figure 14 of Appendix C.1. The results of 100 manually examined captions are depicted in Figure 5, revealing that only 16% of captions are deemed acceptable when compared to humanwritten ones. Among unsatisfactory captions, contextual misinterpretation emerges as the dominant issue, suggesting a need for incorporating more contextual information as suggested in Table 6. Oversimplification is another concern, with generic captions identified. Additionally, 23% and 19% of examined captions suffer from the oversimplification issue and recognition errors in reported numbers/texts in the caption, respectively. The former is attributed to the highly frequent simple caption in the training dataset and the latter issue could be
addressed through potential integration with OCR results. Our manual evaluation suggests future efforts may benefit from incorporating additional context clues, such as paper metadata, improving the model’s fundamental perception abilities, and utilizing external information.
Case Study of MathVista We conduct case studies to illuminate the tuning effects facilitated by our ArXivQA dataset. In the left segment of Figure 6, ArXivQA helps the model accurately answer a question related to the presented bar plot. The right part in Figure 6 demonstrates that ArXivQA can enhance algebraic reasoning abilities. Here, a question involving the derivative of a function is correctly answered, accompanied by a lucid reasoning rationale. Figure 15 in Appendix C.2 highlights a challenging geometry problem where both models generate hallucinated outputs. These illustrative cases collectively affirm the efficacy of our dataset.

heading: 5 Conclusion
text: Our work introduces Multimodal ArXiv, comprising ArXivCap and ArXivQA, aims at advancing the scientific comprehension of LVLMs. Experiments show that fine-tuning on ArXivQA notably enhances LVLMs’ mathematical reasoning capabilities. Moreover, our comprehensive evaluations across four vision-to-text tasks on ArXivCap underscore the challenges in understanding scientific figures for LVLMs, while highlighting the substantial improvements achieved by in-domain training. Our error analysis offers valuable insights for the ongoing development of LVLMs.

heading: Limitations
text: Our study has several limitations worth noting. Firstly, our exploration may not encompass the full spectrum of LVLMs due to the rapid evolution of architectures and training methodologies such as parameter-efficient tuning (Hu et al., 2022; Ma et al., 2024). Nevertheless, we believe our dataset could still be effective for other LVLMs and the findings are generalizable. We show that our ArXivQA dataset could also boost LLaVA-series models across scientific understanding benchmarks in Appendix D. Secondly, our Multimodal ArXiv dataset sources from ArXiv papers due to their accessibility and open-source licenses. This approach may overlook the diversity of disciplines and data modalities present in the broader scientific literature. Future research could incorporate a broader range of datasets and domains to enrich the coverage of scientific knowledge, and explore dynamic data selection methods to improve performance and sample efficiency (Li et al., 2021; Chen et al., 2024).

heading: Acknowledgements
text: We would like to thank all the anonymous reviewers for their insightful comments and suggestions, which helped us improve the quality and clarity of this work. We are particularly grateful to Shuhuai Ren for his valuable feedback in preparing the manuscript. This research was supported in part by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N HKU714/21.

heading: A Details of Multimodal ArXiv

heading: A.1 Caption Cleaning
text: We apply a Python tool to clean the original caption and Table 9 illustrates the caption before and after cleaning.
A.2 Illustration Cases of Multimodal ArXiv
We provide illustrated cases from our dataset for a better understanding. Figure 7 demonstrates a typical single figure-caption pair, and Figure 8 shows the multiple figure-caption case. Figure 10, 11, 12 and 13 illustrate the cases from our ArXivQA dataset, covering different figure types and containing diverse questions.

heading: A.3 Caption Word Cloud
text: We visualize the word cloud of captions in our ArXivCap dataset in Figure 9. It can be seen that the captions have a diverse vocabulary for describing the different figures in the academic papers.

heading: A.4 ArXivQA Prompting Template
text: The prompt used to query GPT-4V is provided in Table 11.

heading: A.5 Quality Analysis of ArXivQA
text: To evaluate the quality of ArXivQA, we manually assess it from six different aspects. We develop
a quality examination guideline for annotators, as shown in Table 12, which addresses various aspects of the QA pairs. We sample 100 examples and ask four authors to conduct the quality analysis. The four authors are divided into two groups, with each group tasks with evaluating 50 examples across six sub-aspects, according to the grading protocol.
The evaluation results are presented in Table 13. We find that most samples feature clear, highquality images with clear and high-quality images, with unambiguous question and option descriptions. However, a small fraction of the generated questions may be unanswerable due to mis-recognizing elements in the figures, as reflected by lower factual alignment scores. Additionally, we consider samples with an aggregate score of 5 or higher from both annotators to be of sufficient quality. Under this stringent criterion, 79 out of 100 samples meet the threshold, demonstrating that the dataset’s quality is generally satisfactory.

heading: B Evaluation Details

heading: B.1 Details of Evaluated Models
text: BLIP2 (Li et al., 2023b), introduces a lightweight Q-Former designed to bridge modality gaps and leverages frozen LLMs. Leveraging LLMs, BLIP-2 can conduct zero-shot image-to-text generation using natural language prompts. We select the BLIP2-OPT-6.7B version for evaluation.3
InstructBLIP (Dai et al., 2023) employs an instruction-aware visual feature extraction module based on BLIP2 (Li et al., 2023b) and is trained with unified multimodal instruction tuning datasets.
3https://huggingface.co/Salesforce/ blip2-opt-6.7b
Sub-Figure 1:
Sub-Caption 1: Performance of the proposed low-complexity channel assignment scheme based on matching theory.
Sub-Figure 2:
Sub-Caption 2: Performance of proposed optimal power allocation scheme for users assigned to the same channel.
We choose InstructBLIP-Vicuna-7B for evaluation.4.
LLaVA (Liu et al., 2023b), adopts Vicuna models as the backbone LLM and is trained on the ChatGPT/GPT-4 generated instruction tun-
4https://huggingface.co/Salesforce/ instructblip-vicuna-7b
Question: Which of the following best describes the 'target link' in the context of this figure? Options: A. A link that connects nodes with the highest values B. The link under examination for feature construction C. A predefined link that guides the construction of other
links D. The most frequently occurring link in the network
Label: B Rationale: The 'target link' is shown with a 'link-value' p, separated from the 'node-value' L, suggesting it is the specific link being analyzed for its features.
Figure 12: A case from our ArXivQA dataset.
Question: Which steps in the workflow are directly involved in preparing the data for differential expression analysis? Options: A. Steps 1-2 B. Steps 7-12 C. Steps 13-14 D. Step 15 Label: C Rationale: The steps directly involved in preparing the data for differential expression analysis include "Feature counting" to get a count table and the subsequent data structuring and normalization processes listed under "Step 14". These are the steps that directly precede the actual differential expression analysis methods like "2-group differential comparison" and "GLM-based differential comparisons".
Figure 13: A case from our ArXivQA dataset.
ing dataset. LLaVA-v1.5 (Liu et al., 2023a) improves on LLaVA models by employing curated task datasets and an enhanced modality alignment module. We evaluate both LLaVA-v1.5-7B5 and
5https://huggingface.co/liuhaotian/llava-v1. 5-7b
LLaVA-v1.5-13B.6
Flamingo (Alayrac et al., 2022) pioneers the development of LVLMs by introducing a cross-
6https://huggingface.co/liuhaotian/llava-v1. 5-13b
gated layer for LLMs to produce visual-grounded text. The training dataset consists of interleaved visual data and text from the web pages, enabling it to generate free-form text as the output. We select the open-source implementation OpenFlamingo-9B (Awadalla et al., 2023) for evaluation.7
IDEFICS is another open-sourced implementation of Flamingo (Alayrac et al., 2022). Trained on publicly available image-text alignment pairs and instruction tuning datasets, it demonstrates comparable results with the original closed-source model on various image-text benchmarks. We select the IDEFICS-Instruct-9B for evaluation.8.
Qwen-VL-Chat (Bai et al., 2023b) is a bilingual LVLM that supports both English and Chinese built on the Qwen LLM (Bai et al., 2023a). During the training phase, Qwen-VL-Chat adopts a packing strategy to create multiple images as inputs, improving its ability to understand the vision context. We select the Qwen-VL-Chat-7B for evaluation.9
GPT-4V (OpenAI, 2023), the proprietary visionlanguage models developed by OpenAI, which are shown to be powerful on various multi-modal tasks (Yang et al., 2023a). The API version we queried is gpt-4-vision-preview.
Bard (Google, 2023), a commercial LVLM developed by Google. We utilize the unofficial API10 querying the model with our task prompts, accessed on 2023-11-17.
Gemini 1.0 Pro Vision (Reid et al., 2024), a upgraded LVLM by Google. We utilize the official API querying the model with our task prompts, accessed on 2024-05-20.

heading: B.2 Task Prompts
text: We evaluate all the models with the same task prompts in our experiments, and the prompts for our four tasks are listed below: Single-Figure Captioning: Create a caption for the provided figure. Multiple-Figure Captioning Create a caption for the provided figures.
7https://huggingface.co/openflamingo/ OpenFlamingo-9B-vitl-mpt7b
8https://huggingface.co/HuggingFaceM4/ idefics-9b-instruct
9https://github.com/QwenLM/Qwen-VL 10https://github.com/dsdanielpark/Bard-API
Contextualized Captioning: We reuse the prompts in previous captioning tasks depending on the current figure type. Title Generation: According to the figures and captions, generate a title for this paper. Title:

heading: B.3 GPT-4 Evaluation of Caption
text: In addition to BLEU-2, ROUGE-L, and BERT-S, we also utilize GPT-4 to evaluate a sample of 500 generated captions. Specifically, we employ GPT4 for the evaluation of single-figure caption tasks following FairEval (Wang et al., 2023). The template for prompting GPT-4 to evaluate generated captions is presented in Table 14. GPT-4 is asked to perform an analysis and then produces a quality score, which is subsequently mapped to a scale from 1 to 5. The results are presented in Table 15. We observe that the ROUGE-L metric exhibits the highest correlation with the GPT-4 Score (Pearson r = 0.91), followed by BLEU-2 (Pearson r = 0.64). BERT-S instead demonstrates a moderate correlation (Pearson r = 0.39). The uniformly low GPT-4 scores across all models suggest that they struggle to produce satisfactory captions, which is consistent with the findings in our main paper. Notably, training on ArXivCap results in a significant 12% improvement in the GPT-4 score compared to the original Qwen-VL-Chat model, leading to the most favorable outcomes in this evaluation.

heading: C Error Analysis

heading: C.1 Caption Type Illustration
text: We illustrate captions of four types in our main paper in Figure 14. The Acceptable caption provides a comprehensive description of the figure presented. The oversimplified caption is too short compared with the original human-written caption. Furthermore, as shown in the third block in Figure 14, Contextual Misinterpretation refers to captions with unmentioned content in the figure, such as the dataset colored in red. Recognition Error denotes the model wrongly identified the number or text in the figure, such as the misidentified model name in the last block of Figure 14.

heading: C.2 Failure Sample of MathVista
text: Figure 15 shows a challenging geometry mathematic reasoning problem where both models fail to produce the correct answer. Echoing the quantitative results in our main paper, we believe future
studies can incorporate more focused corpus for enhancing the geometry and mathematical reasoning ability of LVLMs.

heading: D Results with LLaVA Backbone
text: We investigate whether ArXivQA could also enhance other LVLMs, such as LLaVA models (Liu et al., 2023b). To maintain model performance, we mix our ArXivQA dataset with the LLaVA SFT 665K-instruction tuning dataset. The LLaVA-v1.57B is adopted as the backbone and the model is trained following the original recipe. The results on various benchmarks are listed in Table 16. We find that not only the scientific reasoning performance is improved on multimodal reasoning tasks (MathVista (Lu et al., 2023), MMMU (Yue et al.,
2023), and ScienceQA (Lu et al., 2022)), but the overall capability on MM-Vet (Yu et al., 2024) is also boosted. Together with our results using QwenVL-Chat, these findings indicate that our ArXivQA dataset can enhance different model backbones and is beneficial across various benchmarks.

heading: Future Work
text: Echoing the quantitative results in our main paper, we believe future
studies can incorporate more focused corpus for enhancing the geometry and mathematical reasoning ability of LVLMs