LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content
Authors: Nimrod Shabtay
Felipe Maia Polo
Sivan Doveh
Wei Lin
Muhammad Jehanzeb Mirza
Leshem Choshen
Mikhail Yurochkin
Yuekai Sun
Assaf Arbelle
Leonid Karlinsky
Raja Giryes
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Modal Dataset
Visual Question-Answering
Efficient Evaluation
Primary area: datasets and benchmarks
TL;DR: Automatic and evolving Multi-Modal dataset for VQA with an efficient evaluation method based on arxiv papers

Abstract: The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models’ true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code is available here.

Published as a conference paper at ICLR 2025
LIVEXIV A MULTI-MODAL LIVE BENCHMARK BASED ON ARXIV PAPERS CONTENT
Nimrod Shabtay' .2, Felipe Maia Polo?, Sivan Doveh", Wei Lin', M. Jehanzeb Mirza,, Leshem Choshen
Mikhail Yurochkin', Yuekai Sun Assaf Arbelle?, Leonid Karlinsky", Raja Giryes'
Faculty of Engineering Tel-Aviv University, IBM Research,
3 Department of Statistics, University of Michigan, USA JKU Linz, Austria, 5 MIT CSAIL. MIT-IBM
ABSTRACT
The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain- specific manuscripts any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in in in-the-loop. using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, introduce efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models' true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance deviation is indeed minimal ( (<2.%)) Our dataset is available online on HuggingFace and our code is available on GitHub.
INTRODUCTION
Test Set Contamination Over Time
Saturated Benehmarks
T2
100
Tran
Test Dao
Dass
11
12
13 24
Figure I: Static benchmark contamination. As training data increases, the risk for test set contamination grows and static benchmarks becomes saturated, reflecting falsely improved capabilities.
The internet, with its vast and ever-growing repository of information, serves as a rich data source for training Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Chiang et al., 2023; Raffel et al., 2019; Touvron et al., 2023a;b; Dubey et al., 2024) and Large Multi-modal Models (LMMs) (OpenAI, 2023; Liu al., 2023e; Li et al., 2024e; Zhu et al., 2023; Chen al., 2023a; Alayrac al., 2022; Radford et al., 2021a). This diverse and continuously updated data fits precisely the need to cover varying knowledge in scale in the training data.
Training on such data enables the models to achieve superhuman performance across a wide range of tasks on multiple common benchmarks (Fu et al., 2023; Yue et al., 2024; Li et al., 2024d; Liu et al., 2023d).

---

Published as a conference paper at ICLR 2025
All Models
2000
100
Fosteaton soal andeeh
Fow modeh for cevertiiatioo
10202030
18.24
2007
Creacing New
Efficient Eveluationn
Ever Evollingg Benelhate
Uplate
Figure 2: We propose LiveXiv, a new method for generating Live Question-Answerrin based on ArXiv content. Our pipeline automatically generates scalable and reliable questions along with an efficient evaluation method to reduce the computational and logistic overheads required for continually evaluating past and present models on new versions of the dataset.
dataset for Visual
We hypothesize that a portion of LLMs' reported improvements are due to data contamination (Figure 1) and pose the following question: To what extent does the potential for test set contaminatiio during large-scale training affect our perception of the abilities of L.Ms??
One possible way to safeguard against the contamination of static benchmarks is to design a live benchmark that can continuously harness data from the web and turn it into an ever-evolving benchmark to test the abilities of these models. A live benchmark may be used in one of the following ways: (a) Expand the dataset over time and evaluate the models' overall knowledge over all collected data, while taking into account that the data might be contaminated. (b) Use only the latest version to assess model capabilities while keeping data contamination risk minimal. While we focus on (b), we share key properties of our efficient evaluation method can be applicable to both cases.
Although, live benchmark is a promising direction, it still comes with its fair share of challenges. A live benchmark should ideally be updated frequently, consistently, and automatically, ie. it should be able scrape the data from the web and formulate into benchmark for automated evaluations. Furthermore, as the benchmark is ever-evolving, each time a new version ariies, all the participant models need to be re-evaluated, making the update procedure prohibitively expensive both in time and compute. This requires methodology for efficient evaluation of these models on a continuously updating benchmark. Such a methodology should ease the computational burden of evaluating all the models on each new version of the dataset and reduce the logistic overhead of maintaining inaccessible old models.
In this work, we take a step in this direction and propose LiveXiv - a novel fully automated multi-moda live benchmark that focuses on scientific domains. LiveXiv starts with scraping category-speciffe (e.g., CV.CV, eess.SY, q-bio.BM, etc.) manuscripts from ArXiv and generates visual question answers from figures, charts, and tables present in these manuscripts through a capable multi-modal model, namely, GPT-4o. As is challenging to directly feed information-is PDF documents to GPT-4o, as pre-processing step, we extract relevant information from the papers by processing it with structured document parsing pipeline (Team, 2022) to obtain pertinent information like placements of figures, charts, tables, and the text in the captions or in the tables.
This information is used to extract, egg. by cropping, relevant information from the manuscripts, which is fed to GPT-4o to generate visual questions and answers. Although very capable, GPT-4o is still prone to errors, e.g., due to hallucinations, and may even generate questions that can be answered without visual information. Thus, to mitigate these issues, we add an extensive filtering stage that automatically filters questions requiring only textual information to answer them, and reduce hallucinations through obtaining agreement about the generated questions with another capable multi modil model, namely, Claude- Stnnne. After the extensive filtering, we obtain a large corpora of VQA pairs which are incorporated into our LiveXiv live benchmark.
Over time, the benchmark is expected to grow, either in the size of the dataset or the amount of models to be evaluated, which increases the required resources for evaluation. Moreover, comparing new model to existing models at different times requires re-evaluating the existing models over the latest version of the dataset, which can cause additional overhead for continuous evaluation and comparison to prior works. To make the evaluations on LiveXiv feasible, we take inspiration from Maia Polo et al. (2024a;b) and propose a method to approximate the performance of the existing models in new versions of LiveXiv just by re- evaluating small portion of them. Figure provides a conceptualized overview of our approach.
We summarize our contributions as follows: (a) We propose a scalable live benchmark without any human in the loop that automatically hamesses data from online scientific manuscripts, generates multiple
2

---

Published as a conference paper at ICLR 2025
VQA pairs, filters these questions reduce emrrr,, and formulates them in the form of a benchmark to test the evolving landscape of LMMs; (b) We introduce an efficient evaluation pipeline that requires LMMs to be tested only on a fraction of the data to infer its performance on the latest version of the benchmark, reducing the overall needed evaluations by at least R 75%; (c) We benchmark multiple open and proprietary LMMs on the first version of our benchmark highlighting its challenging nature and providing interesting insights about the models" behavior when evaluated on less contaminated data.
2 RELATED WORKS
Large multi-modal Models (LMMs). LMMs have shown significant advancements in enabling billion- parameter scale LLMs to perform multi-modal tasks such as image captioning, visual reasoning, and visual question answering, Academia and industry have endeavored to develop LMMs targeting the multi-modal competence of advanced proprietary models like GPT-44 (OpenAI, 2023) and Claude (cla, 2024). Instruct- BLIP performs instruction tuning on the pre-trained BLIP-2 (Li et al., 2023) covering 11 Lasoon-la tasks. The LA series models (Liu al., 2023c;;;;; Li et al., 2024b) develop the pipeline of collec- tion of instruction-following data and visual instruction tuning with enhanced vision capabilities. The IntemLM-XCooppsee
XComposer (IXC) series (Dong et al., 2024a;b) target free-form vision- saiionlanguage composition and multilingual comprehension. Models from Idefics release (Laurençon et al., 2024b;a) benefit from the mas- sive collection of cstruction-following data from over 50 vision-language databases, enhancing capabilities of OCR, document understanding, and visual reasoning. In this work, we include 17 top-performing LMMs in our multi-modal live benchmark LiveXiv, covering both open-sourced and proprietary representatives.
Static evaluation benchmarks for LMMs. Most existing LMM benchmarks offer static evaluation with fixed questions and answers (Fu et al., 2023; Yue et al., 2024; Li et al., 2024d; Liu et al., 2023d; Huang et al., 2024; Lin et al., 2024; Zhang et al., 2024b; Onoe et al., 2024; Yanuka et al., 2025). Ben-Kish et al. (2024) use generative models to extend such benchmarks to open settings. MME (Fu et al., 2023) offers evaluation of perception and cogmition on 14 tasks and MMMU (Yue et al., 2024) includes 11.5K questions from college exams, quizzes and text books from six major disciplines. Although these benchmarks cover large variety of multi-modal domain knowledge, evaluation on them is faced with two hazards: the excessive evaluation cost and test data contamination. In this work, we tackle both challenges by proposing a suite that enables efficient evaluation on a contamination-free live benchmark.
Contamination-Free benchmarks. As large foundation models like LLMs and LMMs are trained on combined sources of tremendous amount of web data repurposed version of existing open-sourced datasets, there is high risk of overlap between training data and samples from evaluation benchmarks. Reported evidence and analysis show impact of data contamination on evaluation benchmarks for LLMs (Wei et al., 2023; Zhang et al., 2024a; Cobbe et al., 2021; Roberts et al., 2023; Jain et al., 2024) and LMMs (Chen et al., 2024), indicating the significance of contamination- free evaluation benchmarks. Recent works targeted LLMs are discussed in A.
For LMMs, Vibe-Eval (Padlewski et al., 2024) and -A-iddrr (Li et al., 2024a) perform contamination check on the collected samples that reflect real-world user requests. Most related to our work, the LMMs-Eval LiveBench (Zhang et al., 2024b) collects images from sources of new websites and online forums and employs proprictary LMMs for design and revision of questions. However, the LMMs-Eval LiveBench requires human manual verification of questions which impedes the scalability. Furthermore, it contains only open-ended questions that require LMM-as--- aa--ud-- which is time-consuming. susceptible to judge biases, and difficult to scale. In comparison, our LiveXiv constructs a fully-automated data collection pipeline which generates multiple-choice questions which are challenging to the top-performing LMMs.
Efficient benchmarks. With the increasing amount of tasks and samples in current benchmarks, evaluation of the full suite is time-consuming and cost-intensive. Efforts underway develop efficient benchmarks that reduce computation costs without sacrificing reliability. In A.l at the appendix we cover the works related efficient benchmarks.
3 LIVEXIV
At higher level, our automated LiveXiv is created by first obtaining the domain-specific scientific manuscripts from ArXiv at any given timestamp. Then, to obtain pertinent information from the manuscripts, pass them through structured document parsing pipeline and then generate visual
3

---

Published as a conference paper at ICLR 2025
Final Benchamrk
Text+
+ Caption
Meta Prompi
Filtering By LIM-IMM
BBox
Q
Extract
Q&A
A:
Agreenment
A:
Generation
Figure 3: Our live dataset generation consists of several stages. We first extract the images and their corresponding metadata (i.e, captions and table contents), then we classify the figures into categories using meta- prompppon.. All the extracted data is then fed GPT40 to generate multiple peesstions-nswwe pairs per image. Since generative models are prone to errors, we apply several filtering steps, using an LLM and LMM to ensure that our dataset is truly multi-modal and reliable.
question answers through a capable LMM (Section 3.1). However, the generated questions can contain errors due to hallucinations or might be too straightforward to answer. Thus, to mitigate these issues, we offer an extensive filtering stage (Section 3.2). To evaluate the benchmark, we propose an efficient evaluation framework to infer the overall performance on the benchmark using only small subset of evaluations, making the evaluations extremely resource-efficient (Section 3.3). The data acquisition and filtering steps schematically visualized in Figure 3.
3.1 DATA ACQUISITION AND VQA GENERATION
We start with the data acquisition phase, then pre pre-process the data to obtain the required metadata (e.g,, placements of figures, captions, etc.), and then generate the first iteration of VQA from the multi-modal data (figures and tables) from the manuscripts.
Data Acquisition: At any given timestamp, begin by acquiring only ArXiv papers which have non-exclusive license to distribute from predefined domains such as Computer Science (cs. AI, cs. CV), Electrical Engineering (eess.. SP, eess. SY), and Quantitative Biology (q (q-bio. BM, q-bio. GN). However, these manuscripts contain a lot of information that might not be necessary for the task of VQA data generation. Thus, extract pertinent information we require pre-processing step.
Pre-processing:
The downloaded PDFs undergo structured document parsing pipeline using the DeepSearch toolkit (Team, 2022), which extracts a comprehensive layout of each document, including the positions of figures, tables, captions, and other elements. This structured layout forms the basis for extracting the multi-modal data required for subsequent tasks. To enrich the dataset with additional metadata not captured by the parsing pipeline, we employ a meta-prompting approach with CLIP (Radford et al., 2021b), similar to the method used by Mirza al. (2024). Specifically, we classify the figures into three distinct categories: Block Diagram, Chart, and Qualitative visual examples which facilitates more granular, domain-specific evaluation of LMM performance.
VQA Generation: For Visual Question Answering (VQA), construct pairs of figures and their corresponding captions, and for generating VQA from the data present in the tables, we obtain (e.g., crop) images of tables accompanied by their corresponding data.
The VQA process involves two steps using GPT-4o. First, we input the figure and its caption to GPT-4o generate detailed description of the figure, employing a Chain-of-Though (CoT) approach (Wei et al., 2022). Next, the detailed description and figure are fed back into GPT-4o, with prompts adapted from 'ConMe' (Huang et al., 2024) to suit our scientific use case, enabling the generation of relevant VQA questions. For questions from the tables, we utilize the table's content directly, presenting both the image of the table and its data in markdown format to GPT-4o produce questions that require common-sense reasoning and data manipulation. The automated nature of this process ensures a robust and comprehensive evaluation framework for LMMs, tailored scientific literature specifics. Detailed prompt templates can be found in Appendix A.6.
3.2 FILTERING PHASE
Even though GPT-4o is powerful and has been reported to outperform humans on many different benchmarks (OpenAI, 2023), still it is prone emrrs and sometimes can result in VQA pairs that are answerable without requiring the visual information. Thus, to ensure that the benchmark remains competitive and also has minimum emrrs,, we propose extensive automatic filtering step. At a higher
4

---

Published as a conference paper at ICLR 2025
level, the filtering phase consists of two main parts, each designed to mitigate a separate issue that can arise due to the automatic dataset generation.
Blind test with an LLM: To ensure that the generated VQA pairs trudy multi-modal, pass them through a Large Language Model (LLM), namely LLama-3, 3-7BB (Meta, 2024), without providing any associated images image descriptions. This process, referred to as a blind test, aims to identify questions that the LLM can answer correctly even in the absence of visual context, indicating they are not truly multi-modal. To ensure robustness, this blind evaluation repeated multiple times to eliminate any potential lucky guesses by the LLM. Questions that are consistently answered correctly by the LLM are filtered out, resulting in the removal of approximately 30% of the generated questions. This step ensures that the remaining questions in the dataset are inherently muuti--odal and cannot be answered solely based on linguistic context. The filtered dataset thus represents more challenging benchmark for evaluating multi-modal capabilities of vision- laiiunlangguge models.
Agreement between disjoint models: Generative models, including LMMs, are prone to hallucination, where the model generates incorrect not grounded information. In our case, these hallucinations can lead to erroneous VQA pairs. To address this issue, we introduce an additional filtering step. All questions that pass the initial "blind test" are reviewed along with their generated answers by a different LMM, in this case Staude-Sonnee (cla, 2024), which is provided with the image, question, and the ground truth answer which were all generated by GPT-4o. This second model is asked to either agree or disagree with the generated answer, considering the visual context.
We point out that agreement between models is nuanced process; incorporating more models to validate answers may lead to the exclusion of difficult questions, thereby diluting the difficulty of the dataset. Therefore, we limit this validation step to models with comparable performance the generation model (i.e. GPT-4o). Our preliminary manual evaluation on a subset of the dataset indicates that this agreement step significantly reduces the proportion of incorrect ground-truth (GT) questions, with reduction of 38.5%, while minimally impacting the retention of high-qualln question-G" GT pairs, with only a 6.15% removal of valid pairs. This refinement ensures that the final dataset both challenging and accurate for the evaluation of LMMs' multi-modal reasoning capabilities. The generated corpus of data is ready to be incorporated into LiveXiy and can be updated automatically without any human intervention.
3.3 EFFICIENT EVALUATION
Since LiveXiv is dynamic benchmark, evaluation can be costly: ideally, whenever a new version of the benchmark is released, all models must be re-evaluated on the updated data, which can pose an engineering challenge and become computationally expensive when handling dozens of models. In this section, we describe our approach to efficient evaluation, which avoids re- re-evaluating all models at each step, making LiveXiv's maintenance economically feasible. Our idea is based on Item Response Theory (IRT) (Cai et al., 2016; Van der Linden, 2018; Lord et al., 1968; Maia Polo et al., 2024b;a). a collection of statistical models traditionally used in psychometrics and educational assessment. We briefly give some background on IRT and detail how we use it for our evaluations.
3.3.1 ITEM RESPONSE THEORY (IRT)
We use the IRT model to predict the probability of a certain LMM i answering correctly on a sample (question) j. In mathematical terms, let YúJ E (0.1) denote the correctness on sample when responded by LMM i:
Yi ~ (e
where 6 is an LMM-speiffic parameter, 33 is a sample-specifi parameter, and μ is a function that maps those parameters to the probability of correctness. In this work, follow Maia Polo et al. (2024b) and assume the parameters live in the real line while y induces a logistic regression model. In more detail, we assume
P(Y,,=1:8,,8))) 1+exp|-(01 3,)-
(1)
Here, 6 can be interpreted the skill level of LMM i while B, is the hardness of sample j. By equation 1, if 6; is much greater (resp. smaller) than 3y. then the probability P(Y,,= = -1,0,,8,) will be close to one (resp. zero). This version of the IRT model is known the Rasch model (Georg, 1960; Chen et al.,
5

---

Published as a conference paper at ICLR 2025
2023b), and it is widely used in fields such as recommendation systems (Starke al., 2017), educational testing (Clements et al., 2008), and evaluation of language models (Maia Polo et al., 2024b). Moreover, it has similar formulation the popular Bradley-Tery model (Bradley & Temry, 1952) used in Chatbot Arena (Chiang et al., 2024), popular and dynamic benchmark for Al-powered chatbots. We fit the Rasch model using maximum likelihood estimation as in Chen et al. (2023b) and Maia Polo et al. (2024b).
3.3.2 EFFICIENT EVALUATION WITH IRT
We can estimate old model scores on new data without reevaluating those models. Let I, and Te represent sets of non-negative integers corresponding to LMMs and samples at time 020. It is usually the case that I. II since the set of available models does not shrink over time, except in cases of deprecation, and To, 0J 0 for 117 12 because samples are not repeated across different time steps. Let the set of evaluated models at time be denoted by 4. For t> 0, we assume that I \I+1 is proper subset of Iz, meaning that all newly introduced models are evaluated on the new batch of samples along with some previously existing models. At t 1t=00 0, we assume that Z, = I, meaning all models are evaluated on all samples. Furthermore, we assume that [ is much smaller than [Z]] when t >> 0 so computing power and evaluation time can be saved.
Our goal at time />> O is to estimate the performance of model & I. on the set of samples Jt, using only the correctness scores D,- (Ya: (i.j) E ER,,, where Ω - Uy x Ju. Specifically, we aim to approximate S₁ = [3. E. Yij by estimating its expectation
B[Sa]=
Σ 1(0,=1100,3))). JET.
JT
For a moment, let us assume that 22 is known. Using D,, we can estimate the skill parameters 0,'s of all models in Z, and the difficulty parameters 83 's of all samples in Uer: we denote these estimates as 0, is's and B ss Finally, we obtain an approximation for equation 2, B[Sar]. by substituting Og and Bj by their estimates. The estimator [[S is known as the Performance- IRT estimator (Maia Polo et al., 2024b;a).
Now, we provide method to obtain Qe assuming 94-1 is given; in summary, need to decide which models in Iạ-1 going to be in I,. Our approach to choosing which models are going to be re- re-evaluated is inspired by the concept of optimal design of tests (Van der Linden, 2017, Chapter 9) but in which we choose LMMs instead of samples. First, we set budget my, representing the maximum number of models to be re-evaluated at time step f. Second, assuming that the level of difficulty of the new samples Je is not very different from the ones in Je-1, we choose set of m representative samples in J.-1 by ordering 3, 's and choosing equally spaced samples, based on their quantiles, from the 5th to the 95th percentiles; this will give us questions with a variety of difficulties, excluding outliers. For example, if m, = 3 we would choose questions with difficulties in the 5th, 50th, and 95th percentiles. Denote the chosen core set of samples as (-o.-- and, for each one of these samples js. we choose model in Ię-1 such that the following Fisher information criterion
Es (1) 1P (00-PYy, Yis. - =10 (1.II(Ys--L. -100,00))))
maximized. The model that maximizes 132, maximally informative about the parameter of sample Js and, consequently, about all samples with similar difficulty levels in the new version of LiveXiv; this will help us estimate the difficulties of new samples. We note that some models in I₃-1 might not be available at step I,, e.g., due to deprecation; when choosing models, we do not consider them, but note that we can still estimate their performance on the new batches of data. Moreover, the model selection procedure can also take convenience into account; for example, if two models have very similar Fisher information, we opt for the one that is cheaper to evaluate.
Our experiments demonstrate that re-evaluating just 5 (or even 3) models per step provides accurate model evaluation. With 50 total models, this approach can reduce computing costs by at least x 110, particularly when selecting less expensive models without significantly impacting F In this paper, use a maximum of 19 models; thus, re-evaluating to 5 models represents a saving of zs 75% to 85% in the number of evaluated models.

---

Published as a conference paper at ICLR 2025
4 RESULTS& & ANALYSIS
This section presents the results obtained on the first version of LiveXiv. First, we start by describing the ex- perimental settings. Then, we present the results and finally conclude with detailed analysis of our dataset.
4.1 EXPERIMENTAL SETTINGS
After the generation of the question-answer pairs from our automated pipeline explained in Section 3, we transform the benchmark to multiple-choice questions. We resort to the "generate" inference employed extensively by previous works, such as Li al. (2024d); Huang et al. (2024); Liu et al. (2023d). The model is prompted to choose the letter corresponding to the correct choice and answer with the letter directly. The output letter is then compared with the ground truth and the accuracy is measured. For ease of assimilation and to obtain insights into what type of data the models flourish at, we provide the results from data generated on tables and figures separately. The data generated from figures is labeled as part of Visual Question and Answers (VQA) and the data from the tables is labeled as Table Question and Answers (TQA). Examples for the multiple-choice formulation of the question-answer pairs are added to the Appendix Section A.3.
Evaluation Protocol:
Size of dataset: The first version of our LiveXiv consists of 7328 questions on figures, and 9000 questions on tables, both are generated from 250 papers (25 papers from 10 domains). Overall our first version of the dataset has 16328 questions in total. Thanks to the continual growth in the number of publications in our target domains and the fully automatic nature of our proposed LiveXiv pipeline for benchmark data generation, we will grow LiveXiv by adding an equal-sized large amount of new VQA & TQA data (around 7K VQA and 9K TQA) on a monthly basis.
We extensively evaluate our benchmark by employing a total of 17 LMMs. Specifically, we employ 5 models from the LLaVA family of models including LLaVA 1.5-7B and LLaVA 1.5-13B (Liu et al., 2023c), LLaVA-1.6-77 and LLaVA 1.6-34B (Liu et al., 2023b), and LLaVA Ene-Vision (Li et al., 2024b). Furthermore, we employ IntstructBLIP (Dai et al., 2023). IntemVL2-22B and IntemVL2-88 (Chen et al., 2023c), IntemL.MXccoppose222 Xcomposer2-4KHD (Dong et al., 2024b), DniemLM-Xcooposeee et al., 2023c), Mantis (Jiang et al., 2024), Phi3v (Abdin et al., 2024). Idefics2 (I (Laurençon et al., 2024h) and Idefics3 (Laurençon et al., 2024a), Qwen2-VL. (Wang et al., 2024) and API models Claude-Sonne (cla, 2024) and GPT-4o (OpenAI, 2023) for our evaluations. These models have been chosen because of their varying characteristics and strong performance (except GPT-4o and (loude-Sonney are accessed from the huggingface API, which makes our framework modular for extension to more models as they are being added to the hub in the future.
Models:
(Chen
multiple current benchmarks. All the models
Additional LiveXiv versions: While this section mainly focuses on the analysis of the first version of LiveXiv, new versions continuously being uploaded to the HuggingFace Hub, the time of writing, four additional versions exist: one of past ArXiv papers (vO) and three of more recent papers (v2 - v4). Version 2 consists of 18K samples, introduces 4 new domains from physics (namely, physics.optics, physics.bio-phe anysics.app-ple physics.date data-an), and includes two additional LMMs (Pixtral and Molmo-7B). Version 3 utilizes multiple generation and filtering models (GPT-4o, Claude-Sonnet, Qwen2-VL-72B) and increase diversity and measure the effect of potential biases. In Version 4, we restrict diversity to two models, GPT-4o and Claude-Sonnet, but maintain variability by having each model take on different roles each time. In addition, to perform a deeper analysis, we created variant of version 1 with opposite roles (i.e., Claude as the QA generation model and GPT as the filter model). We found that the question and visual content diversity remained roughly the same which kept the model performance and ranking unaffected. Lastly, version 0 is generated from papers from more than 10 years ago. Our experiments reassure our findings about the performance of our efficient evaluation, strengthening its validity even in the presence of large time gap between the dataset versions. All the details can be found in Appendix A.4.
4.2 EXPERIMENTAL RESULTS
Results on LiveXiv vI: We evaluated 17 LMMS across two prominent tasks, VQA and TQA. Table I provides a detailed summary of the performance across both tasks. One interesting observation is Claude's superior performance on all tasks. This substantial performance gap suggests that Claude's 's architecture and underlying methodologies are particularly well-suited for both VQA and TQA tasks. The results align with other relatively close benchmarks, DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022) and AI2D (Kembhavi al., 2016), where see similar trend: Claude has significantly higher performance
7

---

Published as a conference paper at ICLR 2025
Table I: LiveXiv v1 VQA and TQA average accuracy across ArXiv taxonomy, (V is VQA, T-TQA)
VQA& TQA Arc.
0-b10.BH
D-bLC.CB
Co.A1
0025.5V
CS.CV
CO.30
Q-bio.0N
q-bio.IO
Mean
T
T
T
# Samples
661
429
900
1024
840
697
685
1069
735
472
720
932
672
570
647
1121
844
1195
6.34
894
7:528
9000
IP
21.2
18.1
25.2
16.6
19.5
20.2
24.5
21.8
23.4
18.9
21.3
20.7
22.6
22.8
24.9
169
24.1
18.5
21.1
18.2
23.6
19.1
LLaVA-15-78
29.0
24.9
27.8
20.
29.5
25.4
$1.9
23.5
30.5
25.0
51.0
21.8
34.9
24.9
29.1
22.6
29.3
24.9
32.8
25.4
30.4
23.5
28.1
51.9
28.7
2688
28.6
29.7
53.9
20.2
51.0
36.4
51.4
79.0
33.1
30.
27.0
276
27.9
304
29.5
29.1
29.9
29.3
Muni-LLLam1-388
50.0
33.7
299
L1aM4-115-13B
30.5
31.5
Meffe2-88
35.6
57.1
38.4
35.9
432
40.7
40.5
428
38.6
35.0
79.6
400
36.
369
38.
38
IXC2-1KEID-70
33.0
41.1
36.7
40.
33.0
4688
40.1
38.4
35.8
47.
45.7
37.0
64.5
419
37.9
428
35.8
413
36.1
42.6
37.7
41.5
IXC255-78
46.2
42.3
26.1
40.5
48.2
48.4
53.3
32.8
50.5
50N
45.1
39.
47.0
44.6
47.
47.5
29.4
36.7
268
4G.7
28.1
42.
L2temVL2:28
48.4
42.7
48.1
44.2
50.4
53.7
53.4
23.9
50.5
58.3
46.3
44.7
54.2
516
48.
520
48.2
89.
505
5200
49.8
49.1
11aVA-166-40 6-348
A8.4
49.5
45.6
40.4
47.4
55.7
55.9
49.
$2.5
90.7
518
48.9
54.9
517
47.
492
47.9
46.6
50.2
508
500
902
kdefics3
51,
47.2
50.6
52.3
55.5
57.2
57.0
53.
47.
566
51.9
51.5
51.
51.
4800
56.6
518
53.7
902
LLAVA OneViiin11
Mhidv
00.1
51.4
54.4
59,9
548
645
61.8
3600
48.7
58.
510
560
58.
GP
64.1
50.7
55.9
518
58.8
56.2
62.9
51.3
61.4
62.3
601
508
60.3
56.1
55.2
563
59.0
55.1
61.1
55.0
60.
51.5
64.5
57.5
969
57.5
61.4
65.3
67.0
57.
65.3
67.2
59.9
601
65.3
618
584
608
614
59.1
65.6
614
62.3
602
Qwm2-V7
68.0
60.3
62.4
59.6
71,8
67.3
67.2
59.7
69.3
701
63.3
62.6
64.6
64.6
64.5
61
63.7
592
71.
65.0
66.6
62.
Sladd--Semet
78.9
84.0
72,3
81.2
77.4
80.3
77.7
84.5
78.4
85.6
69.9
84,0
74
86.5
729
829
76.4
86.4
75.9
8233
75.4
835
Table 2: Performance change between LiveXiv (v1) and manually verified subset averaged across all evaluated models. LiveXiv robust, thanks to excessive filtering steps which keep the labeling errors low.
LiveXiv
Verified Subset
Absolute Avg.
VQA
46,734
47.273 46.028
2.336
TQA
45.101
2.105
over the runner-up models such as Owen2-VL, GPT-4o and IntemVLL2-8B.. See Table 4 for more details. However, notable caveat is that Claude plays integral role in the filttrinnii process, which may introduce a potential bias in favor of questions it is predisposed to solve effectively. This implies that while Claude's overall performance remains strong, the evaluation might not fully reflect its robustness to novel or more diverse question types outside the scope of this filtering. Surprisingly. GPT-4o has low performance. To understand the root cause of this observation, we experimented with various typer-purameters such as temperature, image resolution, and diverse textual prompts. Nevertheless, these all yielded similar results.
We further observe that newer models, such as Intern IntemVL2-888 VL2-8B and Qwen2- VL., consistently outperform older models like LLaVA-1.. 16 and Idefics2, suggesting rapid advancements in LMM development over the past few months. This trend highlights the continual improvement in both architecture and training paradigms, leading to better generalization across multi-modal tasks.
Zooming into the -ppeiii-pecifii performance using an ArXiv-based taxonomy, we evaluate each model's effectiveness in distinct scientific fields such as biology, electrical engineering, and mathematics. Our results show that certain models, particularly the newer architectures, exhibit a higher degree of robustness across diverse domains, highlighting that the models' training data might already have potential contamination issues. Conversely, for VQA, models in the Intern- VL22 and the LLaVA families appear to be more sensitive to domain shifts, performing inconsistently across different scientific areas, oppose to the more recent models like Qwen2-VL. Claude and GPT-4o, see Figures 6, 7 for more details. For TQA, it's not the case, probably since the questions test more specific skills such retrieval and arithmetic manipulations, see Figures 8, 9. This domain-specific sensitivity emphasizes the need for further refinements in LMMs, especially when applied to specialized scientific knowledge domains. Overall, this analysis not only underscores the ongoing evolution of LMMs but also highlights areas for further investigation, especially concerning model adaptability to diverse content domains and the potential biases introduced by models.
Contamination free effect: Interestingly, focusing on new data that came after the LMMs were trained, allows LiveXiv to provide a new, contamination-free perspective on the relative performance ranking between strong LMMs. For example, taking the official results from original publications and computing the average ranking of the evaluated LMMs over the established Doc (Mathew et al., 2021), ChartQA (Masry et al., 2022) and AI2D (Kembhavi et al., 2016) benchmarks, and comparing those to the average rankings provided by LiveXiv, we observe some significant ranking changes. e.g., IXC2.5 and IXC2-4KHD drop over 4 points in average ranking. See Table 5 in the appendix for details.
Performance on manually filtered dataset: To further verify our proposed automated question-anse ansser generation and filtering methodology and to obtain measure of errors in the generated data, we manually verified a subset of 1000 samples (500 for both, VQA and TQA) and evaluated all models on this subset. Table 2 presents the results for VQA and TQA on the filtered subset. We that average
8

---

Published as a conference paper at ICLR 2025
VQA
TQA
VQA
TQA
100
Spear maramann correlation (p.)
m=5
--- m=12
45-
95
4-
85
85
2-
2
80-
mm
mm8
80-
m=5
m-
75
75
v4
V2
v4
V2
v3
v4
LiveXiv version
LiveXiv version
LiveXt version
LiveXiv version
Figure 4: Average prediction errors and rank correlations for overall performance. We report MAE (= mean absolute deviation) for non-re-evaluated models and Spearman's 's rank correlation across 19 LMMs on different LiveXiv versions. Re- Re-evaluating just 3 models is generally sufficient for accurate performance prediction.
VQA
TQA
VQA
TQA
m=3
m-8
Speaman contel contelation (e.)
100
100
m=12
95-
95
90-
90-
80-
mm3
m=8
80-
m
m- 12
v4 75
75
v3
v4
V2
LiveXiv
LiveXiv
LiveXty version
LiveXiv version
Figure 5: Average prediction errors and rank correlations when we predict performance for cach domain separately. We report average MAE (+: average mean absolute deviation) for ron-ure-valuateed models and average Spearman's 's rank correlation across 19 LMMs on different LiveXiv versions. A worse performance LiveXiv v3 can be explained by smaller number of evaluations.
the performance only fluctuates by 2.3% and 2.1% for VQA and TQA when comparing the results obtained by all the models the entire dataset and the manually verified subset. These results hint that our automated question-answer generation pipeline and the filtering methodology is quite robust. Detailed results can be found at the Appendix, Tables 6 and 7.
Efficient evaluations of LMMs: In this section, we empirically validate the effectiveness of our proposed efficient re-evaluation method for LMMs using LiveXiv v1-v4. Dynamic benchmarks like LiveXiv present a challenge in terms of evaluation costs since each time a new version of the benchmark is released, all models should be re-evaluated on the updated data. This process, however, can become computationally prohibitive when dealing with numerous models. Our goal is to demonstrate that by re-evaluating only a small subset of models on a new version of LiveXiv, we can still reliably predict the performance of the remaining models. For this experiment, we evaluate 17 models on the first version of LiveXiv but only re-evaluate m E (3,5,8.12) models on the subsequent versions of the benchmark using the model selection methodology detailed in Section 3.3; we focus on VQA or TQA (but not both simultaneously) and consider all ArXiv domains. From LiveXiv v2 onwards, two new extra models were added to the analysis; these do not count towards the budget of m models at v2.
At every step, an IRT model is fitted to the full observed data and we predict the performance of the non-re- evaluated models on the full VQA and TQA versions of LiveXiv and on each ArXiv domain using empirical versions of equation 2. Figure 4 presents the results for the case we aim to predict the performance of the non- re-evaluated models on the full VQA and TQA versions of LiveXiv while Figure presents the averages of errors and correlations across domains. They report both the mean absolute error (MAD) (#: mean absolute deviation) for the non-re-evaluated models when predicting their accuracy and Spearman's 's rank correlation across all 19 LMMs on different LiveXiv versions when comparing real accuracy and predicted accuracy. These results suggest that re-evaluating just 3 or models is likely be sufficient for accurately predicting the performance of the remaining models and the ranking of all models, especially when our focus is predicting the overall performances. This last observation makes sense since the full benchmarks contain many more data points when compared to individual domains and, in that case, individual prediction errors tend to cancel out in the empirical version of equation 2. For the same reason, we ended up having a poorer performance LiveXiv v3, which has fewer data points; as detailed in Appendix A.4.4, LiveXiv v3 has questions generated by three models as part of an ablation study, however, we ultimately use the questions generated by GPT-4o and Claude-Sonne in this study to keep procedures more standardized with v4.
In Appendix A.S, we present additional results to further validate the effectiveness of our method. Specifically, we (i) show detailed error analysis of the results for LiveXiv v4 (from Figures 4/5), (ii)

---

Published as a conference paper at ICLR 2025
Table 3: LiveXiv accuracy (%) on different categories of question and partitions averaged over all evaluated models.
Data Analysis
Reasoning
Attribute
Localization
Reading
Arithmetic
Charts
Block Diagram
Qualitative
VQA 46.93
47.95
46.18
41.91
47.83
46.87
44.17
52.69
48.60
TQA
46.02
63.61
68.69
51.66
59.35
35,56
test situation with high distribution shift using a hypothetical version of LiveXiv consisting of ArXiv papers from 2010 (v0), (iii) validate our efficient evaluation method when questions are also generated by Qwen2-VL-72B v3, and (iv) test our approach on MM-LiveBench (Zhang et al., 2024b).
4.3 ANALYSIS AND ABLATIONS
To analyze various aspects of LiveXiv we provide extensive ablation study. We start by providing an analysis of the results from different models obtained w.r.t the language content partitions, then provide results for different models w.r.t the visual data partitions.
Language analysis- performance according to question type. To discover error slices of models for an analysis of mistakes they commonly make, we classify the questions present in the benchmark into one of the following categories: reasoning, data analysis, reading, localization, attribute, and arithmetic. To achieve this classification, we employ the Llama-3.1 (Meta, 2024) LLM and prompt the model with the question and the list of categories to choose for this question. The prompt is provided in the Appendix Figure 21. Table 3 summarizes the results for all the models. We see that the performance of these models on the arithmetic partition is the lowest on average compared to other partitions highlighting room for potential improvement. We also provide the detailed results for all models on these partitions for VQA and TQA in Tables 9 and 10 of the Appendix.
Vision analysis- performance according to figure type. For a more fine- fine-grained analysis of LMM performance on different types of visual data present in our benchmark, we first categorize the data through Meta-Prompting for CLIP, proposed by Mirza et al. (2024), in a zero-shot classification setup. Specifically, we classify the image content into three categories of figures: Block diagrams, Qualitative visual results, and Charts. We summarize the results in Table 3. Detailed results for each model's performance can be found in Table 8 in the Appendix. The results reveal significant variance in performance across figure types for nearly all models. In most cases, block diagrams are the most favorable category for models. However, IntemL LntemLM-Xccoppserr Xcomposer2-4KHDD.
(Dong al., 2024b) stands out by achieving the highest accuracy on Qualitative figures. Overall, Charts emerge as the most challenging figure type on average, suggesting a lack of sufficient examples in the training data for this category. This kind of analysis can be further expanded to include more categories and discover error slices on which different models struggle so that potential targeted improvements can be designed for these models to mitigate the shortcomings.
Diversity & Difficulty The visual content of scientific papers (figures and tables) evolves over time, and we aim to ensure our questions remain diverse across dataset versions. To achieve this, we use an LLM (Llama- 3.1 Meta (2024)) to classify questions into predefined categories, allowing us to monitor and maintain diversity. Table 18 confirms that LiveXiv maintains question diversity across versions. Additionally, we assess question difficulty by analyzing how many models answer each question correctly. Figure 10 shows that the difficulty distribution for both VQA and TQA tasks remains stable across dataset versions.
5 LIMITATIONS AND CONCLUSIONS
Limitations. LiveXiv relies on capable proprietary LMMs in order to be fully automatic, and with high quality. However, relying on proprietary LMMs is limitation since do not have full control over the models, they can change through time and might affect LiveXiv. Nevertheless, we commonly expect them continuously improve leading positive impact LiveXiv effectiveness.
Conclusions. We propose LiveXiv, an ever-evolving, fully automatic, multi-modal benchmark focused on scientific domains to tackle test set contamination issues and consequently allow a new (contamination- free) perspective on relative ranking of advanced LMMs. We utilize ArXiv, as the data source, carefully and extensively crafting quality dataset to evaluate LMMs. To significantly reduce the computational and logistical overhead of maintaining the dataset throughout time and models, we propose an efficient evaluation method that can save more than 70% of the evaluated models on each dataset version. Our method can be extended to other archives such as BioRXiv to extend our dataset to new domains. For future work, we propose enhancing LMM evaluation by incorporating free-form questions to assess generative abilitics and complex question types to evaluate reasoning capabilities.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This work introduces LiveXiv, live multi-modal benchmark for evaluating LMMs using scientific ArXiv papers. By relying solely publicly available ArXiv manuscripts with proper licenses, we ensure com- pliance with copyright and distribution policies. The automated generation of Visual Question Answering (VQA) and Table Question Answering (TQA) pairs enables scalable evaluation of LMMs without human involvement, minimizing the risk of human biases in data collection. However, we acknowledge the poten- tial for unintentional biases within the models dataset itself. Continuous evaluation and refinement necessary to mitigate these biases and promote the responsible deployment of LMMs in wider applications.
7 ACKNOWLEDGMENNS
This research was partially supported by the TAD center at Tel Aviv University. The authors thank Eli Schwartz for his valuable feedback and assistance.
REFERENCES
Introducing the next generation of claude.
https://www.atthrpggt hwtp://woop....ccooooooooeecoome com/news/
claude-3- fami ly, 2024.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally your phone. arXiv preprint arXiv:2404.142111 2024,
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg. Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024.
Scale AI. Seal leaderboards. https: http://ale..cmmeaeeereeeeeee
2024.
Jean-Baptiste Alayrae, Jeff Donahue, Pauline Luc, Antoine Miech, lain Bar, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. 22Xh::2204144990 2022.
Assaf Ben-Kish, Moran Yanuka, Morris Alper. Raja Giryes, and Hadar Averbuch-Elor Mitigating open-vocabulary caption hallucinations. In Conference on Empirical Methods Natural Language Processing, pp. 22680 22680-22698, 2024.
Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):224-4555 1952.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv:2005. 14165, 2020.
Li Cai, Kilchan Choi, Mark Hansen, and Lauren Harrell, Item response theory. Annual Review of Statistics and Its Application, 3:297-321, 2016.
Jun Chen, Deyao Zhul Xiaoqian Shenl Xiang Li, Zechun Liu2 Pengchuan Zhang, Raghuraman Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning. arXiv preprint 17X0922310.09478, 2023a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- -aiiunl models? arXiv preprint arXiv:2403.2000 203300 2024.
11

---

Published as a conference paper at ICLR 2025
Yunxiao Chen, Chengcheng Li, Jing Ouyang, and Gongjun Xu. Statistical inference for noisy incomplete binary matrix. Journal of Machine Learning Research, 24(95):1-66 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic Ssuall-inguistic tasks. arXiv preprint urhh:2312.14258: 2023c.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* CharGPT Quality, March 2023. URL
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chathot arena: An open platform for evaluating lims by human preference. arXiv preprint arXiv:2403.04132,, 2024.
Douglas H Clements, Julie H Sarama, and Xiufeng H Liu. Development of measure of early mathematics achievement using the rasch model: The research-based early maths assessment. Educational Psychology, 28(4):457-44) 4825, 2008.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP. Towards General-purpose Vision-Language Models with Instruction Tuning. In NewrlPS, 2023.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409. 199917116, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale
hierarchical image database. In Proc. CVPR, 2009.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, al. Interllm-xcompose22. Mastering free-form text-image com- position and comprehension in lisnua-language large model. arXiv preprint arXiv:2401.1644 2024a.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang. Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Deaemiimxccoposer2--H1 language model handling resolutions from 336 pixels to 4k hd. arXiv preprini 4rXiv:240..0651. 2024b.
A pioneering large vision-
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprira arXiv:2306. 13394, 2023.
Rasch Georg. Probabilistic models for some intelligence and attainment tests. Copenhagen: Institute of Education Research, 1960.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0333 2021.
Irene Huang, Wei Lin, M Jehanzeb Mirza, Jacob A Hansen, Sivan Doveh, Victor lon Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuhene, Trevor Darrel, et al. Conme: Rethinking evaluation of compositional reasoning for modem vlms. arXiv preprint 0681422406.01164, 2024.
12

---

Published as a conference paper at ICLR 2025
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2it.:::400007974,, 2024.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprini 1rXiv:2405.01488 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. dia- gram is worth dozen images. In Computer EssonnEECy 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Department of Computer Science, University of Toronto, 2009.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprini arXiv:2408. /2637, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02244 2024b.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, 2024a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024b.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVise Vision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.0m 033266, 2024c.
Bohao Li, Yuying Ge. Yixiao Ge, Guangzhi Wang. Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 13299-13308, 2024d.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre- training with Frozen Image Encoders and Large Language Models. arXiv preprint 121 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Kumnr, al. Holistic evaluation language models. arXiv preprint arXiv:221 1,09110, 2022.
Wei Lin, Muhammad Jehanzeb Mirza, Sivan Doveh, Rogerio Feris, Raja Giryes, Sepp Hochreiter, and Leonid Karlinsky. Comparison visual instruction tuning, arXiv preprint arXiv:2406.09240. 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning, arXiv:2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. LLaVA-Next (LLAVA 1.6). arXin:3310.03744, /alllaaaaaallll
2023b. URL htt.tttpss//laaaa-ll.iiii.t
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jac Lee. Visual Instruction Tuning. In NeurlPS. 2023c.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint Niv:2337.006288 2023d.
Lord, MR Novick, and Allan Birnbuu.. Statistical theories of mental test scores. 1968.
Yujie Lu, Dongfu Jiang. Wenhu Chen, William Yang Wang. Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating langunel models in the wild with human preferences. arXiv preprint arXiv:2406. 1069, 2024.
13

---

Published as a conference paper at ICLR 2025
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. imyyenchmarks: evaluating lims with fewer examples. arXiv preprint arXiv:2402.1499,, 2024a.
Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuckai Sun, and Mikhaii Yurochkin. Efficient multi-prompt evaluation of Ilms. arXiv preprint Xiv:2445511202. 2024b.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: ftn..///i///222...nnn.nn---- https: http:////alaatthoooooo org 20222. findings-acl... 177.
URL
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images.
In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200- 2209,
2021.
Meta. Introducing meta llama The most capable openly available Ilm to date. waa.mmttaa...mm www.cc///occco wello//mttaalammete 2024.
https:
M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger. Meta-Prompy Peta-Promptiing for Automating Zero-shot Visual Recognition with LLMs. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. DOCCI: Descriptions of Connected and Contrasting Images. In ECCV, 2024.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287. 2024.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuee and Leshem Choshen. Efficient benchmarking (of language models). arXiv preprint arXiv:2308. 11696, 2023.
Ameya Prabhu, Vishaal Udandarao, Philip Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. Lifelong benchmarks: Efficient model evaluation in an era of rapid progress. arXiv preprini arXiv:2402. 19472, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In Proc. ICML. 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer. orXiv:1910 10683, 2019.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? longitudinal perspective on Ilm data contamination. In The Twelfth International Conference on Learning Representutions, 2023.
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 100440.29444 19450, 2024.
14

---

Published as a conference paper at ICLR 2025
Alain Starke, Martijn Willemsen, and Chris Snijders. Effective user interface designs to increase energy-efficient behavior in a rasch-based energy recommender system. In Proceedings of the eleventh ACM conference recommender systems, pp. 65-73, 2017.
Deep Search Team. Deep Search Toolkit, 6 2022. URL
seepsearch-toolle
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models. arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXtv:2307.09280 2023b.
Wim J Van der Linden.
Handbook of item response theory: Volume 3: Applications. CRC press, 2017.
Wim Van der Linden. Hundbook of item response theory: Three volume set. CRC Press, 2018.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.1985 2023.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-fre Ilm benchmark. 2024. URL www.aaaaiviiiiieee wwwpxxrrpprrintttaii.22000000
Moran Yanuka, Assaf Ben Kish, Yonatan Bitton, Idan Szpektor, and Raja Giryes. Bridging the visual gap: Fine-tuning multimodal models with mnwwedge--ddaple captions. In NAACL, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- diiiciiiciiiieee multimodal understanding and reasoning benchmark for expert agi. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9561 2024.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.0m 00332, 2024a.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT4: Enhancing Vision- Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304. 10552,.. 2023.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 RELATED WORKS
A.1.1 EFFICIENT BENCHMARKS
With the increasing amount of tasks and samples in current benchmarks, evaluation the full suite is ume-consuming and cost-intensive. Efforts underway to develop efficient benchmarks that reduce computation costs without sacrificing reliability. For LLMs, Perlitz et al. (2023) proposed the first systematic study of the effects of language model benchmark designs reliability and efficiency, and applied efficient benchmark practices on the HELM benchmark (Liang et al., 2022), leading to x 100 computation reduction with minimal loss on reliability. Lifelong benchmarks (Prabhu al., 2024) has an ever-expanding pool of test samples for the categories in CIFARIO (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009); to make this design economically feasible, it reuses past model evaluations on a sample set through dynamic programming to enable efficient evaluation of new incoming models, drastically reducing the evaluation cost. Most related to our work, tinyBenchmarks (Maia Polo et al., 2024a) and PromptEval (Maia Polo et al., 2024b) propose using Item Response Theory (IRT) (Lord et al., 1968) estimate the performance of L.LMs on unseen samples, making efficient evaluation possible by only conducting small fraction of the total number of evaluations. Inspired by the last two works, leverage IRT to estimate the performance of older models new batches of data. More specifically, at each version of LiveXiv, we choose small core set of models 55 previously added the leaderboard and re-evaluate them on the new data. Depending on their responses to the new samples, we estimate the performance of the remaining old models on the new benchmark version.
A.1.2 LLM CONTAMINATION-PREE BENCHMARKS
For LLMs, LMSys Chatbot Arena (Chiang al., 2024) and A12 WildVision (Lu et al., 2024) create user-focused platform that provides contamination free environment for proper evaluations. However, it is expensive to collect tens of thousands of human preferences the compared language models. Furthermore, Seal Benchmark (AI, 2024) proposes private questions paired with human evaluations. Srivastava et al. (2024) update the questions in the MATH dataset (Hendrycks et al., 2021) by changing numbers in the math questions. LiveBench White et al. (2024) collects frequently updated questions from diverse information sources e.g.math competitions, arXiv papers and news articles and more challenging versions of existing benchmark tasks. Concurrently, LiveCodeBench (Jain et al., 2024) contributes live benchmark on broader code-related capabilities. Note that these datasets focus on language data only.
A.2 ANALYSIS & ABLATIONS
Table 4: Average results for relatively close benchmarks (DocVQA, ChartQA and AT2D). We can that Claude, GPT4o, Qwen2-VL. VL and InternVI.2 the top models. The overall ordering is align with our benchmark.
ChanQA
DocVQA
A12D
InstructBLI--7B
10.9
74
40.6
L.VV--.6-Mistra--
51.8
72.2
69
Mantis
42.9
60.4
LLaVA-1.5-7B
17.8
74.4
55.5
LLaVA-1.5-13B
18.2
77.5
70
Idefics2
74
72.3
IntemVLL2-2B
76.2
86.9
74.1
IXC2-4KHD-7B
81
IXC2.5-7B
82.2
81.6
LLaVA-OneVision
80
83.7
81.4
Phi3v
72
84.9
77.8
Idefics3
87.7
76.5
LL.VVA-L6-34B
67.6
78,9
GPT-4o
85.7
92.8
94.2
Qwen2-VL
83
94.5
83
Intem IntemVL2-8D
83.3
91.6
83.8
Claude-Sounet
90.8
95.2
94.7
16

---

Published as a conference paper at ICLR 2025
VOA PPrfformnnce of models according to domains
Vieak models undorf2
Mid-aval models
High sonsituuty tomain
LLA003 6-78
LLVALL 70
LUAV 5-330
DeORR V.
ChuteSoomd
LONA Oww.
LLNDA
Figure 6: Domain sensitivity according to domains. We visualize the performance of each model across all domains. Clear trends revealed where old models or models with a small LLM are "under-fitting" and perform worse across all domains. In the middle have the mid-level models that sensitive the domain, indicating their lack of generalization across domain without any additional training. Lastly the newest models (open-source and proprietary) robust domain shifts and present stable performance across the domains.
Table 5: Average ranking on static benchmarks (ChanQA, DocVQA and A12D) and LiveXiv (vl). We can see from the ranking diffcrence column that some models have significant drop (negative difference) in the relative ranking in LiveXiv compared to the static datascts. The gap highlighting potential risk of test data contamination when using static (frozen in time) benchmark datasets.
Model
Static datasets
LiveXiv Difference (static livexiv)
InstrucBLIP-775
15.33
17.00
-1.67
L
13.67
16,00
-2.33
Mantis
14.50
14.50
0.00
LLaVA-1.5-7B
14.33
14.50
40.17
LLaVA-1.5-13B
12.67
13.00
-0.33
Idefics2
13.00
12.00
1.00
IntemVLL2-2B
9.00
10.00
-1.00
IXC2-KKHD-7B
6.33
10.50
4.17
IXC2.5-7B
5.00
9.50
4.50
LLaVA-Oneission-77
8.00
6,50
1.50
Phi3y
9,00
6.00
3,00
Idefies3
8.50
6.50
2,00
L.LVV-L.6634B
9.33
6.50
2.83
GPT-4o
2.33
4.00
-1.67
Qwen2-VL.
3.33
3.00
0.33
InterV
3.33
2,00
1.33
Claude-Sonnel
1.00
1.00
0.00
17

---

Published as a conference paper at ICLR 2025
according domains
0500
6300
Figure 7: LMMss performance based domain. To complement analysis form Figure 6 visualize the statistical properties of each domain. One clear trend that across all modesl, the performance on cs.CV and es.Al is the most concentrated, hinting lower variance between models.
TQA Rerformance according
Accuaay
LUNAL 1278
LWAALS
11000 5-330
MEM
GP44
Goude
Figure 8: Domain sensitivity according domains. opposed to the high variance some models demonstrated in Figure 6, in TQA the tasks and he visual content are more limited thus shrunken the performance variance greatly.
18

---

Published as a conference paper at ICLR 2025
VQA Accuracy according to domains
ACCUPCC
105.00
1200
1RSSS ST
Figure 9: LMMs performance based on domain. The domains are very similar in their statistical
properties showing high variance in performance. This is probably due wide range of models that differ significantly in their performance.
19

---

Published as a conference paper at ICLR 2025
A.2.1
PERFORMANCE CHANGE COMPARED MANUALLY CURATED SUBSET
Table 6: VQA Performance change between I LiveXiv (v1) and manually curated subset (500 examples).
Model
LiveXiv Manual (%)
Performance Change
LiveXiv Rank
Manual Rank
Change
InstructBLPP..
23.216
21.346
-1.87
17
17
ImcmL.--Xoomoor225-77
47,839
50.769
2.93
10
IntemVLL-88B
61 .5888
66. 154
4.596
3
LLaVA-L.6-Mstrrll
29.163
26.346
-28166
16
LLAVA-One-Viooo-vvve2-777
52.864
56.154
3.29
7
LLaVA-1.5-13B
31.839
30.385
-1475
14
13
LLaVA-1.5-7B
28.654
-1.329
15
14
LLVA--1.634BB
49,196
53,269
4.073
Mantis-IL.ama3
32,094
28.654
-3.44
13
14
Phi3v
58,141
58.654
0.513
5
36,801
33.654
-3.147
12
12
Idefies2-BB
36.851
36.731
-0.12
11
11
MMemVLL2-21
49, 39,548
48.654
-0,94
8
10
Claude-Sonet
75.942
79.615
3,673
Qwen2-VL
66.248
71.346
5.098
2
2
4
GPT-4o
60,303
60.577
0.274
52.881
52.692
Average (absolute) change
Table 7: TQA Performance change between LiveXiv (yl) and manually curated subset (500 examples).
Model
LiveXiv
Manual (%)
Performance Change
LiveXiv Rank
Manual Rank
Change
InstructBLIP-1
18.5
17
17
IntemL
IncmnLM-Xooooo2225-77
49.1
45,9
-3.2
9
IntemVL2-88
62.1
65.3
3.2
2
2
L.VA-1.6-Missra-a
16
LLAVA-OkeViiioo--qeee222
50.2
51.6
1.4
7
8
LLaVA-1.5-13B
30.9
31.2
0,3
13
13
LLaVA-155-BB
29.6
14
14
LLAVA-16634B
51.8
52.2
0.4
5
Mantis-I am833-88
29.3
28.0
-1.3
15
15
Phi3v
7
IntemL--Xxoompee
Idefies2-8B
38.2
42.0
388
12
10
-2
IntemVL2-2B
41.5
39.5
-2.0
11
12
Claude-Sonme
Qwen2-VL
60.2
58.3
-1.9
3
GPT-4o
54.5
55.7
1.3
3
Idefies3
56.4
Average (absolute) change
2.105
A.2.2 FIGURE TYPE
We provide all the details for VQA performance according to figure type content in Table We divide the performance the following figure types: "Chart", "Block Diagram" and "Qualitative".
A.2.3 QUESTION CATEGORY
In Tables 9 and 10 provide detailed results for VQA and TQA performance according the category of the questions as classified by an LLM. We divide the performance to the following categories: "Data Analysis", "Attribute", "Reasoning". "Reading", "Localization" and "Arithmetic"
A.2.4 QUESTIONS' DIFFICULTY
We provide additional analyses regarding the diverse difficulties of LiveXiv's generated questions. Figure 10 demonstrates that for all LiveXiv tasks and for both the first and second versions, a wide range of difficulties is present, where most of the questions concentrate in the middle, suggesting our dataset indeed challenging.
20

---

Published as a conference paper at ICLR 2025
Table 8: Performance LMMs over different figure types from LiveXiv v1 VQA set (the amount samples for each figure type is in brackets).
Model
Chart (4354)
block diagram
Qualitative (864)
(2110)
InstructBLIP-7B
22.9
22.8
22.5
46.7
53.5
41.7
InternVL.2-8B
59.1
68.4
63.8
LLaVA-1.6-Mistrall73
27.3
33.6
33.4
48.7
62.8
57.9
LLaVA-1.5-113B
29.3
35.4
40.2
LLaVA-1.5-7B
28.1
33.1
36.0
LLaVA-1.6-34B
44.6
60.0
52.7
Mantis-LLama3-SB
29.3
36.2
36.1
Phi3v
56.1
65.5
55.2
IntemL
32.9
43.7
47.1
Idefies2-8B
34.3
43.0
41.1
Intern VI.2-2B
47.2
54.9
50.0
Haude-Sonnee
73.7
81.3
69.1
Qwen2-VL
63.1
73,9
66.0
GPT-4o
56.5
68.6
59.5
idefics3
51.1
59.0
54.1
Table 9: LiveXiv v1 VQA Performance by Question Categories (the amount of samples for each category is in
brackets).
Model
Data Analysis (2291)
Reasoning
Attribute (903)
Localization
Reading (1470)
Arithmetic
(872)
(1596)
(154)
InItrcutBIIP
21.16
29.29
22.77
31.25
23.87
23.01
Interl M
47.52
43.43
48.51
31.25
50.46
47.28
Intern VL.2-B
61.74
63.64
65.35
56.25
62.54
62.41
LLaVAI.6-7B
28.91
31.31
30.69
12.50
30.00
30.43
LLaVA-OneVision
52.57
54.55
60.40
56.25
56.76
52.85
LLAVA1.5-13B
32.92
25.25
25.74
43.75
31.85
32.58
LLaVA1.5-BB
30.56
28.28
29.70
25.00
29.71
30.89
LLaVA1.6-23
50.87
43.43
45.54
37.50
49.60
50.00
Mantis
32.50
33.33
30.69
43.75
31.97
31.80
Phi3v
58.05
63.64
59,41
43.75
58.15
59.07
M-XC-4KK
37.51
43.43
39.60
18.75
38.67
37.09
Idefics2
37.79
39.39
35.64
31.25
39.54
36.34
Intern VL2-2B
50.26
46.46
44.55
43.75
51.45
48.62
Stade-Sonner
74.78
78.79
67.33
81.25
75.49
75,64
Qwen2-VL
66,93
66.67
68.32
43.75
68.73
65.01
GPT4o
61.41
60.61
59.41
62.50
59.83
59.76
Idefics3
52.34
63.64
51.49
50.00
54.51
54,00
21

---

Published as conference paper at ICLR 2025
Table 10: LiveXiv TQA Performance by Question Categories (the amount samples for each category in brackets).
Model
Data Analysis
Reasoning
Attribute
Localization
Reading
Arithmetic
(2582)
(123)
(121)
(23)
(2127)
(3934)
InstrucBBLIP-72
24.7
27.6
34.7
13.6
InternL.
54.9
75.6
79.3
60.9
74.6
29.7
InternVL2-8B
58.6
73.2
78.5
65.2
79.8
54.1
1aNAA-1.--Mistrl-77
30.4
39.0
52.1
30.4
32.0
13.0
LLaVA-OneVision
47.5
72.4
80.2
60.9
69.1
40.3
LLaVA-1,5-13B
31.6
49.6
47.9
30.4
32.4
28.5
LLaVA-1.5-7B
30.8
39.0
42.1
43.5
31.6
27.9
LLaVA-1.6-34B
46.4
69.1
76.9
47.8
66.8
46.1
Mantis-LLama3-4
28.7
46.3
32.4
27.3
Phi3v
52.2
76.4
77.7
60.9
72.4
35.2
Internl Mc-Xcmpoorrr24KHH1--
45.4
69.1
77.7
60.9
66.2
25.0
Idefics2-8B
35.0
57.7
57.9
43.5
51.1
32.4
ILter
36.9
57.7
70.2
30.4
62.2
32.1
Claude-Sonnet
85.1
90.2
91.7
87.0
91.0
78.1
Qwen2-VLL VL
64.6
86.2
90.1
65.2
82.0
44,0
GPTT-0
57.5
79.7
86.8
69.6
73.0
40.6
Idefics3
52.0
79.7
77.7
56.5
72.0
36.6
630
230
780
580
583
522
390
230
100
Figure 10: LiveXiv questions' difficulty diversity. LiveXiv is composed of questions varying levels of difficulty. Each row shows a version the dataset on both of its task (VQA and TQA). LiveXiv demon- strates questions in wide range of questions' difficulties, from questions that model able to answer correctly up to questions that all models were able to easily answer, The center mass consternates the middle. hinting the dataset indeed challenging and the fact that the difficulty distribution remains the same for both the first version and second version suggests the stability of our diverse and challenging dataset.
22

---

Published as a conference paper at ICLR 2025
A.3 DETAILED EXAMPLES FOR VQA AND TQA GENERATION
Here we present full and detailed examples of our flow from ArXiv papers until constructing verified multi-choice Q&A. Figure 11 shows the full example for generating questions from figures (VQA). Figure 12 shows the full examples for TQA.
Disa capptin
Figure 11: A detailed example for VQA questions generation.
Anreaiis
0443 EDile
"Jamaekko durets
2040
bet relletee
Ie.13
Figure 12: A detailed example for TQA question generation.
23

---

Published as a conference paper at ICLR 2025
Table 1I: Version
VQA and TQA average accuracy across ArXiv taxonomy (the number of samples brackets).
VQA Accuracy
es. (188)
cs.cy .V (200)
eess.08 (209)
eess. (280)
q bibi... (154)
bio.CC (206)
bio.GN (80)
-iooo. (107)
Mean
(196)
(321)
(2067)
InstrouBB... JP
0.29
0.24
0.20
0.22
0.24
0.25
0.27
0.22
0.19
0.25
0.24
Interl M. Iaterl.MXXC222
042
0.42
0.49
048
0.56
0.488
0.51
0.49
0.57
0.46
0.49
VL22
0.30
0.28
0.30
0.34
0.32
0.36
0.26
0.25
0.22
0.30
0.29
InteraVL2-SB
0.61
0.66
0.64
0.69
0.68
0.58
0.55
0.59
0.57
0.62
LLaVAI
0.54
0.56
0.52
0.02
0.55
0.00
0.50
0.50
0.00
0.43
0.54
T1aVA.OneVision
LLaVAI55-13B
0.29
0.35
0.31
0.35
0.33
0.33
0.33
0.28
0.99
0.55
0.31
LLaVA1.5-BB
LLaVAI66---B
0.30
0.39
0.33
0.36
0.29
0.36
0.33
0.32
0332
0.31
0.33
Mantis
0:55
0.60
0.65
0.57
0.67
0.54
0.58
0:57
042
0.58
Philv
0.2
0.42
0.41
0.1
0.29
0.35
0.39
0.36
0.39
0.26
0.37
I-C-LLM-XC-IMil
0.39
0.44
0.42
0.43
0.45
0.38
0.34
0.32
0.40
0.40
Idefies2
0.47
0.48
0.48
0.52
0.50
0.49
0.49
0:43
0.49
0.48
0.48
Chande-Somnet
0.77
0.74
0.81
0.77
0.89
0.79
0.72
0.74
0.55
0.78
0.75
Qwen2-V1.2
0.62
0.69
0.66
GPTNo
0.49
0.52
0.55
0.55
0.58
0.52
0.47
0.49
0.45
0.49
0.51
Idefios3
0.55
TQA Accuracy
ca.AI
C8. CV. (473)
0a. .LG (516)
08 RO
eess.80 (45)
eess. SY (188)
q q-bio.Be BM (118)
q q-bio.C5 (108)
q q-bio.GH (398)
q-bio. TO (115)
Mean
(167)
(247)
(2375)
InstreutLLIP
0.20
0.19
0.23
0.19
0.30
0.21
0.19
0.14
0.24
0.21
0.20
InterL.MXXC2255
0.46
0.38
0.36
0.51
0.38
0.56
0.45
0.44
0.47
0.44
0.45
IntemVL.2.2B
0.1
0.40
IntemVL2-8BB
0.50
0.56
0.55
0.68
0.64
0.75
0.62
0.64
0.54
0.73
0.62
LLVVA16-7B
0.26
0.22
0.30
LLaVA-OneVision
0.48
0.41
0.43
0.57
0.33
0.64
0.49
0.57
0.4
0.55
0.49
11V3115.138
0.37
0.29
0.28
0.36
0.31
0.32
0.29
031
0.30
0.31
0.31
L.7VAI557B
0.32
0.26
0,29
0.32
0.33
0.30
0.88
0.36
0.30
0.30
0.31
LLVVA16-34B
0.54
0.50
0.46
0.55
0:47
0.63
0.58
0.57
0.47
0.54
0.53
Mantis
0.31
0.24
0.29
0.33
0.32
0.19
032
032
032
0.30
Phitv
0.47
0.41
0.43
0.51
0.49
0.60
0.48
0.56
0.46
0.48
0.49
M.
0.43
0.48
Idelic2
0.36
0.30
0.34
0.38
0.33
0.43
0.34
0.41
0.34
0.30
0.35
Claxde-Sonnet
0.77
0.80
0.79
0.78
0.83
Qwen2-VI-7B
0.59
0.53
0.52
0.63
0.53
0.64
0.3
0.66
0.3
0.63
0.58
GPT4o
0.46
0.41
0.40
0.55
0.47
049
0.50
045
0.48
0.48
Idefics3
0.49
0.43
0.44
0.57
0.36
0.57
0.45
0.58
0.46
0.52
0.49
A.4 ADDITIONAL LIVEXIV DATASET VERSIONS
In this section we describe in detail the additional LiveXiv versions, v0 (generated form papers from 2010), vI -l-oposttte which is the same raw data as vI but this time Claude is the QA generation model and GPT-4o does the filtering, and v2 and v3 which consist of multiple participating models. We start by describing each version and the performance of the evaluated models. Lastly, we summarize the diversity of our generated dataset, both terms of visual content and in terms of the different categories of questions.
A.4.1 LIVEXIV VO
We analyzed 100 scientific papers from 2010 and developed vO, which contains 4,500 questions. Due to the dynamic nature of scientific data, vθ represents a significant distribution shift from vl. We used our efficient evaluation method to predict vI from vO, and our high-accuracy predictions demonstrate the method's robustness across large temporal gap. Refer to Table 11 for performance details.
A.4.2 LIVEXIV V VI-OPPOSITE
To show the impact of the the role of each model in the dataset creation, we created an opposite version of vl, where Claude is the model that generates the questions and GPT is the model that does the filtering. Table 12 demonstrates the changes in the absolute performance and in the ranking between the models. Overall, switching roles between Claude and GPT-4o has minimal impact on most models. However, GPT-4o shows slightly variation, dropping slightly in the rankings when it takes the role of filtering questions. This variation occurs because generating the questions provides some advantage.
A.4.3 LIVEXIV
LiveXiv second version (v2) consists 10.4K questions for VQA and 7.7K questions for TQA. In addition introduced 4 new domains from physics, namely, physics chysics.app pwwssicc.optii.
24

---

Published as conference paper at ICLR 2025
Table 12:
compared LiveXiv first version with different configurations, The original configuration GPT4o generates the questions and Claude filters and an opposite version where Claude generates the questions and GPT4o performs the filtering. We see in it that overall, the ranking remains similar and the changes minor. Indeed, for that GPT down bit in the ranking (Claude remains
the generating model there is some advantage first both cases).
VQA Average change (%)
TQA Average change (%)
Ranking change
Ranking change
InstructBLIP-7B
-1.78
9.66
IXC2.5-7B
0
-0.83
13.73
IntemVL2-88B
0
-1.18
2.37
LLaVA-I.6-Miisral-7B
2
-8.64
O
10.72
LLaVA-OnnVVision-7
1
-3.66
-2
13.42
LLaVA-v1.5-13B
0
-6.49
8.48
LLaVA-vI.5-7B
0
-8.00
-1
12.72
LLaVA-v1.6-34B
2
-5.85
-0.35
Mantis-LLama3-SB
2
-7.13
0
10.24
Phi3v
1
-0.49
13.856
IXC2-KKHD-7B
5
-0.47
3
8.60
Idefics2-8B
-6.00
3.21
IntemVL2-2B
0
-1.42
2.61
Claude-Sone Sonnet
-3.11
10.11
Qwen2-VL
0
-0.31
2881
GPT-4o
4.74
13.46
Idefics3
0
-1.89
12.23
Average
0
-3100
8.70
Table 13: Version 2 VQA and TQA average accuracy across ArXiv taxonomy (the number of samples is in brackets).
4-340..0 GG...G
Moua
0.34
0.11
17%
1575
8.11
(1:1
8.51
3.65
185
(14)
1168
194
1.90
1964
$/44
8.50
8:31
8.90
139
153
1530
133
59
1.07
1550
8.50
$.50
133
4.5
8.50
8.0
8.57
5.
048
855
6.07
0.65
8.50
Mards
3-01
5.0
033
035
131
655
035
033
$.3
003
113
135
1938
8.9
0.31
0.76
8.55
0.70
3.51
0100
1005
8.0
1.6
1929
3.91
Quand 11.70
0.50
0.11
8.50
0.38
5.00
032
025
0.53
025
1.50
8.50
3.00
103
1.5
Roul
0.75
8.71
8.71
035
8.35
0373
671
6.27
8.77
8.51
8.03
(09
855
6.99
0175
Mon
(1h)
16.1
(98)
G35
(14)
12011
20121
Tra A5
6441
0.4
0.32
046
0.50
8.53
0.4
0.45
(44
0.66
044
030
Vo. 98
6.56
0.55
0.0
645
8.5g
0.00
0.0
033
ELANN
0.33
0.1
83
0.1
(31)
030
0.45
8.55
0.5
8.0
0.30
175
811
0250
030
TLANNES 10
033
133
830
030
133
Manis
0.45
0.45
014
179
89
435
1351
155
650
0.43
01.
0165
1188
193
835
100
140
83
199
0.50
1161
688
Surlect
0.32
1136
035
100
133
136
033
830
130
1.90
123
039
003
035
0.5
635
CoTTe
830
005
0.5
055
0.45
0.2
062
130
007
870
0.0
010
Boul
035
855
050
0.9
0.81
1.4
635
80
351
85
690
4.5
031
physics.bie physics data-ataaaa to a total 14 domains. Lastly, we also added Two new models, Pixtral (Agrawal et al., 2024) and Molmo-7B (Deitke et al., 2024). Due to our high frequency updates, Quantitative Biology related domains have overleaping pupers between versions. Thus, we filtered these overleaping papers out. V2 shows similar diversity to v1 (see Table 18), similar performance of the evaluated models, resulting an accurate and reliable efficient evaluation (see Figure ??) and lastly, demonstrating LiveXiv be evolving scalable live dataset. Table 13 reports the performance summary.
25

---

Published as a conference paper at ICLR 2025
A.4.4 LIVEXIV v3
To further mitigate the potential bias due single generation and single filtering model, is a combination of participant models, GPT-40, Claude-Sonne and Qwen2-VL-72B.. v3 is actually 3 subsets, where each time different model takes different role. Table presents the results for all 3 subsets. Table 15 shows the performance per subset. Overall we can see that the model ranking remains similar previous version. This hints that the bias of using single model is small. We extended our manual verification process with an additional check of v3. Tables 16 and 17 present these results in detail. The manual check revealed more variations compared to the manual check on vl, which can be attributed to the participation of three models in v3. As shown in Table 15, (ive22-VL-7222 demonstrated weaker performance compared to GPT and Claude. Consequently, its role as the filtering and generation model likely introduced additional noise into the dataset. However, the average performance change for both VQA and TQA remains below 5%.
Table 14: Version 3- VQA and TQA average accuracy across ArXiv taxonomy (the number of samples is in brackets).
13-30
8:010.40 (943
6.80-25 (425)
(80)
(922
(75)
(79)
(335)
(95)
(1055)
(55)
(425)
(335
(250)
55442
0.36
0.36
0135
1126
0.31
1:31
0133
0.3
0.2
0.3)
0.22
1133
0.20
1134
0.35
0.37
0.36
0.31
0142
0.33
3.35
1134
8.31
0.3
0.36
0.33
1135
0.32
1135
0.34
ELINLL 8-1B
0.55
0.99
021
0665
0.57
8.35
066
8.1
8.5
0.3
0.51
0.30
0.53
0257
0.5
ELANN 5-133
0.37
0338
032
0.4
0.3
3.30
035
83
8.0
836
0.3
0.30
0.36
035
035
ELANN 6-348
0.37
0.0)
031
042
0.3
3.7
037
18
83
836
0.3
0.3
0.34
0.7
035
0.35
0:4
034
146
0.3
1.50
136
880
831
831
0.5
1.31
0.77
105
036
0.34
035
047
0
046
131
035
8.0
940
8.0
048
033
040
000
031
0.85
0.09
046
137
048
2.90
035
872
872
130
048
033
6.0
0.99
0.88
030
040
0.5
8.5
038
0.3
8.5
8.5
8.6
8.57
054
635
Milri
0.62
064
0.1
046
064
D.0
066
0.0
3.0
0/2
0.51
043
0.00
RUS
643
(90)
(19)
1961
(91)
(186
170)
(2M
(196
(186
(40)
(41)
(261)
(961
(18)
6:38
6.35
0.16
0111
011
877
0771
915
815
019
0.10
D11
0.70
073
6.0
0.64
036
0.4
046
0.66
1.50
066
0.5
046
0.60
0.4
0.4
0.47
1467
046
ELANN 16-T
0.36
034
0.44
0.31
0.3
832
033
030
0.50
830
633
0.3
035
036
ELANNI 133
0.35
0.2
0.33
0.38
0.3
830
0.3
830
817
83
033
0.3
0.33
031
031
TLANNI 3:03
0.58
(41
00
0.5
010
875
840
810
0.5
D67
0.5g
051
0.0
0.53
(11)
043
04
071
160
89
861
D63
0%
656
0.81
6:11
039
033
146
050
88
830
0.5
010
0.33
041
64
0.2
(90
093
136
807
170
875
033
076
194
057
0.5
0.09
037
00
05)
239
068
886
85
630
03
0%
(5
656
035
035
00
038
45
011
413
45
431
6m
631
Table 15: Average performance (%) on different versions. We clearly can see that version subset (denote v3-s3) has a large shift in the average performance for TQA, suggests that Owen2-VL--22B might generates easier questions.
VQA
TQA
vl
47.1
45.1
v2
50.7
44.0
v3 - sl
51.6
44.5
v3 s2
49.1
41.7
v3 s3
52.2
52.5
v3
50.8
46.3
26

---

Published as a conference paper at ICLR 2025
Table 16: VQA Manual verification. We re-verified our dataset by curating subset of 300 questions (100 from each subset). We can see that version 3 even though composed of different subset, remains accurate with bounded performance change overall and minimal ranking changes.
Overall
Manual
Performance change
Rank Overall
Rank Manual
Rank change
instructhippviicua-7b
22.90
26.75
-3.84
19
19
0
intemim 2ommulm--ccompee2225--000
49.62
51.37
-1.75
12
12
IntemV112-8B
65.43
68.09
-2.66
4
5
Ilavavvl.--mistral-7b
34.21
40.12
-5.91
18
17
1
lava--onevisioon-qwee22-----
56.69
59.88
-3.19
7
9
-2
Ilavav1.5-133h
36.48
43.16
-6.68
Ilava-v1.5-7b
34,92
39.51
-459
Ilava-v1.6-34b
54.59
61.40
-6.11
8
Mantis-8B--iigii-lllmm
35.87
41.34
-5.47
phi3v
61.79
64.74
-2.95
7
36.07
46.20
-10.13
15
2
idefics2-8b
39.64
45.59
-5.95
13
14
IntemV112-2B
51.67
54.10
-243
10
10
claude
77.95
81.46
-3.51
1
0
qwen2vl
69.56
73.56
-3,99
3
2
gD4
50.82
51.67
-085
idefics3
54.94
64.74
-980
7
pixtral
71.49
72.64
-1.15
3
molmo
61.93
69.91
-7.8
mean
50.87
55.59
4.72
Table 17: TQA Manual verification. We re-verified our dataset by curating subset 300 questions (100 from each subset). We can see that version 3 even though composed of 3 different subset, remains accurate with small performance change overall and minimal ranking changes.
Overall
Manual
Performance change
Rank Overall
Rank Manual
Rank change
instractbippvicuuaa7b
19.93
17.96
1.97
19
0
intemim- 2emminn-cooppser225-776
52.28
49.40
2.88
9
11
-2
IntemVL2-2B
45.69
43.71
1.98
12
13
-1
IntemVV1.28B
65.93
73.65
-7.73
2
2
0
Ilav-vvl.-mmistral-7b
25.22
26.35
-1.13
18
18
0
dava-oneviiioon-qeee-------
53.99
$6.29
-2.30
8
6
2
Ilavav1l.-113h
32.24
31.74
0.50
Ilava-v1.5-7b
31.50
33.23
-1.73
Ilava-v1..34hb
56.74
62.57
-5.3
3
Mantis-8B-sighiplllmmaa
30.74
32.34
-1.60
phi3v
55.49
52.40
3.10
7
intermin-xcome
45.44
40.12
5,32
13
14
idefics2-8h
39.13
46.71
-7.88
14
12
2
claude
81.92
88.62
-6.70
qwen2vi
64.36
62.28
2.09
3
4
gpt
49.23
50.30
-1.07
idefies3
54.41
52.10
2,32
8
pixtral
57.45
56.36
molmo
50.12
51.36
-1.25
mean
47.99
48.82
3.06
A.4.5 DIVERSITY THROUGH TIME
We evaluate LiveXiv diversity though time Table 18. shows that each version has a steady distribution of visual content and diverse set of questions for both figures and tables.
27

---

Published as a conference paper at ICLR 2025
Table 18: Questions and visual content diversity through LiveXiv versions, we can see that the diversity persistent and LiveXiv has a rich set of visual content as well diverse questions throughout the new versions.
VQA
Data Analysis
Reasoning
Attribute
Localization
Reading
Arithmetic
Charts
Block Diagram
Qualitative
v
768
280
256
365
320
42
1239
578
214
vl
2291
872
903
1596
1470
154
4354
2110
864
1471
1797
1732
112
2805
1442
v3 3265
1196
1459
1202
1916
321
5113
2141
1573
TQA
Data Analysis
Reasoning
Attribute
Localization
Reading
Arithmetic
45
425
1168
2582
123
121
2127
3934
v3 2289
V2
228
74
23
1125
3436
1808
130
97
16
1590
2572
A.5
EXPLORING MORE DETAILS EFFICIENT EVALUATION
A.5.1 DETAILED RESULTS FOR LIVEXIV v4
In Figure we present more detailed set of results extracted from the experiment presented in Figures 4/5 for LiveXiv v4. Interestingly, we see that the models selected for re-evaluation cover the possible values of performances and that our method can make reasonable predictions for domains that were not included in LiveXiv vl.
A.5.2 LIVEXIV VO (PAPERS FROM 2010) EFFICIENT EVALUATION RESULTS
In Figure 15 we show the performance prediction results of our efficient re-evaluation method on v1 from vO, when re-evaluating models. Despite the (temporal) big distribution shift, efficient evaluation method does still work well.
A.5.3 LIVEXIV v3 EFFICIENT EVALUATION RESULTS CONSIDERING ALL QUESTIONS (GENERATED BY DIFFERENT MODELS)
The third version of LiveXiv (denoted as v3) introduced multiple participating models for QA generation and filtering. Unlike the default pipeline for the two first versions, in which we have GPT4o as the generation model and Claude-Sonnet for the filtering, we augment LiveXiv v3 with more questions but (i) using Claude-Sonen Sonnet as the generation model and Qwen2-VL-72B as filtering model, and using (ii Qwen2-VL-72B VL-72B the generation model and GPT40 the filtering model. The results can be seen in Figure 14; in summary, we have a greater distribution shift when comes to TQA and when Owen2-VIL-72D) is included the generation model, making the prediction task for the IRT model harder, thus increasing the prediction errors for v3 when compared to Figures 4/5. In v4 we narrow the participants only to GPT-40 and Claude-Sonnee playing both roles, and reducing both potential bias from single model for generation or filtering and maintaining accurate predictions.
A.5.4 EFFICIENT EVALUATION ON MM-LIVEBENCII
In this section, we challenge our efficient evaluation method, by examining its performance over another type of multi-modal live dataset Zhang et al. (2024b). The dataset has 3 versions (May 2024, June 2024, and July 2024), and each version has roughly 250-300 samples of open-ended questions scraped from newspapers. To evaluate of questions where the true answer rephrased and more negative answers are proposed. Then we evaluate 13 LMMs over all the dataset versions. We the first version training set and predict the performance over the new concatenated sets using our IRT-based method. Figure 16 shows that our method still performs well on different benchmark when re-evaluating only 5 models.
method we use GPT-4o to convert the open-ended questions into closed-form
28

---

Published as a conference paper at ICLR 2025
VQA
TQA
MAE
Rank Corr.
MAE
Rank Corr.
VQA
TQA
es.Al
1.721a
99.8
2.32144
98.6
evaluated
CvCV
38822
92.6
38
97.0
C.LG
16gi0
97.4
2.
97.9
predicted acouracy
80
non-evaluated
80
R.RO
3.311s
98.6
35.11
98.5
eess.5
1.7g0s
99.5
2.821.5
96.8
60
60
2.6.1.4
98.9
188.1.2
97.0
physies.ap-1
2.7
7..23.
40
40
pysicchio-ph
2.1=1.6
96.3
3.3.2.1
96.0
MAE 1.4pp
MAE = 2.3pp
dayaic.
2.4g1.
30.21.
1.00
0.99
opysics.ppiics
2.9.2.1
98.9
3.8-2.0
95.3
BM
2.0g1.1
2.
@-bio.CB
3.2
96.9
11.5
98.6
100
100
q-hio GN
2.
97.4
ground truth accuracy (%)
ground truth accuracy (%6
Q-bio. TO
2.9116
96.8
3.0+225
96.6
Figure 13: Performance prediction results our efficient re- re-evaluation method on v4 when re-evaluating models.
VQA
TQA
VQA
TQA
mm3 m=5
Spearman correlation (p)
m=12
95
95
85
85
2
2
80
m=5
m-
75
75
v2
v3
vd
v2
v3
v4
V2
v4
V2
V4
LiveXiv version
LiveXiv version
LiveXtv
LiveXiv
VQA
TQA
100
Spearman cornelation (p))
m
-
m=12
95
95
4-
85
85
2.
2-
m=8
75
75
v4
vf
V2
V2
LiveXiv version
LiveXiv version
LiveXty version
LiveXiv version
This figure is a reproduction of Figures 4 (first row) and 5 (second row) considering the full LiveXiv v3 and not only questions generated by GPT4o and Claude-Sonnet. In this case, that the prediction performance of IRT can deteriorated to distribution shifts the data.
Figure 14:
100
100
evaluated
predicted s.ccrrcy
60
60
2.1pp
3.1pp
20
P. -
20
A 0.96
20
40
60
100
20
40
60
80
100
truth
Figure 15: Performance prediction results of our efficient re- re-evaluation method on vl from vO when re-evaluating models, We can see that even a large temporal gap, our method managed to predict accurately.
29

---

Published as a conference paper at ICLR 2025
MM-LiveBencch
100
predicted accuracy (%)
re evaluated non-re-eve con-re-evaluated
60
MAE 1.9pp
P: - 0.96
20
40
80
100
ground truth accuracy (%6
Figure 16: Our efficient evaluation method applied to MM-LiveBench. We check that method could be successfully applied in this other context, here we convert MM-LiveBench to multiple choice questions and apply our efficient evaluation method between MM-LiveBench versions. The results for MM-LiveBench are optimistic, showing our method predict the results between its version accurately.
30

---

Published as a conference paper at ICLR 2025
A.6 PROMPT TEMPLATE FOR QA GENERATIONS
This is figure
from a scientific paper with the following caption: (text desc), Please describe the image in as much details as possible. For all the details you are confident about include everything you see, and be specific possible such as existing numbers, describing objects, attributes
Figure 17: Prompt template for general detailed caption.
Compositional reasoning defines the understanding of attributes, relations and word order significance. good vision -language model should be able to accurately answer composition reasoning questions about an image. Your task is to fool vision -language model by generating challenging compositions questions about the figure. Given the image and the description you generated:
reasoning
generate in_questions) diverse and challenging compositional reasoning questions which a vision-language model would incorrectly answer. For each question include the following: - A compositional reasoning question correct answer
hard negative options. Each negative option should differ only subtly from the correct answer but still be clearly incorrect given the image, and the question. The geal is for a vision-language model to choose the negative option over the positive option when you asked to answer the question in binary multiple choice format. Only include questions you are confident your answer and make sure there is indeed only a single correct answer and the others are false answers. Format your response as a string in the format [["Q" "quessioon,,, scorscorrect answer >, "n1" "nn":cteaatvve option 1>, "n2"; <negative option 2>, ...ll.
Figure 18: Prompt template for visual -anstion.nnswerin..
31

---

Published as a conference paper at ICLR 2025
Document and table understanding defines the understanding of values, metrics and perform arithmetic operations over numerical values and commonsense reasoning good language model should be able to accurately answer peommnnsenserrassniinn / arithmetic manipulation] questions from a given table. Your task is fool a language model by generating challenging table (comonsense_reasoniin questions about the table. Given the table: (table_content) Generate (n_questions) diverse and challenging ccommnnsense_reasoniiii / arithmetic manipulation] questions on the table questions which language model would incorrectly answer FFor each question include the following: answer 3 hard negative options. Each negative option should differ only subtly from the correct answer but still be clearly incorrect given the figure, caption and the question. The goal is for a language model to choose the negative option over the positive option when you asked to answer the question in binary multiple choice format. Only include questions you are confident your answer and make sure there is indeed only a single correct answer and the others are false answers. Format your response as a string in the format = <corect answer >, "nl":<negative option 1>, "n2"; <negative option 2>, ...Il.
/ arithmetic manipulation
question
correct
Figure 19: Prompt template for table question-answering.
Think step by step before answering.
For the given image and question: (question)
write only the words yes or no if think the option (correct_answer
is indeed the correct answer out of (options) for this question?
Figure 20: Prompt template for agreement filtering.
32

---

Published as a conference paper at ICLR 2025
You are
an insightful assistant, for the questicn/options pair provided by the user, pick a question category from the list below: Question category:
attribute: the question asks about the presence or visibility of an attribute of object (e.g. "What is the color of circles in plot (a)?" "(A. Blue, В. White, C. Green, D. Red]") reasoning: the question
asks about understanding the figure (e.g "What is the object inside the red box?" "[A. Bottle, B. Table, C. Tree, D. Nothing]")
localization: the question asks about
the presence visibility (e.g "On which subplot does the scatter is the most spread?" "[A. Top-Left, B. Bottom-Right, C. 'Middle-Left',
specific location in the image
"Top-Right]")
reading: the question asks about reading
some text from the figure (e.g "What name of the method presneted
green line?" "[A. GPSK, B. FDAH, C. TQWA, D.Ours)") arithmetic: the questions asks about mathematical arithmetic
of numbers (e.g if the maximium accuracy of SIFT would be doubled? what would be the value?" "[A. 2, B. 4 C. 100, D. 50]) data
analysis: the question asks about understanding of a graph (e.g, "Which values intersect at T=2?" "["A. N1, B. N2, N3, D. N4]")
Respond with JSON object with the following format: ("Question category": "'categozy))
Figure 21: Prompt template for question categories analysis.
33


Reviews:
[{"review_id": "Qqdlbl3A5j", "reviewer": "Reviewer_P23K", "summary": "This paper introduces LiveXiv, an automated multi-modal live benchmark system that generates Visual Question-Answering (VQA) pairs from ArXiv papers. The work is motivated by the critical issue of test set contamination in current LMM evaluations, as models increasingly train on web-scraped data that may include benchmark test sets. The authors propose an evolving benchmark using newly published scientific papers. Besides, they introduce an efficient evaluation method to make continuous assessment practical. Multiple open and proprietary LMMs are benchmarked.", "questions": "1. How does the system ensure diversity in question types and difficulty levels?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clear and good: this work addresses the critical issue of test set contamination in current LLM evaluations, as models increasingly train on web-scraped data that may include benchmark test sets.\n\n2. This paper proposes a scalable live benchmark without any human involvement, automatically drawing data from online scientific manuscripts, generating multiple VQA pairs, and filtering these questions to reduce errors.\n\n3. An efficient evaluation methodology is introduced, offering significant computational savings.", "weaknesses": "1. The paper heavily relies on one LLM, i.e., Claude for question filtering. While filtering aims to reduce errors, potential biases may arise from Claude being the only model used for answer verification. This can introduce an inherent bias in the benchmark, potentially skewing results to favor Claude's architecture and underlying methodologies. The superior performance of Claude-Sonnet shown in Table 1 may be partially attributed to the fact that Claude-Sonnet itself verified these questions, potentially making them more aligned with its capabilities. The authors might address this by including additional, distinct filtering mechanisms to mitigate model-dependent biases.\n\n2. The paper suggests that only certain types of scientific data (e.g., block diagrams, qualitative visuals, charts) are categorized for question generation. While this is effective for consistency, it risks oversimplifying the diversity of visual data in scientific publications. Expanding the types of visuals and including more complex multi-modal question types (e.g., cross-referencing multiple figures) would make the benchmark more challenging and comprehensive.\n\n3.The benchmark is restricted to multiple-choice format questions, which limits the evaluation of models' true generative and reasoning capabilities. Including free-form answering would provide a more comprehensive assessment of model understanding and better reflect real-world applications.\n\n4. The benchmark evaluation would benefit from more comprehensive quantitative comparisons with established datasets like DocVQA and ChartQA. Such comparisons would better demonstrate LiveXiv's advantages over static benchmarks and more clearly illustrate how it addresses test contamination issues and aligns with human preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "voF2DmicAi", "reviewer": "Reviewer_fXkv", "summary": "This paper propose a new live multimodal benchmark on scientific ArXiv papers. The authors also designed an efficient evaluation approach to rank the models. The authors verified their evaluation metric to be aligned with human evaluation with a small variance (<2.5%).\n\nThe main contributions are:\n(1) A pipeline to generate and filter scientific paper VQA and TQA data using a document parsing pipeline (preprocessing), gpt-4o (generation), and Claude (filtering).\n(2) An evaluation pipeline that, when evaluating a new model on the latest data, the pipeline re-evaluate old models on a subset of the latest data with a few old models for performance comparison.", "questions": "1. How do the authors make sure the diversity of the VQA is also scalable? For example, if gpt-4o keeps generating many questions like \"what is the number in the third row and second column of the table?\", which will only evaluate a fixed OCR capability regardless of which timestamp the question is generated from.\n2. Why the GPT-4o performs very badly in Table 1 while it is the model to generate the questions and answers?\n3. How does the domain impact the evaluation scores? Could the authors show how models perform on different categories (as shown in Table 3) in different domains and see the correlation?\n4. Despite I like the benchmark, could the authors clarify their innovation in method design beyond (1) taking an efficient evaluation strategy from nlp domain and apply it to vlm domain; (2) synthetic evaluation data generation using one proprietary api for generation and another for filtering which has been applied in nlp broadly even using the same apis? The pdf parsing is also a basic step in previous works  on paper QA. Could the author provide a clear statement of their key technical innovations and how they differ from or improve upon existing approaches and include a discussion of how combining these existing techniques in a novel way for this specific application represents an innovation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The author propose to have a live benchmark is a promising direction to mitigate the impact of data contamination on holistic evaluation. It is challenging to maintain a live and scalable benchmark, and the author proposed several methods to resolve the challenges: a question-answer generation pipeline grounded on a structured pdf processor and powerful proprietary VLMs for generation and filtering, and an efficient evaluation algorithm to make model comparison affordable.\n2. The proposed methods at both data curation stage and the evaluation stage have been shown to be effective in previous works on language-only datasets. The authors extend the thoughts to the multimodal domain and show these methods are still effective.\n3. The human study shows the automatic data filtering pipeline is effective in removing annotation errors during data curation.", "weaknesses": "1. All questions are generated by gpt-4o, which may introduce issues such as the lack of diversity in questions.\n2. The authors only did human study for the question-answer filtering, while they did not verify if the final ranking of models is aligned with human perception and other established benchmarks. \n3. As the author mentioned, the data curation replies on proprietary models, which makes the benchmark prone to bias and low reproducibility. \n4. The question type is limited to multi-choice answering, while it could be more interesting if the authors could extend it to a broader categories of tasks such as long-form generation.\n5. The novelty of the method: the authors should clarify their novelty by comparing with existing works that ask a powerful model to generate evaluation benchmark. For example, [1] also parses papers and ask an LLM to generate question for long-form paper question-answering. These works could be extended to VLM domains by swapping out the LLM to VLM and keeping add live new data to the benchmark in a straightforward way.\n\n[1] KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tiPypdA3DQ", "reviewer": "Reviewer_LXM8", "summary": "This paper introduces an evolving benchmark derived from arXiv papers, utilizing LLMs to automatically identify and generate visual-based question-answer pairs from the multi-modal content within manuscripts. To manage the growing dataset efficiently, an evaluation method is proposed that avoids the need to assess every method as the dataset expands.", "questions": "In Line 235, which LLM do you use here?\nHow do you select the test set in Table 1?\nLine 458, which 5 models are selected by the algorithm exactly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A large-scaled dataset that could be very useful for people in the VQA domain. \n2. I like the idea of 'evaluating the dataset in a dynamic way'. As most of the datasets get contaminated as training data increases by time. \n3. A rather comprehensive evaluation of the dataset, covering most state-of-the-art LMMs.", "weaknesses": "1. A limitation of this work is that it relies on OCR to extract only tables and figures, using only their captions as input to create the dataset. This approach ignores the broader context of the paper, which may contain valuable information. For instance, many tables have generic captions like \"Main experimental result of our proposed method,\" which may lead GPT-4 to generate only simple structure-related questions, such as \"Which column has a value of **,\" rather than more insightful content-based questions.\n\n2. As the dataset is entirely LLM generated, it's necessary to perform some human evaluation to avoid hallucinations and errors. Even though it's not feasible to perform such evaluation on the entire dataset, at least a small portion could be sampled and more detailed manual check could be performed beyond the simple one briefly described in Line 255.\n\n3. I'm not sure about the efficiency and robustness of the evaluation model, even though I appreciate the idea, given the diverse styles and topics of newly added arXiv papers. Previous LMMs may become outdated, exhibit biases, and struggle to handle shifts in data distribution effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w2Y7YskKkl", "reviewer": "Reviewer_VmiA", "summary": "The study presents LiveXiv, a multi-modal live VQA benchmark designed to evaluate the capabilities of LMMs on scientific content from ArXiv papers. The benchmark addresses the issue of test data contamination by continuously evolving and utilizing the latest data, thereby providing an up-to-date assessment of model performance. LiveXiv automatically VQA pairs from figures, charts, and tables in scientific manuscripts without human intervention, leveraging advanced LMMs like GPT-4o and Claude. The study also introduces an efficient evaluation approach that estimates the performance of all models on the evolving benchmark by evaluating only a subset, thereby reducing evaluation costs.", "questions": "Please address my concerns above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) It is a innovative approach to creating a live, contamination-free benchmark that is both scalable and efficient. \n2) The use of advanced LMMs for automatic generation and filtering of VQA pairs is commendable, as is the development of an efficient evaluation method inspired by Item Response Theory, which reduces computational overhead. \n3) he benchmark's design allows for the expansion into new domains, potentially increasing its relevance and applicability.", "weaknesses": "1) The scope of work presented may not be substantial enough for a benchmark paper, as the number of samples and the diversity of domains covered could be expanded. \n2) While the idea of a contamination-free benchmark is valuable, the study's focus on Knowledge-VQA tasks narrows its applicability and relevance. Data contamination is a broader issue that extends beyond Knowledge-VQA tasks. \n3) The reliance on proprietary LMMs for the benchmark's operation introduces potential variability and a lack of control over the evaluation process\n4) The efficient evaluation method shows promise, its effectiveness and reliability over time and across different versions of the benchmark need further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)