LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content

## Abstract

Abstract The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks.
However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated.
To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific Ar Xiv papers.
LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate
visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables.
Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost.
We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models’ true abilities, avoiding contamination.
Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal ( < 2.5 % absent percent 2.5 <2.5\% < 2.5 % ).
Our dataset is available online on HuggingFace , and our code will be available here .

## 1 Introduction

Figure: Figure 1: Static benchmark contamination. As training data increases, the risk for test set contamination grows and static benchmarks becomes saturated, reflecting falsely improved capabilities.
Refer to caption: x1.png

The internet, with its vast and ever-growing repository of information, serves as a rich data source for training Large Language Models (LLMs) \citepgpt3, openai2023gpt4, vicuna2023, t5, llama, touvron2023llama, dubey2024llama and Large Multi-modal Models (LMMs) \citepopenai2023gpt4, llava, llava-ov, minigpt, minigptv2, flamingo, clip.
This diverse and continuously updated data fits precisely the need to cover varying knowledge in scale in the training data.

Training on such data enables the models to achieve superhuman performance across a wide range of tasks on multiple common benchmarks  \citepmme,yue2024mmmu,li2024seed,liu2023mmbench.

We hypothesize that a portion of LLMs’ reported improvements are due to data contamination (Figure 1) and pose the following question: *To what extent does the potential for test set contamination during large-scale training affect our perception of the abilities of LMMs?*

One possible way to safeguard against the contamination of static benchmarks is to design a live benchmark that can continuously harness data from the web and turn it into an ever-evolving benchmark to test the abilities of these models.
A live benchmark may be used in one of the following ways:
(a) Expand the dataset over time and evaluate the models’ overall knowledge over all collected data, while taking into account that the data might be contaminated.
(b) Use only the latest version to assess model capabilities while keeping data contamination risk minimal.
While we focus on (b), we share key properties of our efficient evaluation method that is applicable to both cases.

Although, a live benchmark is a promising direction, it still comes with its fair share of challenges.
A live benchmark should ideally be updated frequently, consistently, and automatically, i.e. it should be able to scrape the data from the web and formulate it into a benchmark for automated evaluations.
Furthermore, as the benchmark is ever-evolving, each time a new version arrives, all the participant models need to be re-evaluated, making the update procedure prohibitively expensive both in time and compute. This requires a methodology for efficient evaluation of these models on a continuously updating benchmark.
Such a methodology should ease the computational burden of evaluating all the models on each new version of the dataset and reduce the logistic overhead of maintaining inaccessible old models.

Figure: Figure 2: We propose LiveXiv, a new method for generating Live multi-modal dataset for Visual Question-Answering based on ArXiv content. Our pipeline automatically generates scalable and reliable questions along with an efficient evaluation method to reduce the computational and logistic overheads required for continually evaluating past and present models on new versions of the dataset.
Refer to caption: x2.png

In this work, we take a step in this direction and propose LiveXiv – a novel fully automated multi-modal live benchmark that focuses on scientific domains.
LiveXiv starts with scraping category-specific (e.g. cs.CV, eess.SY, q-bio.BM, etc.) manuscripts from ArXiv and generates visual question answers from figures, charts, and tables present in these manuscripts through a capable multi-modal model, namely, GPT-4o.
As it is challenging to directly feed information-rich PDF documents to GPT-4o, as a pre-processing step, we extract relevant information from the papers by processing it with a structured document parsing pipeline \citepDeepSearchToolkit to obtain pertinent information like placements of figures, charts, tables, and the text in the captions or in the tables.

This information is used to extract, e.g. by cropping, relevant information from the manuscripts, which is fed to GPT-4o to generate visual questions and answers.
Although very capable, GPT-4o is still prone to errors, e.g. due to hallucinations, and may even generate questions that can be answered without visual information.
Thus to mitigate these issues, we add an extensive filtering stage that automatically filters questions requiring only textual information to answer them, and reduce hallucinations through obtaining agreement about the generated questions with another capable multi-modal model, namely, Claude.
After the extensive filtering, we obtain a large corpora of VQA pairs which are incorporated into our LiveXiv live benchmark.

Over time, the benchmark is expected to grow, either in the size of the dataset or the amount of models to be evaluated, which increases the required resources for evaluation.
Moreover, comparing a new model to existing models at different times requires re-evaluating the existing models over the latest version of the dataset, which can cause additional overhead for continuous evaluation and comparison to prior works.
To make the evaluations on LiveXiv feasible, we take inspiration from \citetpolo2024tinybenchmarks,polo2024efficient and propose a method to approximate the performance of the existing models in new versions of LiveXiv just by re-evaluation small portion of them. Figure 2 provides a conceptualized overview of our approach.

We summarize our contributions as follows:
(a) We propose a scalable live benchmark without any human in the loop that automatically harnesses data from online scientific manuscripts, generates multiple VQA pairs, filters these questions to reduce errors, and formulates them in the form of a benchmark to test the evolving landscape of LMMs;
(b) We introduce an efficient evaluation pipeline that requires LMMs to be tested only on a fraction of the data to infer its performance on the latest version of the benchmark,
reducing the overall needed evaluations by $70\%$;
(c) We benchmark multiple open and proprietary LMMs on the first version of our benchmark highlighting its challenging nature and providing interesting insights about the models’ behavior when evaluated on less contaminated data.

## 2 Related Works

Large multi-modal Models (LMMs).
LMMs have shown significant advancements in enabling billion-parameter scale LLMs to perform multi-modal tasks such as image captioning, visual reasoning, and visual question answering. Academia and industry have endeavored to develop LMMs targeting the multi-modal competence of advanced proprietary models like GPT4o \citepopenai2023gpt4 and Claude \citepclaude.
InstructBLIP performs instruction tuning on the pre-trained BLIP-2 \citepblip2 covering 11 vision-language tasks. The LLaVA series models \citepllava, llava+, llava-next, llava-onevision develop the pipeline of collection of instruction-following data and visual instruction tuning with enhanced vision capabilities. The internLM-XComposer (IXC) series \citepdong2024internlm,dong2024internlm4khd target free-form vision-language composition and multilingual comprehension. Models from Idefics release \citepidefics2,idefics3 benefit from the massive collection of instruction-following data from over 50 vision-language databases, enhancing capabilities of OCR, document understanding, and visual reasoning.
In this work, we include 17 top-performing LMMs in our multi-modal live benchmark LiveXiv, covering both open-sourced and proprietary representatives.

Static evaluation benchmarks for LMMs.
Most existing LMM benchmarks offer static evaluation with fixed questions and answers \citepmme,yue2024mmmu,li2024seed,liu2023mmbench,huang2024conme,lin2024comparison,zhang2024lmms. MME \citepmme offers evaluation of perception and cognition on $14$ tasks and MMMU \citepyue2024mmmu includes 11.5K questions from college exams, quizzes and text books from six major disciplines. Although these benchmarks cover a large variety of multi-modal domain knowledge, evaluation on them is faced with two hazards: the excessive evaluation cost and test data contamination. In this work, we tackle both challenges by proposing a suite that enables efficient evaluation on a contamination-free live benchmark.

Contamination-free benchmarks.
As large foundation models like LLMs and LMMs are trained on combined sources of tremendous amount of web data or repurposed version of existing open-sourced datasets, there is a high risk of overlap between training data and samples from evaluation benchmarks.
Reported evidence and analysis show impact of data contamination on evaluation benchmarks for LLMs \citepwei2023skywork, zhang2024careful,cobbe2021training,roberts2023cutoff,jain2024livecodebench and LMMs \citepchen2024we, indicating the significance of contamination-free evaluation benchmarks.
For LLMs, LMSys Chatbot Arena \citepchiang2024chatbot and AI2 WildVision \citeplu2024wildvision create a user-focused platform that provides contamination-free environment for proper evaluations. However, it is expensive to collect tens of thousands of human preferences on the compared language models. Furthermore, Seal Benchmark \citepsealbench proposes private questions paired with human evaluations.
\citetsrivastava2024functional update the questions in the MATH dataset \citephendrycks2021measuring by changing numbers in the math questions.
LiveBench [livebench] collects frequently updated questions from diverse information sources e.g.math competitions, arXiv papers and news articles and more challenging versions of existing benchmark tasks.
Concurrently, LiveCodeBench \citepjain2024livecodebench contributes a live benchmark on broader code-related capabilities. Note that these datasets focus on language data only.

For LMMs, Vibe-Eval \citeppadlewski2024vibe and LLaVA-Wilder \citepli2024llava perform contamination check on the collected samples that reflect real-world user requests. Most related to our work, the LMMs-Eval LiveBench \citepzhang2024lmms collects images from sources of new websites and online forums and employs proprietary LMMs for design and revision of questions.
However, the LMMs-Eval LiveBench requires human manual verification of questions which impedes the scalability. Furthermore, it contains only open-ended questions that require LMM-as-a-judge which is time-consuming, susceptible to judge biases, and difficult to scale.
In comparison, our LiveXiv constructs a fully-automated data collection pipeline which generates multiple-choice questions which are challenging to the top-performing LMMs.

Efficient benchmarks.
With the increasing amount of tasks and samples in current benchmarks, evaluation of the full suite is time-consuming and cost-intensive. Efforts are underway to develop efficient benchmarks that reduce computation costs without sacrificing reliability.
For LLMs, \citetperlitz2023efficient proposed the first systematic study of the effects of language model benchmark designs on reliability and efficiency, and applied efficient benchmark practices on the HELM benchmark \citepliang2022holistic, leading to $\times$100 computation reduction with minimal loss on reliability.
Lifelong benchmarks \citepprabhu2024lifelong has an ever-expanding pool of test samples for the categories in CIFAR10 \citepcifar and ImageNet \citepimagenet; to make this design economically feasible, it reuses past model evaluations on a sample set through dynamic programming to enable efficient evaluation of new incoming models, drastically reducing the evaluation cost. Most related to our work, tinyBenchmarks \citeppolo2024tinybenchmarks and PromptEval \citeppolo2024efficient propose using Item Response Theory (IRT) \citeplord1968statistical to estimate the performance of LLMs on unseen samples, making efficient evaluation possible by only conducting a small fraction of the total number of evaluations. Inspired by the last two works, we leverage IRT to estimate the performance of older models in new batches of data. More specifically, at each version of LiveXiv, we choose a small core set of models ($\leq 5$) previously added to the leaderboard and re-evaluate them on the new data. Depending on their responses to the new samples, we estimate the performance of the remaining old models on the new benchmark version.

## 3 LiveXiv

At a higher level, our automated LiveXiv is created by first obtaining the domain-specific scientific manuscripts from ArXiv at any given timestamp.
Then, to obtain pertinent information from the manuscripts, we pass them through a structured document parsing pipeline and then generate visual question answers through a capable LMM (Section 3.1).
However, the generated questions can contain errors due to hallucinations or might be too straightforward to answer.
Thus, to mitigate these issues, we offer an extensive filtering stage (Section 3.2).
To evaluate the benchmark, we propose an efficient evaluation framework to infer the overall performance on the benchmark using only a small subset of evaluations, making the evaluations extremely resource-efficient (Section 3.3).
The data acquisition and filtering steps are schematically visualized in Figure 3.

### 3.1 Data Acquisition and VQA Generation

We start with the data acquisition phase, then pre-process the data
to obtain the required metadata (e.g. placements of figures, captions, etc.), and then generate the first iteration of VQA from the multi-modal data (figures and tables) from the manuscripts.

Figure: Figure 3: Our live dataset generation consists of several stages. We first extract the images and their corresponding metadata (i.e. captions and table contents), then we classifying the figures into categories using meta-prompting. All the extracted data is then fed to GPT4o to generate multiple questions-answer pairs per image. Since generative models are prone to errors, we apply several filtering steps, using an LLM and LMM to ensure that our dataset is truly multi-modal and reliable.
Refer to caption: x3.png

Data Acquisition:
At any given timestamp, we begin by acquiring only ArXiv papers which have non-exclusive license to distribute from predefined domains such as Computer Science (cs.AI, cs.CV), Electrical Engineering (eess.SP, eess.SY), and Quantitative Biology (q-bio.BM, q-bio.GN).
However, these manuscripts contain a lot of information that might not be necessary for the task of VQA data generation.
Thus, to extract pertinent information we require a pre-processing step.

Pre-processing:
The downloaded PDFs undergo a structured document parsing pipeline using the DeepSearch toolkit \citepDeepSearchToolkit, which extracts a comprehensive layout of each document, including the positions of figures, tables, captions, and other elements.
This structured layout forms the basis for extracting the multi-modal data required for subsequent tasks.
To enrich the dataset with additional metadata not captured by the parsing pipeline, we employ a meta-prompting approach with CLIP \citepradford2021learning, similar to the method used by \citetmirza2024mpvr.
Specifically, we classify the figures into three distinct categories: Block Diagram, Chart, and Qualitative visual examples which facilitates a more granular, domain-specific evaluation of LMM performance.

VQA Generation:
For Visual Question Answering (VQA), we construct pairs of figures and their corresponding captions, and for generating VQA from the data present in the tables, we obtain (e.g. crop) images of tables accompanied by their corresponding data.

The VQA process involves two steps using GPT-4o.
First, we input the figure and its caption to GPT-4o to generate a detailed description of the figure, employing a Chain-of-Thought (CoT) approach \citepwei2022chain.
Next, the detailed description and figure are fed back into GPT-4o, with prompts adapted from ‘ConMe’ \citephuang2024conme to suit our scientific use case, enabling the generation of relevant VQA questions.
For questions from the tables, we utilize the table’s content directly, presenting both the image of the table and its data in markdown format to GPT-4o to produce questions that require common-sense reasoning and data manipulation.
The automated nature of this process ensures a robust and comprehensive evaluation framework for LMMs, tailored to scientific literature specifics.
Detailed prompt templates can be found in Appendix A.4.

### 3.2 Filtering Phase

Even though GPT-4o is powerful and has been reported to outperform humans on many different benchmarks \citepopenai2023gpt4, still it is prone to errors and sometimes can even result in VQA pairs that are answerable without requiring the visual information.
Thus, to ensure that the benchmark remains competitive and also has minimum errors,
we propose an extensive automatic filtering step.
At a higher level, the filtering phase consists of two main parts, each designed to mitigate a separate issue that can arise due to the automatic dataset generation.

Blind test with an LLM: To ensure that the generated VQA pairs are *truly* multi-modal, we pass them through a Large Language Model (LLM) without providing any associated images or image descriptions.
This process, referred to as a *blind test*, aims to identify questions that the LLM can answer correctly even in the absence of visual context, indicating they are not truly multi-modal.
To ensure robustness, this blind evaluation is repeated multiple times to eliminate any potential *lucky guesses* by the LLM.
Questions that are consistently answered correctly by the LLM are filtered out, resulting in the removal of approximately $30\%$ of the generated questions.
This step ensures that the remaining questions in the dataset are inherently multi-modal and cannot be answered solely based on linguistic context.
The filtered dataset thus represents a more challenging benchmark for evaluating multi-modal capabilities of vision-language models.

Agreement between disjoint models:
Generative models, including LMMs, are prone to hallucination, where the model generates incorrect or not grounded information.
In our case, these hallucinations can lead to erroneous VQA pairs.
To address this issue, we introduce an additional filtering step.
All questions that pass the initial “blind test” are reviewed along with their generated answers by a different LMM, in this case Claude-Sonnet \citepclaude, which is provided with the image, question, and the ground truth answer which were all generated by GPT-4o.
This second model is asked to either agree or disagree with the generated answer, considering the visual context.

We point out that agreement between models is a nuanced process; incorporating more models to validate answers may lead to the exclusion of difficult questions, thereby diluting the *difficulty* of the dataset.
Therefore, we limit this validation step to models with comparable performance to the generation model (i.e. GPT-4o).
Our preliminary manual evaluation on a subset of the dataset indicates that this agreement step significantly reduces the proportion of incorrect ground-truth (GT) questions, with a reduction of $38.5\%$, while minimally impacting the retention of high-quality question-GT pairs, with only a $6.15\%$ removal of valid pairs.
This refinement ensures that the final dataset is both challenging and accurate for the evaluation of LMMs’ multi-modal reasoning capabilities.
The generated corpus of data is ready to be incorporated into  LiveXiv and can be updated automatically without any human intervention.

### 3.3 efficient evaluation

Since LiveXiv is a dynamic benchmark, evaluation can be costly: ideally, whenever a new version of the benchmark is released, all models must be re-evaluated on the updated data, which can pose an engineering challenge and become computationally expensive when handling dozens of models. In this section, we describe our approach to efficient evaluation, which avoids re-evaluating all models at each step, making LiveXiv’s maintenance economically feasible. Our idea is based on Item Response Theory (IRT) \citepcai2016item, van2018handbook, lord1968statistical, polo2024efficient,polo2024tinybenchmarks, a collection of statistical models traditionally used in psychometrics and educational assessment. We briefly give some background on IRT and detail how we use it for our evaluations.

#### 3.3.1 Item Response Theory (IRT)

We use the IRT model to predict the probability of a certain LMM $i$ answering correctly on a sample (question) $j$. In mathematical terms, let $Y_{ij}\in\{0,1\}$ denote the correctness on sample $j$ when responded by LMM $i$:

$$ $\displaystyle Y_{ij}\sim\text{Bernoulli}(\mu(\theta_{i},\beta_{j})),$ $$

where $\theta_{i}$ is an LMM-specific parameter, $\beta_{j}$ is a sample-specific parameter, and $\mu$ is a function that maps those parameters to the probability of correctness. In this work, we follow \citetpolo2024efficient and assume the parameters live in the real line while $\mu$ induces a logistic regression model. In more detail, we assume

$$ $\displaystyle{\mathbb{P}}(Y_{ij}=1;\theta_{i},\beta_{j})=\frac{1}{1+\mathrm{ exp}[-(\theta_{i}-\beta_{j})]}.$ (1) $$

Here, $\theta_{i}$ can be interpreted as the skill level of LMM $i$ while $\beta_{j}$ is seen as the hardness of sample $j$. By equation 1, if $\theta_{i}$ is much greater (resp. smaller) than $\beta_{j}$, then the probability ${\mathbb{P}}(Y_{ij}=1;\theta_{i},\beta_{j})$ will be close to one (resp. zero). This version of the IRT model is known as the Rasch model \citepgeorg1960probabilistic,chen2023statistical, and it is widely used in fields such as recommendation systems \citepstarke2017effective, educational testing \citepclements2008development, and evaluation of language models \citeppolo2024efficient. Moreover, it has a similar formulation to the popular Bradley-Terry model \citepbradley1952rank used in Chatbot Arena \citepchiang2024chatbot, a popular and dynamic benchmark for AI-powered chatbots. We fit the Rasch model using maximum likelihood estimation as in \citetchen2023statistical and \citetpolo2024efficient.

#### 3.3.2 Efficient evaluation with IRT

We can estimate old model scores on new data without reevaluating those models. Let $\mathcal{I}_{t}$ and $\mathcal{J}_{t}$ represent sets of non-negative integers corresponding to LMMs and samples at time $t\geq 0$. We assume that $\mathcal{I}_{t}\subseteq\mathcal{I}_{t+1}$ since the set of available models does not shrink over time, and $\mathcal{J}_{t_{1}}\cap\mathcal{J}_{t_{2}}=\emptyset$ for $t_{1}\neq t_{2}$ because samples are not repeated across different time steps. Let the set of evaluated models at time $t$ be denoted by $\hat{\mathcal{I}}_{t}$. For $t>0$, we assume that $\mathcal{I}_{t}\setminus\mathcal{I}_{t-1}$ is a proper subset of $\hat{\mathcal{I}}_{t}$, meaning that all newly introduced models are evaluated on the new batch of samples along with some previously existing models. At $t=0$, we assume that $\hat{\mathcal{I}}_{t}=\mathcal{I}_{t}$, meaning all models are evaluated on all samples. Furthermore, we assume that $|\hat{\mathcal{I}}_{t}|$ is much smaller than $|\mathcal{I}_{t}|$ when $t>0$ so computing power and evaluation time can be saved.

Our goal at time $t>0$ is to estimate the performance of a model $i\notin\hat{\mathcal{I}}_{t}$ on the set of samples $\mathcal{J}_{t}$, using only the correctness scores $\mathcal{D}_{t}=\{Y_{ij}:(i,j)\in\Omega_{t}\}$, where $\Omega_{t}\triangleq\cup_{t^{\prime}\leq t}\hat{\mathcal{I}}_{t^{\prime}}
\times\mathcal{J}_{t^{\prime}}$. Specifically, we aim to approximate $S_{it}=\frac{1}{|\mathcal{J}_{t}|}\sum_{j\in\mathcal{J}_{t}}Y_{ij}$ by estimating its expectation

$$ $\displaystyle{\mathbb{E}}[S_{it}]=\frac{1}{|\mathcal{J}_{t}|}\sum_{j\in \mathcal{J}_{t}}{\mathbb{P}}(Y_{ij}=1;\theta_{i},\beta_{j}).$ (2) $$

For a moment, let us assume that $\Omega_{t}$ is known. Using $\mathcal{D}_{t}$, we can estimate the skill parameters $\theta_{i}$’s of all models in $\mathcal{I}_{t}$ and the difficulty parameters $\beta_{j}$’s of all samples in $\cup_{t^{\prime}\leq t}\mathcal{J}_{t^{\prime}}$; we denote these estimates as $\hat{\theta}_{i}$’s and $\hat{\beta}_{j}$’s. Finally, we obtain an approximation for equation 2, $\hat{{\mathbb{E}}}[S_{it}]$, by substituting $\theta_{i}$ and $\beta_{j}$’s by their estimates. The estimator $\hat{{\mathbb{E}}}[S_{it}]$ is known as the Performance-IRT estimator \citeppolo2024efficient,polo2024tinybenchmarks.

Now, we provide a method to obtain $\Omega_{t}$ assuming $\Omega_{t-1}$ is given; in summary, we need to decide which models in $\mathcal{I}_{t-1}$ are going to be in $\hat{\mathcal{I}}_{t}$. Our approach to choosing which models are going to be re-evaluated is inspired by the concept of optimal design of tests \citep[Chapter 9]van2017handbook but in which we choose LMMs instead of samples. First, we set a budget $m_{t}$, representing the maximum number of models to be re-evaluated at time step $t$. Second, assuming that the level of difficulty of the new samples $\mathcal{J}_{t}$ is not very different from the ones in $\mathcal{J}_{t-1}$, we choose a set of $m_{t}$ representative samples in $\mathcal{J}_{t-1}$ by ordering $\hat{\beta}_{j}$’s and choosing equally spaced samples, based on their quantiles, from the 5th to the 95th percentiles; this will give us questions with a variety of difficulties, excluding outliers. For example, if $m_{t}=3$ we would choose questions with difficulties in the 5th, 50th, and 95th percentiles. Denote the chosen core set of samples as $\{j_{0},\cdots,j_{m_{t}-1}\}$ and, for each one of these samples $j_{k}$, we choose a model $i$ in $\mathcal{I}_{t-1}$ such that the following Fisher information criterion

$$ $F_{j_{k}}(i)=\mathbb{P}\left(Y_{ij_{k}}=1;\hat{\theta}_{i},\hat{\beta}_{j_{k}} \right)\left[1-\mathbb{P}\left(Y_{ij_{k}}=1;\hat{\theta}_{i},\hat{\beta}_{j_{k }}\right)\right]$ $$

is maximized. The model that maximizes $F_{j_{k}}$ is maximally informative about the parameter of sample $j_{k}$ and, consequently, about all samples with similar difficulty levels in the new version of LiveXiv; this will help us estimate the difficulties of new samples. We note that some models in $\mathcal{I}_{t-1}$ might not be available at step $t$, e.g., due to deprecation; when choosing models, we do not consider them, but note that we can still estimate their performance on the new batches of data. Moreover, the model selection procedure can also take convenience into account; for example, if two models have very similar Fisher information, we opt for the one that is cheaper to evaluate.

In our experiments, we show that re-evaluating 5 models at each step is enough for good performance prediction. When the total number of models is 50, for example, we expect this procedure to save us at least $\times 10$ computing resources considering that we can opt to re-evaluate cheaper models if that does not imply a big loss in terms of $F_{j_{k}}$.

## 4 Results & Analysis

This section presents the results obtained on the first iteration of LiveXiv.
First, we start by describing the experimental settings.
Then, we present the results and finally conclude with a detailed analysis of our dataset.

### 4.1 Experimental Settings

##### Evaluation Protocol:

After the generation of the question-answer pairs from our automated pipeline explained in Section 3, we transform the benchmark to multiple-choice questions.
We resort to the ‘generate’ inference employed extensively by previous works, such as  [li2024seed, huang2024conme, liu2023mmbench]. The model is prompted to choose the letter corresponding to the correct choice and answer with the letter directly.
The output letter is then compared with the ground truth and the accuracy is measured.
We report the average accuracy over all the samples evaluated in Table 1. For ease of assimilation and to obtain insights into what type of data the models flourish at, we provide the results from data generated on tables and figures separately.
The data generated from figures is labeled as part of Visual Question and Answers (VQA) and the data from the tables is labeled as Table Question and Answers (TQA).
Examples for the multiple-choice formulation of the question-answer pairs are added to the Appendix Section A.2.

Size of dataset:
The current version of our LiveXiv consists of 7328 questions on figures, and 9000 questions on tables, both are generated from 250 papers (25 papers from 10 domains).
Overall our first version of the dataset has 16328 questions in total. Thanks to the continual growth in the number of publications in our target domains and the fully automatic nature of our proposed LiveXiv pipeline for benchmark data generation, we will grow LiveXiv by adding an equal-sized large amount of new VQA & TQA data (around 7K VQA and 9K TQA) every month. Such large-scale updates might be significantly more difficult for benchmarks relying on manual data collection for live updates \citepzhang2024lmms.

Models: We extensively evaluate our benchmark by employing a total of $17$ LMMs.
Specifically, we employ 5 models from the LLaVA family of models including LLaVA 1.5-7B and LLaVA 1.5-13B \citepllava, LLaVA-1.6-7B and LLaVA 1.6-34B \citepllava-next and LLaVA One-Vision \citepllava-onevision.
Furthermore, we employ IntstructBLIP \citepinstructblip, InternVL2-2B and InternVL2-8B \citepchen2023internvl, InternLM-Xcomposer2-4KHD \citepdong2024internlm4khd and InternLM-Xcomposer2.5 \citepchen2023internvl, Mantis \citepjiang2024mantis, Phi3v \citepabdin2024phi, Idefics2 \citepidefics2 and Idefics3 \citepidefics3, Qwen2-VL \citepwang2024qwen2 and API models Claude-Sonnet \citepclaude and GPT4o \citepopenai2023gpt4 for our evaluations.
These models have been chosen because of their varying characteristics and strong performance on multiple current benchmarks.
All the models (except GPT-4o and Cloude-Sonnet) are accessed from the huggingface API, which makes our framework modular for an extension to more models as they are being added to the hub in the future.

### 4.2 Experimental Results

**Table 1: VQA and TQA average accuracy across ArXiv taxonomy (the number of samples is in brackets).**
| VQA Accuracy | eess.SP | q-bio.BM | q-bio.CB | cs.AI | eess.SY | cs.CV | cs.RO | q-bio.GN | cs.LG | q-bio.TO | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | (651) | (900) | (840) | (685) | (735) | (720) | (672) | (647) | (844) | (634) | (7328) |
| InstructBLIP-7B | 21.2 | 25.2 | 19.5 | 24.5 | 23.4 | 21.3 | 22.6 | 24.9 | 24.1 | 21.1 | 23.6 |
| LLaVA-1.5-7B | 29.0 | 27.8 | 29.5 | 31.9 | 30.5 | 31.0 | 34.9 | 29.1 | 29.3 | 32.8 | 30.4 |
| LLaVA-1.6-Mistral-7B | 28.1 | 28.7 | 28.6 | 33.9 | 31.0 | 31.4 | 33.3 | 27.0 | 27.9 | 29.5 | 29.9 |
| Mantis-LLama3-8B | 32.3 | 28.6 | 32.7 | 33.7 | 30.2 | 36.9 | 32.6 | 29.2 | 30.8 | 34.9 | 32.1 |
| LLaVA-1.5-13B | 32.6 | 29.4 | 31.5 | 33.4 | 33.2 | 35.9 | 35.7 | 30.6 | 30.0 | 32.2 | 32.3 |
| Idefics2-8B | 35.6 | 38.4 | 35.9 | 40.7 | 40.5 | 38.6 | 39.6 | 30.3 | 36.9 | 38.8 | 37.6 |
| IXC2-4KHD-7B | 33.0 | 36.7 | 33.0 | 40.1 | 35.8 | 45.7 | 44.5 | 37.9 | 35.8 | 36.1 | 37.7 |
| IXC2.5-7B | 46.2 | 46.1 | 48.2 | 53.3 | 50.5 | 45.1 | 47.0 | 47.9 | 49.4 | 46.8 | 48.1 |
| InternVL2-2B | 48.4 | 48.1 | 50.4 | 53.4 | 50.5 | 46.3 | 54.2 | 48.4 | 48.2 | 50.9 | 49.8 |
| LLaVA-1.6-34B | 48.4 | 45.6 | 47.4 | 55.9 | 52.5 | 51.8 | 54.9 | 47.9 | 47.9 | 50.2 | 50.0 |
| Idefics3 | 54.4 | 50.6 | 52.3 | 57.2 | 57.0 | 53.3 | 54.6 | 51.5 | 51.5 | 56.6 | 53.7 |
| LLaVA-OneVision-7B | 53.1 | 49.7 | 51.8 | 57.2 | 52.8 | 57.2 | 57.6 | 51.6 | 51.1 | 59.1 | 53.9 |
| Phi3v | 60.1 | 54.4 | 59.9 | 64.5 | 61.8 | 56.0 | 58.5 | 58.9 | 56.0 | 58.2 | 58.7 |
| GPT-4o | 64.1 | 55.9 | 58.8 | 62.9 | 64.4 | 60.1 | 60.3 | 55.2 | 59.0 | 64.4 | 60.3 |
| InternVL2-8B | 64.5 | 56.9 | 61.4 | 67.0 | 65.3 | 59.9 | 65.3 | 58.4 | 61.4 | 65.6 | 62.3 |
| Qwen2-VL | 68.0 | 62.4 | 71.8 | 67.2 | 69.3 | 63.3 | 64.6 | 64.5 | 63.7 | 71.9 | 66.6 |
| Claude-Sonnet | 78.9 | 72.3 | 77.4 | 77.7 | 78.4 | 69.9 | 74.1 | 72.9 | 76.4 | 75.9 | 75.4 |

Results on entire dataset:
We evaluated 17 large multi-modal models (LMMs) across two prominent tasks, VQA and TQA. Table 1
provides a detailed summary of the performance across both tasks.
One interesting observation is the Claude’s superior performance across the board.
This substantial performance gap suggests that Claude’s architecture and underlying methodologies are particularly well-suited for both VQA and TQA tasks.
The results align with other relatively close benchmarks, DocVQA \citepmathew2021docvqa, ChartQA \citepmasry-etal-2022-chartqa and AI2D \citepai2d, where we see a similar trend, Claude has significantly higher performance over the runner-up models such as Qwen2-VL, GPT-4o and InternVL2-8B. See Table 4 for more details.
However, a notable caveat is that Claude plays an integral role in the question-filtering process, which may introduce a potential bias in favor of questions it is predisposed to solve effectively.
This implies that while Claude’s overall performance remains strong, the evaluation might not fully reflect its robustness to novel or more diverse question types outside the scope of this filtering.

We further observe that newer models, such as InternVL2-8B and Qwen2-VL, consistently outperform older models like LLaVA-1.6 and Idefics2, suggesting rapid advancements in LMM development over the past few months.
This trend highlights the continual improvement in both architecture and training paradigms, leading to better generalization across multi-modal tasks.

Zooming into the domain-specific performance using an ArXiv-based taxonomy, we evaluate each model’s effectiveness in distinct scientific fields such as biology, electrical engineering, and mathematics.
Our results show that certain models, particularly the newer architectures, exhibit a higher degree of robustness across diverse domains, highlighting that the models’ training data might already have potential contamination issues.
Conversely, for VQA, models in the Intern-VL2 and the LLaVA families appear to be more sensitive to domain shifts, performing inconsistently across different scientific areas, as oppose to the more recent models like Qwen2-VL, Claude and GPT4o, see Figures 5, 6 for more details. For TQA, it’s not the case, probably since the questions test more specific skills such a retrieval and arithmetic manipulations, see Figures 7, 8.
This domain-specific sensitivity emphasizes the need for further refinements in LMMs, especially when applied to specialized scientific knowledge domains.
Overall, this analysis not only underscores the ongoing evolution of LMMs but also highlights areas for further investigation, especially concerning model adaptability to diverse content domains and the potential biases introduced by models.

Contamination free effect:
Interestingly, focusing on new data that came after the LMMs were trained, allows LiveXiv to provide a new, contamination-free, perspective on the relative performance ranking between strong LMMs. For example, taking the official results from original publications and computing the average ranking of the LMMs from Table 1 over the long-established DocVQA \citepmathew2021docvqa, ChartQA \citepmasry-etal-2022-chartqa and AI2D \citepai2d benchmarks
, and comparing those to average rankings provided by LiveXiv (Table 1), we observe some significant ranking changes. e.g.  GPT-4o drops almost 2 points and IXC2.5 and IXC2-4KHD drop over 4 points in average ranking, see Table 5 for all the details.

Performance on manually filtered dataset:

**Table 2: Performance change between LiveXiv and a manually verified subset averaged across all evaluated models. LiveXiv is robust, thanks to excessive filtering steps which keep the labeling errors low.**
|  | LiveXiv | Verified Subset | Absolute Avg. |
| --- | --- | --- | --- |
| VQA | 46.734 | 47.273 | 2.336 |
| TQA | 45.101 | 46.028 | 2.105 |

To further verify our proposed automated question-answer generation and filtering methodology and to obtain a measure of errors in the generated data,
we manually verified a subset of $1000$ samples ($500$ for both, VQA and TQA) and evaluated all models on this subset.
Table 2 presents the results for VQA and TQA on the filtered subset.
We see that on average the performance only fluctuates by $2.3\%$ and $2.1\%$ for VQA and TQA when comparing the results obtained by all the models on the entire dataset and the manually verified subset.
These results hint that our automated question-answer generation pipeline and the filtering methodology is quite robust.
Detailed results can be found at the Appendix, Tables 6 and  7.

Figure: Figure 4: Performance prediction results of our efficient re-evaluation method on the hypothetical second (next) version of the LiveXiv benchmark (please see text for details). The table on the left shows the mean absolute error (MAE) and Spearman rank correlation when comparing true and predicted accuracies across individual ArXiv domains, while the graph on the right presents the overall benchmark performance. The results demonstrate that re-evaluating only 5 out of 17 models is sufficient to accurately predict the performance of the remaining models, as well as maintain high rank correlation, validating the effectiveness of our approach.
Refer to caption: x4.png

Efficient evaluations of LMMs: In this section, we empirically validate the effectiveness of our proposed efficient re-evaluation method for LMMs. Dynamic benchmarks like LiveXiv present a challenge in terms of evaluation costs since each time a new version of the benchmark is released, all models should be re-evaluated on the updated data. This process, however, can become computationally prohibitive when dealing with numerous models. Our goal is to demonstrate that by re-evaluating only a small subset of models on the new version of LiveXiv, we can still reliably predict the performance of the remaining models.

For this experiment, we focus on either VQA or TQA (but not both simultaneously) and consider the 10 ArXiv domains. We chronologically split each domain’s papers and samples into training and test sets where the test sets contain $\approx 85\%$ of the more recent papers and samples. The training set represents a hypothetical first version of LiveXiv, while the test set simulates a second version for which we would like to perform the efficient updates. All 17 LMMs are fully evaluated on the first version, but only 5 are re-evaluated on the second version using the model selection methodology detailed in Section 3.3. An IRT model is then fit to the full observed data, and we predict the performance of the non-re-evaluated models on each ArXiv domain and the overall benchmark using empirical versions of equation 2. Figure 4 presents these results, with domain-specific outcomes on the left and full benchmark results on the right. We report both the mean absolute error (MAE) ($\pm$ mean absolute deviation) for the test models when predicting their accuracy and Spearman’s rank correlation across all 17 LMMs on the second LiveXiv version when comparing real accuracy and predicted accuracy. These results suggest that re-evaluating just 5 models is likely to be sufficient for accurately predicting the performance of the remaining models and the ranking of all models.

In Appendix A.3, we present additional experiments to further validate the effectiveness of our method. Specifically, we (i) examine different numbers of re-evaluated models, (ii) show that accuracy prediction error negatively correlates with test sample size, and (iii) test our approach on MM-LiveBench \citepzhang2024lmms. The second point suggests that in real-world applications, we expect our efficient evaluation strategy to achieve lower MAE than those reported in Figure 4, given that test datasets will be larger and unaffected by data splitting.

### 4.3 Analysis and Ablations

To analyze various aspects of LiveXiv we provide an extensive ablation study.
We start by providing an analysis of the results from different models obtained w.r.t the language content partitions, then provide results for different models w.r.t the visual data partitions.

**Table 3: LiveXiv accuracy (%) on different categories of question and partitions averaged over all evaluated models.**
|  | Data Analysis | Reasoning | Attribute | Localization | Reading | Arithmetic | Charts | Block Diagram | Qualitative |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VQA | 46.93 | 47.95 | 46.18 | 41.91 | 47.83 | 46.87 | 44.17 | 52.69 | 48.60 |
| TQA | 46.02 | 63.61 | 68.69 | 51.66 | 59.35 | 35.56 | - | - | - |

Language analysis - performance according to question type.
To discover error slices of models for an analysis of mistakes they commonly make, we classify the questions present in the benchmark into one of the following categories: reasoning, data analysis, reading, localization, attribute, and arithmetic.
To achieve this classification, we employ the Llama-3.1 \citepllama3 LLM and prompt the model with the question and the list of categories to choose for this question.
The prompt is provided in the Appendix Figure 19.
Table 3 summarizes the results for all the models.
We see that the performance of these models on the arithmetic partition is the lowest on average as compared to other partitions highlighting room for potential improvement.
We also provide the detailed results for all models on these partitions for VQA and TQA in Tables 9 and 10 of the Appendix.

Vision analysis - performance according to figure type.
For a more fine-grained analysis of LMM performance on different types of visual data present in our benchmark, we first categorize the data through Meta-Prompting for CLIP, proposed by \citetmirza2024mpvr, in a zero-shot classification setup.
Specifically, we classify the image content into three categories of figures: Block diagrams, Qualitative visual results, and Charts.
We summarize the results in Table 3. Detailed results for each model’s performance can be found in Table 8 in the Appendix.
The results reveal a significant variance in performance across figure types for nearly all models.
In most cases, block diagrams are the most favorable category for models.
However, InternLM-Xcomposer2-4KHD-7B \citepdong2024internlm4khd stands out by achieving the highest accuracy on Qualitative figures.
Overall, Charts emerge as the most challenging figure type on average, suggesting a lack of sufficient examples in the training data for this category.
This kind of analysis can be further expanded to include more categories and discover error slices on which different models struggle so that potential targeted improvements can be designed for these models to mitigate the shortcomings.

## 5 Limitations and Conclusions

Limitations.
LiveXiv relies on capable proprietary LMMs in order to be fully automatic, and with high quality. However, relying on proprietary LMMs is a limitation since we do not have full control over the models, they can change through time and might affect LiveXiv. Nevertheless, we commonly expect them to continuously improve leading to a positive impact on LiveXiv effectiveness.

Conclusions. We propose LiveXiv, an ever-evolving, fully automatic, multi-modal benchmark focused on scientific domains to tackle test set contamination issues and consequently allow a new (contamination-free) perspective on relative ranking of advanced LMMs. We utilize ArXiv, as the data source, carefully and extensively crafting a quality dataset to evaluate LMMs. To significantly reduce the computational and logistical overhead of maintaining the dataset throughout time and models, we propose an efficient evaluation method that can save more than $70\%$ of the evaluated models on each dataset version. Our method can be extended to other archives such as BioRXiv to extend our dataset to new domains. A possible future direction is to evaluate data contamination on past versions of the benchmark, using a comparison of the efficient evaluation vs. a full naive evaluation.

## 6 Ethics Statement

This work introduces LiveXiv, a live multi-modal benchmark for evaluating LMMs using scientific ArXiv papers. By relying solely on publicly available ArXiv manuscripts with proper licenses, we ensure compliance with copyright and distribution policies. The automated generation of Visual Question Answering (VQA) and Table Question Answering (TQA) pairs enables scalable evaluation of LMMs without human involvement, minimizing the risk of human biases in data collection. However, we acknowledge the potential for unintentional biases within the models or dataset itself. Continuous evaluation and refinement are necessary to mitigate these biases and promote the responsible deployment of LMMs in wider applications.

## Appendix A Appendix

### A.1 Analysis & Ablations

Figure: Figure 5: Domain sensitivity according to domains. We visualize the performance of each model across all domains. Clear trends revealed where old models or models with a small LLM are ”under-fitting” and perform worse across all domains. In the middle we have the mid-level models that are sensitive to the domain, indicating their lack of generalization across domain without any additional training. Lastly the newest models (open-source and proprietary) are robust to domain shifts and present a stable performance across the domains.
Refer to caption: x5.png

Figure: Figure 6: LMMs performance based on domain. To complement our analysis form Figure 5 we visualize the statistical properties of each domain. One clear trend is that across all modesl, the performance on cs.CV and cs.AI is the most concentrated, hinting lower variance between models.
Refer to caption: x6.png

Figure: Figure 7: Domain sensitivity according to domains. As opposed to the high variance some models demonstrated in Figure 5, in TQA the tasks and he visual content are more limited thus shrunken the performance variance greatly.
Refer to caption: x7.png

Figure: Figure 8: LMMs performance based on domain. The domains are very similar in their statistical properties showing high variance in performance. This is probably due to wide range of models that differ significantly in their performance.
Refer to caption: x8.png

**Table 4: Average results for relatively close benchmarks (DocVQA, ChartQA and AI2D)**
|  | Average performance |
| --- | --- |
| InstructBLIP-7B | 41.83 |
| LLaVA-1.6-Mistral-7B | 64.33 |
| Mantis | 51.65 |
| LLaVA-1.5-7B | 49.23 |
| LLaVA-1.5-13B | 55.23 |
| Idefics2 | 73.15 |
| InternVL2-2B | 79.07 |
| IXC2-4KHD-7B | 84.00 |
| IXC2.5-7B | 84.90 |
| LLaVA-OneVision-7B | 81.70 |
| Phi3v | 78.23 |
| Idefics3 | 82.10 |
| LLaVA-1.6-34B | 76.83 |
| GPT-4o | 90.90 |
| Qwen2-VL | 86.83 |
| InternVL2-8B | 86.23 |
| Claude-Sonnet | 93.57 |

**Table 5: Average ranking on static benchmarks (ChartQA, DocVQA and AI2D) and LiveXiv. We can see from the ranking difference column that some models have a significant drop (negative difference) in the relative ranking in LiveXiv compared to the static datasets. The gap is highlighting a potential risk of test data contamination when using static (frozen in time) benchmark datasets.**
| Model | Static datasets | LiveXiv | Difference (static - livexiv) |
| --- | --- | --- | --- |
| InstructBLIP-7B | 15.33 | 17.00 | -1.67 |
| LLaVA-1.6-Mistral-7B | 13.67 | 16.00 | -2.33 |
| Mantis | 14.50 | 14.50 | 0.00 |
| LLaVA-1.5-7B | 14.33 | 14.50 | -0.17 |
| LLaVA-1.5-13B | 12.67 | 13.00 | -0.33 |
| Idefics2 | 12.00 | 12.00 | 0.00 |
| InternVL2-2B | 9.00 | 10.00 | -1.00 |
| IXC2-4KHD-7B | 6.33 | 10.50 | -4.17 |
| IXC2.5-7B | 5.00 | 9.50 | -4.50 |
| LLaVA-OneVision-7B | 8.00 | 6.50 | 1.50 |
| Phi3v | 9.00 | 6.00 | 3.00 |
| Idefics3 | 8.50 | 6.50 | 2.00 |
| LLaVA-1.6-34B | 9.33 | 6.50 | 2.83 |
| GPT-4o | 2.33 | 4.00 | -1.67 |
| Qwen2-VL | 3.33 | 3.00 | 0.33 |
| InternVL2-8B | 3.33 | 2.00 | 1.33 |
| Claude-Sonnet | 1.00 | 1.00 | 0.00 |

We provide additional details regarding the ablations:

#### A.1.1 Performance change compared to manually curated subset

We provide a detailed table for VQA performance compared to a manually curated subset of 500 samples.

**Table 6: VQA Performance change between LiveXiv and a manually curated subset.**
| Model | LiveXiv (%) | Manual (%) | Performance Change |
| --- | --- | --- | --- |
| LLaVA-1.5-7B | 29.983 | 28.654 | -1.329 |
| InternVL2-2B | 49.548 | 48.654 | -0.894 |
| LLaVA-OneVision-Qwen2-7B | 52.864 | 56.154 | 3.290 |
| InternVL2-8B | 61.558 | 66.154 | 4.596 |
| LLaVA-1.5-13B | 31.859 | 30.385 | -1.475 |
| InternLM-Xcomposer2-4KHD-7B | 36.801 | 33.654 | -3.147 |
| LLaVA-1.6-34B | 49.196 | 53.269 | 4.073 |
| LLaVA-1.6-Mistral-7B | 29.163 | 26.346 | -2.816 |
| InstructBLIP-7B | 23.216 | 21.346 | -1.870 |
| InternLM-Xcomposer2.5-7B | 47.839 | 50.769 | 2.930 |
| Mantis-LLama3-8B | 32.094 | 28.654 | -3.440 |
| Phi3v | 58.141 | 58.654 | 0.513 |
| Idefics2-8B | 36.851 | 36.731 | -0.120 |
| Claude-Sonnet | 75.942 | 79.615 | 3.673 |
| Qwen2-VL | 66.248 | 71.346 | 5.098 |
| GPT-4o | 60.303 | 60.577 | 0.274 |
| Idefics3 | 52.881 | 52.692 | -0.189 |
| Average (absolute) change |  |  | 2.336 |

We provide a detailed table for TQA performance compared to a manually curated subset of 500 samples.

**Table 7: TQA Performance change between LiveXiv and a manually curated subset.**
| Model | LiveXiv (%) | Manual (%) | Performance Change |
| --- | --- | --- | --- |
| InstructBLIP-7B | 19.1 | 18.5 | -0.6 |
| InternLM-Xcomposer2.5-7B | 49.1 | 45.9 | -3.2 |
| InternVL2-8B | 62.1 | 65.3 | 3.2 |
| LLaVA-1.6-Mistral-7B | 23.5 | 23.2 | -0.3 |
| LLaVA-OneVision-Qwen2-7B | 50.2 | 51.6 | 1.4 |
| LLaVA-1.5-13B | 30.9 | 31.2 | 0.3 |
| LLaVA-1.5-7B | 30.0 | 29.6 | -0.3 |
| LLaVA-1.6-34B | 51.8 | 52.2 | 0.4 |
| Mantis-LLama3-8B | 29.3 | 28.0 | -1.3 |
| Phi3v | 50.2 | 54.1 | 4.0 |
| InternLM-Xcomposer2-4KHD-7B | 42.1 | 41.7 | -0.4 |
| Idefics2-8B | 38.2 | 42.0 | 3.8 |
| InternVL2-2B | 41.5 | 39.5 | -2.0 |
| Claude-Sonnet | 83.5 | 89.2 | 5.6 |
| Qwen2-VL | 60.2 | 58.3 | -1.9 |
| GPT-4o | 54.5 | 55.7 | 1.3 |
| Idefics3 | 50.6 | 56.4 | 5.7 |
| Average (absolute) change |  |  | 2.105 |

#### A.1.2 Figure Type

We provide a detailed table for VQA performance according to figure type content. We divide the performance to the following figure types: ”Chart”, ”Block Diagram” and ”Qualitative”.

**Table 8: Performance of LMMs over different figure types from the VQA set (the amount of samples for each figure type is in brackets).**
| Model | Chart | block_diagram | Qualitative |
| --- | --- | --- | --- |
|  | (4354) | (2110) | (864) |
| InstructBLIP-7B | 22.9 | 22.8 | 22.5 |
| InternLM-Xcomposer2.5-7B | 46.7 | 53.5 | 41.7 |
| InternVL2-8B | 59.1 | 68.4 | 63.8 |
| LLaVA-1.6-Mistral-7B | 27.3 | 33.6 | 33.4 |
| LLaVA-OneVision-Qwen2-7B | 48.7 | 62.8 | 57.9 |
| LLaVA-1.5-13B | 29.3 | 35.4 | 40.2 |
| LLaVA-1.5-7B | 28.1 | 33.1 | 36.0 |
| LLaVA-1.6-34B | 44.6 | 60.0 | 52.7 |
| Mantis-LLama3-8B | 29.3 | 36.2 | 36.1 |
| Phi3v | 56.1 | 65.5 | 55.2 |
| InternLM-Xcomposer2-4KHD-7B | 32.9 | 43.7 | 47.1 |
| Idefics2-8B | 34.3 | 43.0 | 41.1 |
| InternVL2-2B | 47.2 | 54.9 | 50.0 |
| Claude-Sonnet | 73.7 | 81.3 | 69.1 |
| Qwen2-VL | 63.1 | 73.9 | 66.0 |
| GPT-4o | 56.5 | 68.6 | 59.5 |
| idefics3 | 51.1 | 59.0 | 54.1 |

#### A.1.3 Question Category

We provide detailed tables for VQA and TQA performance according to the category of the questions as classified by an LLM. We divide the performance to the following categories:
”Data Analysis”, ”Attribute”, ”Reasoning”, ”Reading”, ”Localization” and ”Arithmetic”

**Table 9: VQA Performance by Question Categories (the amount of samples for each category is in brackets).**
| Model | Data Analysis | Reasoning | Attribute | Localization | Reading | Arithmetic |
| --- | --- | --- | --- | --- | --- | --- |
|  | (2291) | (872) | (903) | (1596) | (1470) | (154) |
| InstrcutBLIP | 21.16 | 29.29 | 22.77 | 31.25 | 23.87 | 23.01 |
| InterLM-XC-2.5 | 47.52 | 43.43 | 48.51 | 31.25 | 50.46 | 47.28 |
| InternVL2-8B | 61.74 | 63.64 | 65.35 | 56.25 | 62.54 | 62.41 |
| LLaVA1.6-7B | 28.91 | 31.31 | 30.69 | 12.50 | 30.00 | 30.43 |
| LLaVA-OneVision | 52.57 | 54.55 | 60.40 | 56.25 | 56.76 | 52.85 |
| LLaVA1.5-13B | 32.92 | 25.25 | 25.74 | 43.75 | 31.85 | 32.58 |
| LLaVA1.5-7B | 30.56 | 28.28 | 29.70 | 25.00 | 29.71 | 30.89 |
| LLaVA1.6-34B | 50.87 | 43.43 | 45.54 | 37.50 | 49.60 | 50.00 |
| Mantis | 32.50 | 33.33 | 30.69 | 43.75 | 31.97 | 31.80 |
| Phi3v | 58.05 | 63.64 | 59.41 | 43.75 | 58.15 | 59.07 |
| InterLM-XC-4Khd | 37.51 | 43.43 | 39.60 | 18.75 | 38.67 | 37.09 |
| Idefics2 | 37.79 | 39.39 | 35.64 | 31.25 | 39.54 | 36.34 |
| InternVL2-2B | 50.26 | 46.46 | 44.55 | 43.75 | 51.45 | 48.62 |
| Claude-Sonnet | 74.78 | 78.79 | 67.33 | 81.25 | 75.49 | 75.64 |
| Qwen2-VL | 66.93 | 66.67 | 68.32 | 43.75 | 68.73 | 65.01 |
| GPT4o | 61.41 | 60.61 | 59.41 | 62.50 | 59.83 | 59.76 |
| Idefics3 | 52.34 | 63.64 | 51.49 | 50.00 | 54.51 | 54.00 |

**Table 10: TQA Performance by Question Categories (the amount of samples for each category is in brackets).**
| Model | Data Analysis | Reasoning | Attribute | Localization | Reading | Arithmetic |
| --- | --- | --- | --- | --- | --- | --- |
|  | (2582) | (123) | (121) | (23) | (2127) | (3934) |
| InstructBLIP-7B | 24.7 | 27.6 | 34.7 | 43.5 | 20.4 | 13.6 |
| InternLM-Xcomposer2.5-7B | 54.9 | 75.6 | 79.3 | 60.9 | 74.6 | 29.7 |
| InternVL2-8B | 58.6 | 73.2 | 78.5 | 65.2 | 79.8 | 54.1 |
| LLaVA-1.6-Mistral-7B | 30.4 | 39.0 | 52.1 | 30.4 | 32.0 | 13.0 |
| LLaVA-OneVision-Qwen2-7B | 47.5 | 72.4 | 80.2 | 60.9 | 69.1 | 40.3 |
| LLaVA-1.5-13B | 31.6 | 49.6 | 47.9 | 30.4 | 32.4 | 28.5 |
| LLaVA-1.5-7B | 30.8 | 39.0 | 42.1 | 43.5 | 31.6 | 27.9 |
| LLaVA-1.6-34B | 46.4 | 69.1 | 76.9 | 47.8 | 66.8 | 46.1 |
| Mantis-LLama3-8B | 28.7 | 39.0 | 46.3 | 21.7 | 32.4 | 27.3 |
| Phi3v | 52.2 | 76.4 | 77.7 | 60.9 | 72.4 | 35.2 |
| InternLM-Xcomposer2-4KHD-7B | 45.4 | 69.1 | 77.7 | 60.9 | 66.2 | 25.0 |
| Idefics2-8B | 35.0 | 57.7 | 57.9 | 43.5 | 51.1 | 32.4 |
| InternVL2-2B | 36.9 | 57.7 | 70.2 | 30.4 | 62.2 | 32.1 |
| Claude-Sonnet | 85.1 | 90.2 | 91.7 | 87.0 | 91.0 | 78.1 |
| Qwen2-VL | 64.6 | 86.2 | 90.1 | 65.2 | 82.0 | 44.0 |
| GPT-4o | 57.5 | 79.7 | 86.8 | 69.6 | 73.0 | 40.6 |
| Idefics3 | 52.0 | 79.7 | 77.7 | 56.5 | 72.0 | 36.6 |

### A.2 Detailed examples for VQA and TQA generation

Figure: Figure 9: A detailed example for VQA questions generation.
Refer to caption: x9.png

Figure: Figure 10: A detailed example for TQA question generation.
Refer to caption: x10.png

Here we present full and detailed examples of our flow from ArXiv papers until constructing verified multi-choice Q&A. Figure 9 shows the full example for generating questions from figures (VQA). Figure 10 shows the full examples for TQA.

### A.3 Exploring more details on efficient evaluation

#### A.3.1 Extra results for LiveXiv

We start showing what would happen if our method for efficient evaluation is applied setting the number of re-evaluated models to be 3 or 8. As expected, Figures 11 and 12 show that overall performance is positively related to the number of re-evaluated models. We found that re-evaluating 5 models offers a good trade-off.

Figure: Figure 11: Performance prediction results of our efficient re-evaluation method on the hypothetical second version of the LiveXiv benchmark when re-evaluating 3 models.
Refer to caption: x11.png

Figure: Figure 12: Performance prediction results of our efficient re-evaluation method on the hypothetical second version of the LiveXiv benchmark when re-evaluating 8 models.
Refer to caption: x12.png

In Figure 13, we can see that the number of testing samples negatively correlates with prediction error, suggesting that our efficient evaluation strategy achieves lower MAE than those reported in Figure 4 in a real application, given that test datasets will be larger and unaffected by data splitting.

Figure: Figure 13: Number of testing samples negatively correlates with prediction error, suggesting that our efficient evaluation strategy will perform even better in practical situations in which test sets are larger. The plots represent the cases for 3, 5, and 8 re-evaluated models.
Refer to caption: x13.png

#### A.3.2 Efficient evaluation on MM-LiveBench

In this section, we challenge our efficient evaluation method, by examining its performance over another type of multi-modal live dataset [zhang2024lmms]. The dataset has 3 versions (May 2024, June 2024, and July 2024), and each version has roughly 250-300 samples of open-ended questions scraped from newspapers.
To evaluate our method we use GPT-4o to convert the open-ended questions into closed-form of questions where the true answer is rephrased and 3 more negative answers are proposed. Then we evaluate 13 LMMs over all the dataset versions. We use the first version as a training set and we predict the performance over the new concatenated sets using our IRT-based method. Figure 14 shows that our method still performs well on a different benchmark when re-evaluating only 5 models.

Figure: Figure 14: The results for MM-LiveBench are optimistic and we check that our method could be successfully applied in this other context.
Refer to caption: x16.png

### A.4 Prompt template for QA generations

Figure: Figure 15: Prompt template for general detailed caption.

Figure: Figure 16: Prompt template for visual question-answering.

Figure: Figure 17: Prompt template for table question-answering.

Figure: Figure 18: Prompt template for agreement filtering.

Figure: Figure 19: Prompt template for question categories analysis.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How does the system ensure diversity in question types and difficulty levels?
soundness: 2
strengths: 1. The motivation is clear and good: this work addresses the critical issue of test set contamination in current LLM evaluations, as models increasingly train on web-scraped data that may include benchmark test sets.

2. This paper proposes a scalable live benchmark without any human involvement, automatically drawing data from online scientific manuscripts, generating multiple VQA pairs, and filtering these questions to reduce errors.

3. An efficient evaluation methodology is introduced, offering significant computational savings.
summary: This paper introduces LiveXiv, an automated multi-modal live benchmark system that generates Visual Question-Answering (VQA) pairs from ArXiv papers. The work is motivated by the critical issue of test set contamination in current LMM evaluations, as models increasingly train on web-scraped data that may include benchmark test sets. The authors propose an evolving benchmark using newly published scientific papers. Besides, they introduce an efficient evaluation method to make continuous assessment practical. Multiple open and proprietary LMMs are benchmarked.
weaknesses: 1. The paper heavily relies on one LLM, i.e., Claude for question filtering. While filtering aims to reduce errors, potential biases may arise from Claude being the only model used for answer verification. This can introduce an inherent bias in the benchmark, potentially skewing results to favor Claude's architecture and underlying methodologies. The superior performance of Claude-Sonnet shown in Table 1 may be partially attributed to the fact that Claude-Sonnet itself verified these questions, potentially making them more aligned with its capabilities. The authors might address this by including additional, distinct filtering mechanisms to mitigate model-dependent biases.

2. The paper suggests that only certain types of scientific data (e.g., block diagrams, qualitative visuals, charts) are categorized for question generation. While this is effective for consistency, it risks oversimplifying the diversity of visual data in scientific publications. Expanding the types of visuals and including more complex multi-modal question types (e.g., cross-referencing multiple figures) would make the benchmark more challenging and comprehensive.

3.The benchmark is restricted to multiple-choice format questions, which limits the evaluation of models' true generative and reasoning capabilities. Including free-form answering would provide a more comprehensive assessment of model understanding and better reflect real-world applications.

4. The benchmark evaluation would benefit from more comprehensive quantitative comparisons with established datasets like DocVQA and ChartQA. Such comparisons would better demonstrate LiveXiv's advantages over static benchmarks and more clearly illustrate how it addresses test contamination issues and aligns with human preferences.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How do the authors make sure the diversity of the VQA is also scalable? For example, if gpt-4o keeps generating many questions like "what is the number in the third row and second column of the table?", which will only evaluate a fixed OCR capability regardless of which timestamp the question is generated from.
2. Why the GPT-4o performs very badly in Table 1 while it is the model to generate the questions and answers?
3. How does the domain impact the evaluation scores? Could the authors show how models perform on different categories (as shown in Table 3) in different domains and see the correlation?
4. Despite I like the benchmark, could the authors clarify their innovation in method design beyond (1) taking an efficient evaluation strategy from nlp domain and apply it to vlm domain; (2) synthetic evaluation data generation using one proprietary api for generation and another for filtering which has been applied in nlp broadly even using the same apis? The pdf parsing is also a basic step in previous works  on paper QA. Could the author provide a clear statement of their key technical innovations and how they differ from or improve upon existing approaches and include a discussion of how combining these existing techniques in a novel way for this specific application represents an innovation?
soundness: 3
strengths: 1. The author propose to have a live benchmark is a promising direction to mitigate the impact of data contamination on holistic evaluation. It is challenging to maintain a live and scalable benchmark, and the author proposed several methods to resolve the challenges: a question-answer generation pipeline grounded on a structured pdf processor and powerful proprietary VLMs for generation and filtering, and an efficient evaluation algorithm to make model comparison affordable.
2. The proposed methods at both data curation stage and the evaluation stage have been shown to be effective in previous works on language-only datasets. The authors extend the thoughts to the multimodal domain and show these methods are still effective.
3. The human study shows the automatic data filtering pipeline is effective in removing annotation errors during data curation.
summary: This paper propose a new live multimodal benchmark on scientific ArXiv papers. The authors also designed an efficient evaluation approach to rank the models. The authors verified their evaluation metric to be aligned with human evaluation with a small variance (<2.5%).

The main contributions are:
(1) A pipeline to generate and filter scientific paper VQA and TQA data using a document parsing pipeline (preprocessing), gpt-4o (generation), and Claude (filtering).
(2) An evaluation pipeline that, when evaluating a new model on the latest data, the pipeline re-evaluate old models on a subset of the latest data with a few old models for performance comparison.
weaknesses: 1. All questions are generated by gpt-4o, which may introduce issues such as the lack of diversity in questions.
2. The authors only did human study for the question-answer filtering, while they did not verify if the final ranking of models is aligned with human perception and other established benchmarks. 
3. As the author mentioned, the data curation replies on proprietary models, which makes the benchmark prone to bias and low reproducibility. 
4. The question type is limited to multi-choice answering, while it could be more interesting if the authors could extend it to a broader categories of tasks such as long-form generation.
5. The novelty of the method: the authors should clarify their novelty by comparing with existing works that ask a powerful model to generate evaluation benchmark. For example, [1] also parses papers and ask an LLM to generate question for long-form paper question-answering. These works could be extended to VLM domains by swapping out the LLM to VLM and keeping add live new data to the benchmark in a straightforward way.

[1] KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In Line 235, which LLM do you use here?
How do you select the test set in Table 1?
Line 458, which 5 models are selected by the algorithm exactly?
soundness: 3
strengths: 1. A large-scaled dataset that could be very useful for people in the VQA domain. 
2. I like the idea of 'evaluating the dataset in a dynamic way'. As most of the datasets get contaminated as training data increases by time. 
3. A rather comprehensive evaluation of the dataset, covering most state-of-the-art LMMs.
summary: This paper introduces an evolving benchmark derived from arXiv papers, utilizing LLMs to automatically identify and generate visual-based question-answer pairs from the multi-modal content within manuscripts. To manage the growing dataset efficiently, an evaluation method is proposed that avoids the need to assess every method as the dataset expands.
weaknesses: 1. A limitation of this work is that it relies on OCR to extract only tables and figures, using only their captions as input to create the dataset. This approach ignores the broader context of the paper, which may contain valuable information. For instance, many tables have generic captions like "Main experimental result of our proposed method," which may lead GPT-4 to generate only simple structure-related questions, such as "Which column has a value of **," rather than more insightful content-based questions.

2. As the dataset is entirely LLM generated, it's necessary to perform some human evaluation to avoid hallucinations and errors. Even though it's not feasible to perform such evaluation on the entire dataset, at least a small portion could be sampled and more detailed manual check could be performed beyond the simple one briefly described in Line 255.

3. I'm not sure about the efficiency and robustness of the evaluation model, even though I appreciate the idea, given the diverse styles and topics of newly added arXiv papers. Previous LMMs may become outdated, exhibit biases, and struggle to handle shifts in data distribution effectively.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please address my concerns above.
soundness: 2
strengths: 1) It is a innovative approach to creating a live, contamination-free benchmark that is both scalable and efficient. 
2) The use of advanced LMMs for automatic generation and filtering of VQA pairs is commendable, as is the development of an efficient evaluation method inspired by Item Response Theory, which reduces computational overhead. 
3) he benchmark's design allows for the expansion into new domains, potentially increasing its relevance and applicability.
summary: The study presents LiveXiv, a multi-modal live VQA benchmark designed to evaluate the capabilities of LMMs on scientific content from ArXiv papers. The benchmark addresses the issue of test data contamination by continuously evolving and utilizing the latest data, thereby providing an up-to-date assessment of model performance. LiveXiv automatically VQA pairs from figures, charts, and tables in scientific manuscripts without human intervention, leveraging advanced LMMs like GPT-4o and Claude. The study also introduces an efficient evaluation approach that estimates the performance of all models on the evolving benchmark by evaluating only a subset, thereby reducing evaluation costs.
weaknesses: 1) The scope of work presented may not be substantial enough for a benchmark paper, as the number of samples and the diversity of domains covered could be expanded. 
2) While the idea of a contamination-free benchmark is valuable, the study's focus on Knowledge-VQA tasks narrows its applicability and relevance. Data contamination is a broader issue that extends beyond Knowledge-VQA tasks. 
3) The reliance on proprietary LMMs for the benchmark's operation introduces potential variability and a lack of control over the evaluation process
4) The efficient evaluation method shows promise, its effectiveness and reliability over time and across different versions of the benchmark need further validation.

Meta-review: This submission introduces LiveXiv. It is a scalable and evolving benchmark based on ArXiv papers for evaluating multi-modal LLMs. First,  LiveXiv generates visual question-answer pairs (using their graphs, charts, and tables) automatically from domain-specific arxiv submissions without human involvement. The proposed method includes an evaluation method that reduces the cost by assessing only a subset of models. By benchmarking existing multimodal LLMs, the authors argued for the effectiveness of the benchmark. The dataset is available on HuggingFace.

The reviewers identified the strengths of this work as:
- it creates a dynamic/continuous, contamination-free  benchmark, which can help address test data contamination in LLM evaluations (Reviewers P23K, fXkv, VmiA).
- the use of advanced LMMs for automatic generation and filtering of VQA pairs, along with an efficient evaluation algorithm, significantly reduces computational costs (Reviewers P23K, fXkv, VmiA).

The weaknesses pointed out by the reviewers include: 
- corresponding to the strength point above, the reliance on a single model (e.g., Claude or GPT-4) for filtering and question generation could introduce biases, affecting the benchmark's fairness and generalizability (Reviewers P23K, fXkv).
- the focus on multi-choice questions and specific scientific data types (e.g., charts and tables) limits the evaluation’s diversity and real-world applications (Reviewers P23K, fXkv, LXM8).

During the rebuttal, reviewer P23K (5) increased rating to 5. And reviewer VmiA (5) did not engaged with the authors. Both reviewer fXkv (6) and reviewer LXM8 (6) engaged with the authors and acknowledged that the authors' rebuttal helped answer their concern and questions, but did not help move the needle in terms of overall ratings. Therefore, this work is still at the borderline towards a negative direction. 

Overall, the final ratings for this work are 6, 6, 5, 5 (up from 3). The main reason for potential acceptance is on its early attempt to build a live benchmark by naturally using arxiv submissions --- several reviewers also appreciated this. The primary reason for rejection concerns its execution particularly the robustness of the metric and complete reliance on llm as the judge. Weighing these two factors, the AC leans towards an acceptance in recognition of the conceptual idea of this live benchmark. However, if the program is too crowded,  it would be acceptable to give this work another round of improvement. In any case, the authors are encouraged to revise the work based on the review comments.

Decision: Accept (Poster)