ScImage: How good are multimodal large language models at scientific text-to-image generation?
Authors: Leixin Zhang
Steffen Eger
Yinjie Cheng
WEIHE ZHAI
Jonas Belouadi
Fahimeh Moafian
Zhixue Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
multimodality
science
image generation
Primary area: datasets and benchmarks
TL;DR: We evaluate multimodal LLMs on scientific text-to-image generation

Abstract: Multimodal large language models (LLMs) have demonstrated impressive capabilities in generating high-quality images from textual instructions. However, their performance in generating scientific images—a critical application for accelerating scientific progress—remains underexplored. In this work, we address this gap by introducing ScImage, a benchmark designed to evaluate the multimodal capabilities of LLMs in generating scientific images from textual descriptions. ScImage assesses three key dimensions of understanding: spatial, numeric, and attribute comprehension, as well as their combinations, focusing on the relationships between scientific objects (e.g., squares, circles). We evaluate seven models, GPT-4o, Llama, AutomaTikZ, Dall-E, StableDiffusion, GPT-o1 and Qwen2.5-Coder-Instruct using two modes of output generation: code-based outputs (Python, TikZ) and direct raster image generation. Additionally, we examine four different input languages: English, German, Farsi, and Chinese. Our evaluation, conducted with 11 scientists across three criteria (correctness, relevance, and scientific accuracy), reveals that while GPT4-o produces outputs of decent quality for simpler prompts involving individual dimensions such as spatial, numeric, or attribute understanding in isolation, all models face challenges in this task, especially for more complex prompts. ScImage is available: huggingface.co/datasets/casszhao/ScImage

Published as a conference paper at ICLR 2025
SCIMAGE: HOW GOOD ARE MULTIMODAL LARGE LANGUAGE MODELS AT SCIENTIFIC TEXT-TO-IMAGE GENERATION?
Leixin Zhang', Steffen Eger", Yinjie Cheng", Weihe Zhai", Jonas Belouadi", Fahimeh Moafian", 8 Zhixue Zhao?
University of Twente 1. zhwng- 5eutteete.een.
Universityy of Technology Nuremberg steffen. dt
University of Sheffield (ycheng80, zhixue. zhao 2ta.) (at.aa)))d@sfffeedd.e.... *Harbin Institute of Technology weihezhaii@insu..itttt.
University of Mannheim jonas. be www.ooaadddelmmmmmmmeeeeeeeeeeee de
Technische Universität Dresden f
ABSTRACT
Multimodal large language models (LLMs) have demonstrated impressive capabilities in generating high-quality images from textual instructions. However, their performance in generating scientific images-4 a critical application for accelerating scientific progress remains underexplored. In this work, we address this gap by introducing ScImage, benchmark designed to evaluate the multimodal capabilities of LLMs in generating scientific images from textual descriptions. ScImage assesses three key dimensions of understanding: spatial, numeric, and attribute comprehension, as well as their combinations, focusing on the relationships between scientific objects (e.g.. squares, circles). We evaluate seven models, GPT-4o, Llama, AutomaTikZ, Dall-E, StableDiffusion, GPT-ol and Qwen2.5- Coder-Instrucl using two modes of output generation: code-based outputs (Python, TikZ) and direct raster image generation. Additionally, we examine four different input languages: English, German, Farsi, and Chinese. Our evaluation, conducted with 11 scientists across three criteria (correctness, relevance, and scientific accuracy), reveals that while GPT-4o produces outputs of decent quality for simpler prompts involving individual dimensions such spatial, numeric, or attribute understanding in isolation, all models face challenges in this task, especially for more complex prompts
1 INTRODUCTION
Artificial intelligence (AI) has become an increasingly valuable tool in academic research, offering support across
various aspects of the scientific process (Byun & Stuhlmüller 2023 Chen & Eger 2023 Lu al. 2024aa] Nechakhin
et al. 2024 Shao et al. 2024). For instance, Al-based tools, such as Deep Research Elicit (Byun & Stuhlmüller 2023 and ResearchRabbii facilitate finding relevant literature for specific research topics. Tools like Grammarly assist with grammatical refinement and phraseology in academic writing and LLM-assisted text production is nowadays common (Liang et al. 2024). LLMs can also generate new ideas for scientific papers that rival the ideas produced by human scientists (Si et al. 2024). Even more holistically, approaches like The Al Scientist (Lu et al. (024a)) have demonstrated the capability to generate entire research output, encompassing everything from initial conceptualization to experimental design and paper drafting.
Despite these advancements, a critical subproblem remains relatively unexplored: the Al-driven generation scientific visualizations, including illustrative figures, charts, and plots (Voigt al.) 2024). These visual elements play pivotal role in scientific communication (Lee et al. 2016), serving as essential tools for researchers, educators, and students to convey complex ideas, data, and concepts. The ability to automate the creation of accurate scientific images from textual descriptions could significantly enhance both the efficiency and effectiveness of scientific communication and production. Compared previous attempts at automating image generation, Al-driven generation of scientific visualizations does not strictly rely on tabular data input (Yamada et al. 2018), does not require cumbersome parameter adjustments (Lindsay al. 2017), and produces a variety of outputs beyond just statistical images (Waskom 202
1SC Image prompts:
Code: https://gitan
com httm//leixin--2ha///iiieeee Saalaanng//ciiaaag
http:://wwwu...comm////////ommtp hhtps://elicit.com "ttps:/wwww.essarchabbbiaa

---

Published as a conference paper at ICLR 2025
Real-life v.s. Scientific
Text-Image v.s. Text-code-Image
Concrete v.s. Abstract
G1
02
013
014
0g2
4g2
033
004
"A binary with total of nodes."
3x] and its inverse function."
"A three-by-four matrix."
Figure 1:
Illustration of scientific text-to-image generation. The text shown below the generation query. Images on the left meet the expectations for general text-to-image tasks, while those the right highlight the specific requirements of scientific image generation. All figures are from our ScImage experiments.
Scientific visualizations often require precise spatial composition, accurate numeric representations, and correct attribution of complex scientific objects. These elements must be combined in ways that adhere to established conventions within scientific domains. While ceneral-purpose text-to-image models have made significant strides (Esser al. 2024 Touvron et al. 2023 Ramesh et al. 2021), the requirements, e.g., precise and high-resolution graphical representations, pose unique challenges for scientific image generation, as illustrated in Figure []] Moreover, the representation of objects in scientific domains-suun as batteries in circuit diagrams or trees in graph theory-differ, differs significantly from their appearance in real-life images.
In response to this need, present ScImage, comprehensive benchmark aimed at evaluating the capabilities of multimodal LLMs generating scientific images conditioned on textual descriptions. Our benchmark includes a diverse set of skills that test key dimensions of scientific image production individually and in combination, covering wide range of scientific objects, their attributes, and relations. As scientific figures are often generated with high-level coding languages such as TikZ or Python, evaluate standard LLMs (all capable of generating code output) such LLAMA 3.1 8B and GPT-40, in addition to inherent multimodal models such as DALL-E on ScImage.
Our findings highlight the need for continued research in enhancing the capabilities of multimodal LLMs for scientific image generation. By providing standardized benchmark and in-depth analysis, ScImage aims to drive progress in this critical area, ultimately supporting more efficient scientific image production.
Key contributions of this work include:
We provide a benchmark, ScImage, for testing the model capability of scientific text-to-image generation along (predominantly) three understanding dimensions: numeric, spatial, and attribute comprehension.
We explore seven state-of-the-art models on Sc Tmage, including code-based and genuine multimodal.
We comprehensively assess the models using almost a dozen human scientists across three evaluation aspects: correctness, relevance, and scientificness, and four languages: English, German, Chinese, and Farsi.
We analyze model performances across different object types, comprehension dimensions, and input languages.
We provide human evaluation scores for ~-3k generated scientific images, totaling an annotation value of approximately 3,000 USD These evaluation scores serve "ground truth" for the evaluation of generation performance and support future research on developing automated metrics for assessing scientific images.
2 RELATED WORK
In computer vision and multimodal studics, there are many benchmarks and datasets serving various purposes, including object detection (Lin et al., (014), image classification (Krizhevsky 2009 Deng al.; 2009). hand-written digits recognition (Deng) 2012). and image captioning (Sharma et al. [0188] Chen et al. 2013). but the majority focus on
real world images. Although datasets like Paper2Fig (Rodriguez et 2023) and DikikZ (Belouadi al. 2024a include scientific figures and captions extracted from research papers, there is no structured evaluation of the limitations
"Our scientists are PhD students and higher. We use the term 'scientist" to differentiate them from growd-worken or early career academics such Bachelor students.
'For English, we evaluate 404 prompts for 7 different models, yielding 2828 individual images (308 of which have compile errors, receiving an automatic score of zero). For the later multilingual phase, we evaluate 540 images (58 with compile errors). In total, we thus evaluate 3368 generations, 366 of which have compile emrors.

---

Published as a conference paper at ICLR 2025
and capabilities of scientific text-to- text-to-image models. In Section 2.1 and 2.2 we review existing benchmarks designed to
assess model abilities: visual understanding (e.g., using images as inputs (Thrush et al. 2022 Huang et al. [023] Wu
et al. 2024), discussed in Section 2.1 and ability of text-to-image generation (Section b.2b. All surveyed datasets and benchmarks in this study are summarized in Table i Appendix A
2.1 IMAGE AS INPUT
Benchmarks that use images as input often take the form of visual question answering (VQA), where images are paired with questions about their content (Biten et al., 2019: Das et al. 2024 Yue et al. 2023 Wang et al. (024a).. For example, the Multimodal Visual Patterns (MMVP) Benchmark (Tong et al. (024) focuses on challenging cases, comprising 150 CLIP-blind pairs (images that the CLIP model perceives as similar despite clear visual distinctions)
with questions designed to probe specific image details, such as relative position, object counting, or other attributes.
In the scientific domain, VQA examples are typically sourced from exams, quizzes, or textbooks (Yue et al. 20233 Lu
et al. 2024b [i et al., 2024a). Additionally. ScienceQA (Lu et al. 2022) is benchmark that uses images as contextual inputs for questions, rather than directly asking about the image's s content. This dataset also incorporates Chain of Thought (CoT) reasoning to enhance interpretability alongside the answers. CharXiv (Wang et al., 2024b) evaluates
models' abilities to describe and reason about charts through multiple-choice questions.
Another type of visual understanding benchmark focuses on caption-image alignment. Winoground (Thrush et al. 2022), for instance, challenges models to match images with their corresponding captions. The dataset includes pairs where objects or predicates are swapped, such as "there is a mug in some grass" versus "there is some grass in a mug", to test fine- -in-igr comprehension of texts. MMSCI (Li et al. (024b) extends this focus to the scientific domain, offering figure-captioning benchmark spanning 72 subjects. Additionally, SciFlBench (Roberts et al.) 2024) evaluates figure-caption alignment through tasks such as selecting the appropriate figure for given caption or choosing the correct caption for a specific figure from multiple choices.
2.2 IMAGE AS OUTPUT
Compared to visual understanding, benchmarks that assess individual dimensions of abilities in text-to-image generation models remain relatively scarce. One benchmark designed for this purpose is T21-CompBench (Huang et al., 2023), which includes 6k compositional text prompts, categorized into three groups: attribute binding (c.g., color and shape), object relationships (e.g., spatial arrangements), and complex compositions. While are inspired by this benchmark, we note that it does not target the scientific domain.
In the context of vector graph and scientific figure generation, Zou et al. (2024) develop an evaluation set to assess models' abilities in prompt comprehension and vector graph generation. Belouadi et al.] (2024a) introduce a dataset that pairs scientific paper captions (as input) with TikZ code (as output), which can be compiled into vector graphs. Additionally, [Shi et al. (2024) explore models' capabilities to replicate chart images by converting them into Python code. Compared to these works, which focus on specific evaluation settings (such as TikZ vector graphic or chart generation), our evaluation setup is broader, more targeted and more structured: we assess model performance across different input languages and output formats (TikZ vs. Python vs. plain image), object types and aspects of understanding.
2.3 EVALUATION OF TEXT-TO-IMAGE MODELS
Existing evaluations of text-to-image models primarily focus on text-image alignment and image quality for real-world images, demonstrated by benchmarks such as MS COCO (Lin et al., 2014) and studies in [Shamaa et al. (2018)
and Chen et al. (2015). Later works, such as Lee et al. (2024) and Cho et al. (2023). broaden the scope of evaluation to include aspects like aesthetics, originality, social bias, and efficiency, but these still remain within the domain of real-world images.
Benchmarks designed for evaluating real-life images are insufficient for assessing the quality of generated scientific graphs. Compared to general images, scientific graphs must prioritize accuracy in representing scientific concepts and ideas. This includes ensuring the precision of numerical values in charts or plots, and adhering to established conventions when translating real-world objects into graphical representations (e.g., depicting a "battery" in electric circuit diagrams within the domains of engineering and physics).
For figures in the scientific domain, metrics like CLIPScore and Fréchet Inception Distance (FID) are employed to assess the quality of generated graphies (Zou et al. 2024). Additionally, (Shi et al. (2024) utilize GPT-4V for automated evaluations of image quality. However, as highlighted by the MMVP benchmark (Tong et al. 2024). automated
3

---

Published as a conference paper at ICLR 2025
evaluations can be unreliable, particularly when recognizing precise directions in text and images, such as "up" and "down". The precision required for evaluating scientific graphs presents significant challenges for current automated metrics. To address this gap, we resort to human evaluation instead of automatic evaluation, using panel of 11 scientists. Additionally, we show that, indeed, standard multimodal metries employed in the community have low correlations to our human annotators in scientific domain.
3 SCIMAGE
3.1 TASK SETUP
Sc Smamage evaluates the capability of multimodal LLMs to generate scientific graphs from textual descriptions. We design prompts that require models to understand and visualize scientific concepts, emphasizing three key dimensions of understanding: (a) Spatial understanding: Assessing the models' ability to interpret and represent spatial relationships between objects, such as "left of and "on top of"". (b) Numeric understanding: Evaluating the models' capacity to handle and visualize numerical requests accurately, such as the exact number of objects or requests like 'more' and "half" (c) Attribute binding: Testing the models' ability to correctly represent object attributes such as color, size, and shape. Figure (egureedemoon demonstrates these three key dimensions of understanding.
Sparial
benkke.
Thase aquans and DMX crder where Ni 5-1.
Intangle
Namoric
Attritudo
Spatial
Numeric
Spetial
tbr Taniaan
Numerii
Spotial
Cxetioo
A beary tre with zadee The code use cossacted amoes point the
belle Meck bell.
comiies
Men ta sade
jetmoast blo..
dullie
Figure 2: Illustration of the three understanding dimensions. The first row shows the individual dimensions of Attribute, Numeric and Spatial understanding. The second row illustrates the combination of two three dimensions.
Output mode The task involves generating images either (i) directly text-image or (ii) text-code-image through intermediate code (Python or TikZ) which then has to be compiled to images based on textual prompts.
Prompting We instruct the models to generate scientific graphs with prompts. Each prompt consists of an auxiliary instruction and generation query. The auxiliary instruction is used to constrain the model to generate scientific graphs in either (i) direct text-image or (ii) text-code-image mode. Language models can exhibit sensitivity to variations prompts (Leiter & Eger 2024). To mitigate the impact of this variability and ensure fair comparison between models, we conduct pilot tests to find prompts that generally lead all tested models to generate required output type (i.e., Python code, TikZ code, or images) in scientific style. Our resulting auxiliary instructions shown in Table in Appendix B Examples of code and image output are presented in Appendix D and Appendix
8
3.2 DATASET CONSTRUCTION
We begin with comprehensive survey of relevant scientific datasets and benchmarks, detailed in Table 7 in Ap- pendix A also including math and science textbooks. This gave us the intuition that scientific graphs are described by objects and their properties (attributes) well their relative positioning (spatial relations) and numeric information (e.g., how many objects). Additionally, annotations often emphasize parts of the scientific image.
'The articles 'a' and 'an" interpreted numerical descriptors.

---

Published as a conference paper at ICLR 2025
Thus, we develop prompt requirements to ensure that varying aspects of scientific text generation are covered. We require that each prompt must explicitly define: (a) the core visual elements (objects) to be generated in the graph, e.g. cycle, square, etc.; (b) specific attributes of the object (attribute binding), e.g. red cycle, or count of the object (numeric), e-g. three cycles. the positioning arrangement and placement (spatial) of objects within the graph, e.g. on the bottom in relation to another object (to the left). (d) We finally consider any required labels, legends, additional textual elements (annotations). Further, for graphs containing multiple objects, the prompt must additionally specify the quantity of each object type, the relative spatial logical relationships between objects, and the individual properties of each object group. Individual aspects are typically optional, i.e., not every prompt has to specify numerical or spatial components. Specific details on dataset construction follow below.
Generation Queries Q We adopt a structured methodology that leverages a dictionary D along with a set of query templates T to create a diverse, comprehensive, and traceable set of generation queries Q for the ScT Image evaluation dataset.
Dictionary D defines key elements relevant to scientific figures, including objects (e.g., square and circle), attributes (e.g., color and size), spatial relations (e.g., left, right, between), and numeric values (e.g., three, five, two more). To construct this dictionary, we begin by building the list of objects. First, we manually extract frequent object entities from DATIKZ (Belouadi et al., 2024a), an image-caption dataset derived from scientific publications.
We filter out objects highly dependent on the context of the original paper, such as mathematical formulas adjacent to figures and line segments with specific values Additionally, we simplify complex objects- srrectt-succhee as intricate circuit designs and automata intended for specific applications. Next, we manually define representative spatial relations, such as "to the left" and "at the center of", to describe the positions of objects within graphs. Additionally, create attribute sets to capture detailed object properties, including size, color, and line thickness. Numerical requests are also incorporated into the dictionary to replace values in bar charts assess models' accuracy in representing object counts. We then collect attributes and relations that appear at least three times and then manually compile a list of attributes and spatial relations by merging similar ones and removing domain-specific ones.
At the top level, D is organized into classes for objects, attributes, numeric, and spatial relationships. Each class then contains a list of descriptive words specifying the class. When selecting a descriptive word from D for given blank in the query templates t;, we first locate the specific list corresponding to the word class and then randomly choose an item from it I For clarity, we present a snippet of D below:
D -
"12D_objects", ["square", "circle", ....), "'D_oobjccts" ["cube", "sphere", ...].
"colors"; ["red", "blue", ...], ...), "spatial
["left", "right", ...,, ...)
We define a set of query templates T, where each template t; E T is one or several sentences with one or more placeholders. These placeholders are designed to be filled with elements dj E D, which may include objects, attributes, or relations. To construct a generation query qs E Q, we select a query template t; and populate its placeholders with appropriate attributions from D. This process allows us to create diverse queries with meta-info. The final step in preparing our generation prompt involves prefixing the constructed generation query qu with task-specific auxiliary instructions, as detailed in Section 3.T This structured approach ensures transparency and flexibility across diverse queries, while minimizing potential biases toward specific objects, attributes, and other contextual elements.
Further, ScImage is crafted to cover a wide range of scientific graph types and complexities. We classify the object types of all prompts based on the following categories: 2D geometric shapes, 3D geometric shapes, charts, graph theory representations, matrices, real-life object modeling, tables, additional annotations, functions & coordinates. For each category, we create multiple templates that vary in complexity and combine different aspects of spatial, numeric, and attribute understanding. For each template, we create four different generation queries by sampling different elements from the dictionary D. In total, we have 101 query templates and 404 generation queries. Examples of Sc Image are shown in Table 10
*DATiKZ is unsuitable for exploration, as it contains textual descriptions 'in the wild' and additionally its captions are often unsuitable for reconstructing the output image at hand.
"During the word collection pracess, we excluded complex and rare terminologies due to potential bias. Consequently, attributes such as 'hinges" and 'hyperstatic structures' were omitted.
II our dataset construction, we aimed to provide clearly specified target dimensions for scientific image generation, where the queried attributes are precisely known. This approach enables specific and targeted evaluation. It contrasts with 'roal-world' evaluations that often utilize lengthy query prompts encompassing multiple dimensions simultaneously, where model failures cannot be clearly disentangled across different aspects. For instance, while Datikz (Belouadi et al., 2024) provided a worl-world dataset that
5

---

Published as a conference paper at ICLR 2025
3.3 EVALUATION
We employ a multi-faceted evaluation approach to assess the quality and accuracy of the generated scientific graphs:
Human Evaluation Our human evaluators assess the generated images based on three criteria: Correctness: Assessing the accuracy of the visual representation in re- lation to the textual prompt. Relevance: Evaluating how well the model avoids generating redundant or irrelevant objects or attributes of objects. Scientific Style: Evaluat- ing the appropriateness of the image for use in scientific publications. Each criterion is rated on scale from 1 to 5, with 5 being the highest score. An additional score of 0 is assigned in cases where code generation cannot be compiled into an image due to compilation errors |T| The detailed evaluation guideline is given in Appendix F
Agreement
Correctness
Relevance
Scientificness
Joint Spearman
0.67
0.62
0.73
Joint Pearson y
0.20
0.64
0.71
Joint Weighted Kappa
0.50
0.41
0.45
Pair Spearman r (Eng)
0.73
0.64
0.63
Pair Pearson (Eng)
0.75
0.65
0.63
Pair Weighted Kappa (Eog)
0.61
0.52
0.47
Pair Spearman (Mulit)
0.80
0.75
0.73
Pearson (Multi)
0.80
0.77
0.79
Phir Weighted Kappa (Multi)
0.64
0.60
0.66
Table 1: Agreement of joint evaluation in the calibration session (joint) and pair agreement (agreement across all examples of two sets of annotations) in the final evaluation for English and multilingually.
We employ a panel of 11 expert annotators, carefully selected represent the target users of scientific plots and graphs, This panel consists of: eight Ph.D. students, one postdoctoral researcher, and two faculty members from mathematics and computer science, ensuring domain expertise in evaluating scientific visualizations. We provide each annotator with detailed annotation guidelines given in the Appendix [F] Furthermore, before formal annotation distribution, we conduct a calibration session to match the understanding of the annotation standard. We randomly assign examples to annotators and further assign at least two annotators per instance to mitigate annotation biases.
Agreements Table I presents the agreement scores (Spearman's r. Pearson's 's r and weighted Kappa) from both the small-scale calibration session (joint evaluation: 315 images are evaluated by all evaluators) and the later pairwise evaluation (pair evaluation: every image is evaluated by pair of evaluators) using various models (see Section ( A relatively strong positive correlation with Pearson and Spearman r between 0.62 and 0.80 is observed across all evaluation criteria. Weighted kappa, a chance-corrected measure of agreement for ordinally scaled samples, is within commonly accepted ranges for agreement, with almost all measures above 0.5. The multilingual evaluation, conducted by subset of 6 more experienced evaluators, shows a higher level of agreement than the English evaluation. Overall, weighted kappa is 0.62 for correctness for combined English and multilingual evaluation and above 0.52 for relevance and scientificness.
We tax the value of our evaluation roughly 3,000 USD, with up to 11 annotators involved for up to hours each, all working for conservative estimate of 40 USD per hour (including taxes), on average.
Automatic Evaluation We also test how well recent automatic text-to-image evaluation metrics correlate with our human judgements. We explore 5 recent multimodal metrics. These achieve a highest Kendall correlation with human scores of 0.26, where the agreement on the correctness dimension highest and lowest on scientifieness (maximum Kendall of 0.15). This underscores the value and necessity of our human annotation campaign. Details are given in Appendix
4 EXPERIMENTS
We employ two types of models for image generation, corresponding to the two output modes described in Section 3.1 For (i) direct text-image mode, we include DALL-E and STABLE DIFFUSION; for (ii) text-code-image, we include GPT-4o, LLAMA 3. 8B and AUTOMATIKZ (Belouadi al., 2024a), where the model prompted generate Python or TikZ code. AUTOMATIKZ is specifically fine-tuned for generating TikZ code, therefore, we only prompt AUTOMATIKZ to generate TikZ.
The overall model results are presented in Table 2 showing averages based on per-instance human evaluations and further averaged across annotator pairs. Instances where code generation resulted in compilation
Overall performance:
inspired our selection of scientific objects, it lacks specificity and combines various understanding properties within each prompt. Our evaluation framework draws inspiration from the established CheckList methodology of Ribeiro Ribeiro et al. (2020).
Il "While assigning score of 0 may seem harsh, we also report results when ignoring all compile errors, which would constitute an upper bound.

---

Published as a conference paper at ICLR 2025
errors are penalized with a score of 0 In Appendix G we show scores where compilation errors are ignored: potentially compilation errors could be fixed by self-consistency checks future approaches. Overall, GPT-4o text-code-image mode (GPT-40_iie and GPT-40 GPT-4o_python) stands out as the model achieving the best scores across all evaluation dimensions. However, scores below 4 in all three evaluation dimensions, indicating at least some mistakes average in every output.
Correctness: GPT-40 outperforms all other models by a large margin (more than 1.3 points), achieving the highest scores for
Scientilic style
Compile Error Rate
Model
Correctness
Relevance
both the text-tikz -ix-tiizz-mmage (3.50) and text-python-imeee -mmonn (3.51)
Automatikz
2.05
2.31
3.35
0.04
output. All other models have correctness scores between 1.7 and 2.2, indicating low correspondence between instruction and output image. LLAAMA 3.1 8B tikz (1.78) and AUTOMATIKZ (2.05) are the worst models. Interestingly, LLAMA 3. I 8B performs considerably better with Python as output than with TikZ as output (2. vs. 1.78). Similarly, GPT-40 performs marginally better with Python the coding language. When ignoring compile errors [9b.le[19]. the code generation models become substantially better, especially LLAMA 3. 8B im- proves by almost point, leaving the direct text-image generation models STABLE DIFFUSION and DALL-E among the worst models.
Ilama tikz
1.78
1.94
2.61
0.29
GPT-4a-tte Aikz
3.50
3.67
3.75
0.09
Alama_pythoa
2.10
2,54
3.18
0.28
GPT-4o python
3,51
3,40
3,93
0.07
Diffusion
2.19
2,09
1.96
DALLEE
2.16
2,00
1,55
Table 2: Overall model performance, averaged across instances and annotators (compile error of code output is penalized with score 0).
Relevance: GPT-40 also dominates relevance, but with smaller margin (1.13 vis-a-vis the second best model LLAMA 3.1 8B with Python output). LLAMA 3.1 8B again prefers Python output while GPT-40 prefers TikZ. Visual models STABLE DIFFUSION and DALL-E often include irrelevant details in their output and are among the worst; Table 10 in Appendix for examples.
Images converted from Python or TikZ code receive notably higher scores (> 2.5) in scientific style compared to direct image generation from DALL-E and STABLE DIFFUSION (< 2). This suggests that code output as intermediate step offers significant advantage for scientific graph generation, as opposed to visual models that are primarily trained on real-life images.
Scientificness:
However, a significant drawback of models that generate code is the potential for compilation failures. For instance, GPT-40 experiences 35 TikZ code and 27 Python code compilation errors. LLAMA 3. 8B has even much higher failure cases: 116 for TikZ mode and 113 for Python code, representing approximately 28% of all prompts. Aut kz performs best in terms of compilation success, with only 17 cases in total. The low scores observed for LLAMA 3.1 8B in Table 2 can largely be attributed to penalties for these compilation errors. If these were ignored (or could be fixed), LLAMA 3.1 8B would outperform AUTOMATIKZ as shown in Table [9 in Appendix (note, however, that the comparison for both models includes different instances, thus is not fully fair in the table).
5 ANALYSIS
Text=C odde-maaee
Text-Image
We conduct more detailed analysis of model perfor- mance, focusing on understanding types (attribute, nu- merical and spatial understanding) and object types to identify which categories present the greatest challenges for the models.
sttribte numerical spatial
attribute numerical spatial numerical spatial
three types
Types of Understanding Table 3 presents the fine- grained correctness scores for different understanding types. FigureBJillsstenn illustrates the performance of two modes of generation (text-code-mmage and text-image) separately. Notably, spatial understanding appears to be the most challenging across all textual models. For instance, while GPT-40 achieves scores around 4.0 for attribute binding, its performance drops substantially for spatial understanding, remaining well below 3.5.
Figure 3: Comparison of text-code-image and text-image: correctness scores averaged across model types. 'Three types' means attribute. numerical and spatial understanding combined.
A2 score of does not apply if a compile enror occurs due to missing TikZ code like Veegin (dcumenttlasss). Models sometimes assume that a document has already been set up. so we check and add codes at head and tail for generated codes before compiling the image.
7

---

Published as a conference paper at ICLR 2025
Attribute & Numerical
Attribute Spatial
Numericul &
Attribute & Numerical Spatial
Model
Attribate
Numerical
Spatial
Spatial
Automatikz
2.42
1.91
1,71
2.29
2.04
2.13
1.77
Llama,
2.53
1.55
1.77
1.69
1.84
1.91
1.3
GPT-4o tikz
4.11
3.49
3,35
3.41
3.53
3.59
3.13
2.24
2,38
1.96
2.28
2.28
1.63
1,97
GPT--4o GIPT-4o_python python
3.95
3,92
3,47
3.46
3.34
3.28
3.13
Stable Diffusion
2.75
1.73
2,06
2.41
2.46
2.11
DALL EE
2.68
1.77
2.13
2.36
2.31
1.94
2.07
Average
2.95
2.39
2.35
2.56
2.54
2.35
2.21
Sample Size
56
Table 3: Correctness evaluation within each understanding category (compile errors are penalized with score 0).
Tikz Image
Pwthon--hhhaaaee Image
Text- Image
Autommat for
Diveett
LiiTa python 307-46 python Edgel
Rew Object
Stanie Difiuston
Lama skz 67T44 Hice
DALL E
Tack
Arectaaioo
Arnotalion
Naccions Condbagte
Funcios Coosigte
Funcion Coodiagle
Figure 4: Generation performance models on different object types. The same scale is used three radar bars, with the center correctness score O, and the outermost circle as 5.
In contrast, for the image generation models STABLE DIFFUSION and DALL-E, numerical comprehension poses the greatest challenge (Figure3). Both models score between below 1.8 for numerical understanding, substantially lower than their scores for attribute understanding (~2..7 and spatial understanding (above 2.0). This indicates an interesting discrepancy between model types.
Due to their weakness in spatial understanding, tasks that involve combined understanding types- types-including including numerical & spatial understanding, as well as numerical & spatial & attribute understanding also Both GPT-4o __tthon and GPT-4o tikz record their lowest scores when addressing prompts that require all three understanding types, in comparison to prompts focused on individual understanding types.
also tend to receive lower scores.
Function& Coordinate
Model
shape
shape
Chart
Graph theory
Matrix
Real-life object
Table
Annotation
Automatikz
1.71
2.40
181
2.13
1.33
1.90
Llama, tikz
2.72
1.45
0.47
0.10
2.06
1.42
1.13
1.33
1.55
GPT.44
3.00
4.38
3.56
Llama_python python
2.49
1.39
3,15
0.00
0.94
2.37
0.00
2,00
2.13
GPT-40__pyhho python
3,11
3.25
2.73
3.13
3.25
3.88
3,39
3.20
Stable Diffusion
2.08
2.43
2.11
1.43
1,56
2,96
2.13
2.11
1.75
DALL-E
2,08
2.47
1.86
1.25
1.50
3.17
1.75
2.11
1,58
Average
2.24
2.00
2.20
2.26
2.22
Sample Size
162
97
34
20
8
38
9
20
Table 4: Object Complexity for Models: Correctness scores by object type.
Object Categories Table 4 presents the average correctness scores for each object category. The performance of different model types Tikz--maaa.. Text-Python- Image, and Text-Image) is visualized separately in Figure In general, graph theory representation (e.g. nodes and edges in a binary tree or graph) poses great challenges for models, with an average score below 1.7 across all models, compared to above 2.0 the remaining object categories.
As illustrated in Figure GPT-40 the top top-performing model across all object types, with both TikZ and Python- generated images consistently achieving the highest scores within each category. However, GPT-4o shows slightly lower performance in generating matrices (TikZ. score: 3.00), 3D shapes (Python score: 3.1 1)) and real-life object
models (TikZ score: 3.14).

---

Published as a conference paper at ICLR 2025
Criteria
Correctness
Relevance
Scientific style
Language
EN
DE
ZH
FA
EN
DR
ZH
FA
EN
DE
ZH
FA
Llama_tie likz
1.88
1.48
1.50
1.23
2.18
1.78
2.10
1.68
2.78
2.23
2.80
2.90
GPT-4o tikz
3.85
4.03
3.98
3.68
4.03
4.23
4.60
3.98
4.10
4.43
4.40
3.98
OpenAl-ol tikz
4.43
3.68
3.83
4.05
4.45
3.80
4.10
4.18
4.40
3.88
4.03
4.05
Llama_ Llama_python
2.53
1.35
1.75
1.78
2.70
1.53
2,00
1.90
3.20
2.50
3.10
3.30
GPT-4o_python
3.38
4.15
4.13
3.48
3.35
4.18
4.23
3.35
3.88
4.50
4.83
3.85
OpenAl-ol_ __tthoohhon
4.28
3.45
4.10
3.60
4.10
3.45
3.93
3.60
4.50
4.08
4.30
4.05
Qwen2.5_ S_pthon
3.10
2.30
2.05
2.40
3.08
2.48
2.25
2.53
3.70
3.43
3.28
3.68
DALL-E
1.98
2.15
1,83
1.93
1.88
2.03
2.03
2.00
1.40
1.58
1.53
1.50
Average
3.18
2.82
2.89
2.77
3.22
2.93
3.15
2.90
3.49
3.33
3.53
3.41
Table 5: Multilingual performance of overall generations. The best highlighted in bold per metrics per language.
LLAMA 3.1 8B (in both TikZ and Python code) clearly struggles with the representation of graph theory structures, such as nodes and edges, with a correctness score close to O. Similarly, it performs poorly in table generation, scoring 1.13 with TikZ and O with Python. The largest discrepancy between TikZ. and Python output is observed in chart generation, where TikZ achieves a score of 0.47, while Python scores 3.15. In contrast, TikZ outperforms Python in matrix generation, with correctness scores of 2.06 and 0.94, respectively. This may be attributed to the availability of libraries with Python and TikZ code. Matplotlib in Python is frequently used for chart and plot representation, while the usage of matrix presentation with proper math format is rare.
AUTOMATIK2 shows the lowest scores for annotation (1.33) and 3D geometrie shape generation (1.52) across all object types. It performs best in 2D geometrie shape generation (2.52), though it still lags behind GPT-4o, which scores above 3.
Figure 4 reveals that while visual models exhibit the opposite trend. STABLE DIFFUSION and DALL-E perform best in real-life object modeling, with scores of 3. 32 and 3.24, respectively, and in 3D geometric shape generation, with scores of 2.43 and 2.47.
images are of higher quality for 2D geometric shapes compared to 3D shapes,
Multilingual Evaluation We further evaluate model performance across diverse languages. Due to the high annotation costs, we sample 20 instructions and translate them into Chinese, German and Farsi by native language co-authors. Each prompt is derived from a unique tem- plate to ensure diversity. Moreover, these instructions are curated to encompass all understanding dimensions: the single dimension of spatial, attribute, numerical under- standing; combinations of two dimensions (e.g., prompts requiring both attribute and numerical understanding); and prompts requiring all three understanding dimensions. We then feed the 20 prompts to all models except for AUTOMATIKZ, which is our only model fine-tuned on English TikZ data, and STABLE DIFFUSION. We additionally include the recently released OpenAl ol -p-rreve, which emphasizes its science, coding, and mathematics capabilities 5--55--oor--BBBieee
Code
Correciness
Relevance 2.31
Scientific style
Error rate 0.19
TIKZ.
2.64
3.18
Python 2.81
2.97
3.56
0.17
Table 6: Comparison of TikZ code performance and Python code performance: scores averaged from model GPTT-40 and LLAMA 3.1 8B. Compile errors are penalized with a score
OpenAl 2024), as well as
Results are shown in Table 5 Interestingly. English does not always lead to the best results on average. While correctness and relevance is highest with English prompts with a margin of 0.29 and 0.07 ahead of Chinese, outputs generated from Chinese prompts are better in terms of scientificness. Farsi is worst on average. Model wise, LLAMA 3.1 8B becomes considerably worse regarding correctness and relevance in languages other than English, while GPT-40 often even perform better in non-English languages. Remarkably, OpenAl ol -revveww is better than GPT-40 in English (and regarding its maximum scores), often by a considerable margin (e.g., 4.28 vs. 3.38 in correctness for English input with python output), but performs substantially worse in non non-English languages (e.g.. 3.45 vs. 4.15 in German with python output), except for Farsi. three evaluation dimensions, although Qwen generally exhibits robustness when responding to Chinese queries (Bai et al. 2023).
7B- Instruct demonstrates superior performance in English across all
Output code difference As shown in Table 6 Python code generation outperforms TikZ output across all three evaluation criteria: Correctness, Relevance and Scientific style. Furthermore, the Python code output exhibits a lower
We only present the results of Qwen2.5 2-t.----CoC-777 7B Instruct in text- extt-pyhhon-oe image settings in Table 5 as its generated TikZ code frequently contains rendering errors that prevent successful conversion into images.

---

Published as a conference paper at ICLR 2025
compile error rate (0.17 for Python vs. 0.19 for TikZ). The disparity may be attributed to the richer resources of available Python code in the training data for L.Mss.
Qualitative Analysis We diagnose some issues by closely examining the generated output images, specifically highlighting problems that point areas for future improvement in the models. For instance, the generated images from some models reveal a lack of physics knowledge. In cases requiring an image of liquid in a container (Figure 5), the liquid is often placed incorrectly, and LLAMA 3.1 8B, but it is not observed in models like GPT-40, STABLE DIFFUSION and DALL-E.
the bottom of the container. This issue occasionally
with AUTOMATIKZ
Liama Python
GPT-4o Python
Automatikz
GPT-40 Python
GPT-4o
measuring is filled with 300 ml of the light blue salution.
container is half filled
A football moves along parabola.
football moves along parabola.
There are boxes on 30-degree slope.
There are five boxes on 30-degree slope.
Figure 5: Incorrect model outputs can often be attributed to lack of empirical, physical world knowledge.
A challenging scenario for most models is generating an object moving along parabolic path. GPT-4o and LLAMA 3.1 8B occasionally depict a correct downward-opening parabola, but upward-opening parabolas also exist in their generation, indicating lack of understanding of the trajectory of how object moves. Another common issue across models their difficulty in generating images that depict "boxes placed on a slope at a specific angle". Although GPT-4o sometimes manages to generate the correct image, their performance is inconsistent. This suggests lack of understanding of the interaction between gravity and the support surface, as well as difficulty positioning objects at the correct angle 2D plane.
6 CONCLUDING REMARKS
Our study presents the first comprehensive evaluation of multimodal LLMs for scientific image generation, using our novel Sc Image benchmark. Our assessment reveals both significant progress and persistent challenges in the field. While models like GPT-40 sometimes demonstrate proficiency in tasks involving individual dimensions of understanding (spatial, numeric, or attribute-based in isolation), all evaluated models struggle with complex tasks requiring combined understanding, On average, even GPT-40 performs below 4 correctness our benchmark. For example, due its lack world knowledge or an inability to correctly plan how 3D object should be presented, GPT4 sometimes has difficulty arranging objects correctly in two-dimensional image. Code based models have difficulties especially with spatial understanding, while image based models struggle the most with numeric understanding.
We find that code- based outputs generally outperform direct image generation in producing scientifically styled images. However, performance varies considerably across different object types and languages, highlighting the need for more robust and consistent modeling approaches in the scientific domain. These findings underscore the importance of continued research to enhance multimodal L.Ms capabilities scientific image generation. By providing a standardized benchmark and detailed analysis, ScImage aims to drive progress in this critical area, supporting more efficient scientific communication and accelerating cross-disciplinary research.
As multimodal LLMs evolve, their potential revolutionize scientific content generation remains an exciting frontier in Al research. Future work should focus on improving models' ability to handle complex, molli-diimenionall reasoning tasks and ensure consistent performance across diverse scientific domains and languages. As science serves humanity and should be accessible by everyone to foster diversity and inclusion, this concerns particularly open-source models which can be considered at best mediocre for the tasks sketched in ScImage, with performances that substantially lag behind closed-soures proprictary models like GPT-4.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. In The Twelfth International Conference on Learning Representations, 2024a. URL /RK.httpps///ppereeiie net aettforuumidd=v333TT
Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. Detikzify; Synthesizing graphics programs for scientific figures and sketches with tikz, 2024b. URL
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Jungwon Byun and Andreas Stuhlmüller. http::/woo.oooghggsss////o/oooop conttatt/component///4eeesse
Elicit:
Language
models as research tools.
2023. URL https: http://wwooollbrrrr.....oom
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zimick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:/504.0032.. 2015.
Yanran Chen and Steffen Eger. Transformers go for the LOLs: Generating (humourous) titles from scientific abstracts end-to-end. In Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, and Andreas Rücklé (eds.), Pruceedings of the 4th Workshop on Evaluation and Compurison of NLP Systems, pp. 62-84, Bali, Indonesia, November 2023. Association for Computational Linguistics. doi: 10. N0.11865/v1 1tt5553/w112222..244lllll2
URL
org 2023 eva.l41p-11.6666 2tt.e///g-22.....111.1-.-1--1.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3043-3054, 2023.
Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Tont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-graing fine-grained evaluation for text-to-image generation. In ICLR, 2024.
Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. CoRR, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-254-25 2009. doi: 52005.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE Signal Processing Magazine, 296):141-442, 2012. doi: 10. 109MM..22222 1477.
Patrick Esser, Sumith Kulal, Andreas Blatmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
URL https: // arxiv.... org/ abs 2400. 03206
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: URL.
10. 10.18653/v1/2021 10t33//1/22211emlp--oe
logy. wwwp:orgg.2220 emn www.eenwmmmmm. 1p-mapmmin 595
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for woon-world compositional text-to-image generation. Advances Neural Information Processing Systems, 36: 78723-78747 2023,
11

---

Published as a conference paper at ICLR 2025
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL /iittpss//arriii.oor////////////
01569
A Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
Po-Shen Lee, Jevin D. West, and Bill Howe. Viziometrics: Analyzing visual information in the scientific literature. IEEE Transactions on Big Data, 4:117-129, 2016. URL https://apppp http:://p.....aaaattoooooooooo.p 3665638
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, al. Holistic evaluation of eoxt-to-image models. Advances in Neural Information Processing Systems, 36, 2024.
Christoph Leiter and Steffen Eger. Prexme! large scale prompt exploration of open source Ilms for machine translation and summarization evaluation. ArXiv, abs/2406.18528 2024. URL https: /aaps:://ppp..// www.nticscoooa...oooooom
(CorpusDD:270737773
Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. The Eval4NLP 2023 shared task on prompting large language models as explainable metrics. In Daniel Deutsch, Rotem Dror. Steffen Eger, Yang Gao, Christoph Leiter. Juri Opitz, and Andreas Rücklé (eds.), Proceedings of the 4th Workshop on Evaluation and Comparison of NL.P Systems, pp. 117-138, Bali, Indonesia, November 2023. Association for Computational Linguistics. doi: 10. (0.18663/1122023eeaalmm 1.10. htt.:pp://acclaaooooooooroor////
eeva4nnlp-1.0
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.0m 00231,, 2024a.
Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, al. Mmsci: dataset for graduate-level multi-diseipline multimodal scientific understanding. arXiv preprint X41v:2407.04903 2024b.
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp. Wenlong Ji, Xuandong Zhao Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, and James Y. Zou. Mapping the increasing use of LLMs in scientific papers. First Conference on Language Modeling, 2024. URL https: //openreview. net/ forum? 1d=
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence 2014: 13th European Conference,
Zitnick. Microsoft coco: Common objects in context. In Computer Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Theodore Lindsay, Peter Weir, and Floris van Breugel. Figurefirst: A layout-first approach for scientific figures. In Python in Science Conference, pp. 57-63, 01 2017. doi: 22250000/hinmma-77466777-----...
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.0622 06292, 2024a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Aei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /tnttpps://ppnrev// wwt.//eeeevvveeeeeeeeoeeeeeeeeee ew. net. forum? in j.j.j:Hjjj-k............. Tc_
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models visual contexts. The Twelfth International Conference on Learning Representations, 2024b. URI htt.:tttss://peeeevieeee net.
Yu Meng, Mengzhou Xia, and Dangi Chen. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of What Art? A Call for Multi-Prompt L.M Evaluation. Transactions of the Association for Computational Linguistics, 12:933-944 08 2024. ISSN 2307-387X. doi: 10.1166. 162/tacl 1. hRLhttp:: .et..org/110 http:/10..111/taaa 00681
12

---

Published as a conference paper at ICLR 2025
Vladyslav Nechakhin, Jennifer D'Souza, and Steffen Eger. Evaluating large language models for structured science summarization the open research knowledge graph. Information, 15(6), 2024. ISSN 2078-2489. doi: 10.3390/ info15060328.
http:::/wwwww.pp com
OpenAL.
Introducing openai ol (preview).
htto:://opemaa.comm/mee
oct 2024. Accessed: 2024-10-26.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL httth:/tp::aaattt http://xxivvvcoo wwg. hbtp://////// 2102. 12092
/
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 22.4992--4 4912, Online, July 2020. Association for Computational Linguistics. doi: 10. (/553//1/202.acc-mmi 442. URL https: http:/://alaaathholoaoooo org/
2020.ac/ mwa.anann..mmmm......... 442/
Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. arXiv preprint (rXii:2405.08800 2024.
Juan A Rodriguez, David Vazquez, Issam Laradji. Marco Pedersoli, and Pau Rodriguez. Ocr-vqgan: Taming text- within-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3689-3698, 2023.
Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluatioos? objectively scoring text-to-image prompt coherence metries with 12iscorescore (ts2), 2024. URL
Bttp:///arrxx.vorrr//bb//// 04251
Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in writing wikipedia-like articles from scratch with large language models, 2024. URL htttps://aaxiii wwwpaaiwwwwwioiaaaaaaaaaaoa argg//bs//422
14207
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/P18- 1238. URL httpps:/aclaatholoogy
1238
Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing. Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating Imm's 's cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406. 09961, 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can ilms generate novel research ideas? large-scale human study with 100+ nlp researchers, 2024. URL
04109]
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint
2024.
Tristan Thrush. Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint MxXiv:2401.06200 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume I ample. Llama: Open and efficient foundation language models, 2023. httpthttss///xxii.orttee
abs / 220223322123 13971
13

---

Published as a conference paper at ICLR 2025
Henrik Voigt, Kai Lawonn, and Sina Zarrieß. Plots made quickly: An efficient approach for generating visualizations from natural language queries. In Nicoletta Calzolari, Min Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (IREC-COLING 2024), pp. 12787-12793, Torino, Italia, May 2024, ELRA and ICCL. URLhttps: http://aaaaattotoooo org 2024 1rec main. 1119
Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models. In Forty-forst International Conference on Machine Learning, 2024.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang. Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solee solving abilities of large language models. In Forty-first International Conference on Machine Learning. 2024a.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal Ilms, arXiv preprint
arXiv:2406.18521, 2024b.
Michael Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6:3021, 04 2021. doi: 10211105//oss.0002
Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in large vision-language models. In Forty-first International Conference on Machine Learning. 2024.
Ryoya Yamada, Manabu Ohta, and Atsuhiro Takasu. An automatic graph generation method for scholarly papers based on table structure analysis. In Richard Chbeir, Hiroshi Ishikawa, Kazutoshi Sumiya, Kenji Hatano, and Mario Koeppen (eds.), Proceedings of the 10th International Conference Management of Digital EcoSystems, MEDES 2018, Tokyo, Japan, September 25-28, 2018, pp. 130-140. ACM, 2018. doi: 10. 10. 10.1145/2281355.2288833 URL nttps://cci.org: 1ete///1..111/33223 1141 011111///2113555 3281389
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large lision-language models towards multitask agi. In Forty-first International Conference on Machine Learning, 2024.
Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference Machine Learning, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.1650) 2023.
Bocheng 7ou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. Vghench: Evaluating large language models on vector graphies understanding and generation. arXiv preprint arXiv:2407. 10972, 2024.
14

---

Published as a conference paper at ICLR 2025
A BENCHMARK SURVEY
Dataset
Size
Sci-domain
Input
Output
Type of challenges
STVOA
2019
23k
image question
anwwer the question
text ideutification, recognation and reasoning
Singh ml.
TeaVQQ
2019
~28 images
--45k questions
image question
answer the question
text identifccarine, recegration and reasoning
FNAMS.y
Das et al.
2024
~20.9k
image questions
answer the question
text identifccation, exam question reasoning
MMVP
Tong ct al
2024
image question
answer the question
images with similar CLIP emberddings despite
visual distinctions.
Yue al.
MMMU
2023
11.5K
image question
snswer the question
knowledge and reasoning college esams
Lu ct al. 2022
science 04
21K
Image question
answer explanation (CoT)
scientific problem and image reasoning
Mist
ct al.
2024h
~6k
image question
answer the question
math problem solving
SeiBench
Wang
4024
image question
anwwer the question
colloge lovel problem solving
ArXivOA
Li al.
2024g
100K
image question
answer the question
GPT-4V generated questions for arXiv paper figures
VGhench
Zou ot al.
2024
4219
image question
answer the question
object calgorr, color, objoct function, position, etc.
CharXiv
Wang cl.
2024b
Image + question
answer the question
chart understanding and reasoning
Wadhawan
CONTEXTUAL
2024
image question
answer the question
image reasoning
(avoid textual recognition
reasoning from language models)
Ying al.
MMTRench
2024
image question
answer the question
visual recognition, localization. OCR, counting. perception. temporal understanding, et al.
R-Bench
4500 images questions
ballucination test: object reasoning. relutionship (between objects) reasoning
Wa
image question
the question
MM Vet
187 imaegs 205 questions
object recognition, spatial awareness, knowledge reasoning, math capubility OCR, generation
2024
image question
the question
Rodriguez al.
Paper2Fig
2023
100k
caption
image
scientific image generation
Belouadi
Datikz
2024a
120k
caption
image
scientific image generation with Tikz code
Zou et al.
VGbench
2024
caption
Image
generate SVG, TikZ. and Graphviz images
121-CompBenn
attributc binding, object relationshiis. complex composition
Huang
2023
caption
image
PAINTSKILLS
Cho cl
2023
-665k
caption
image
object recegnition, abject counting spatial relation, etc
Lcc et al.
2024
caption
image
reasoning, knowledge, mulilinguality, etc
SeiFIRench
1000
multiple choice question: select correct caption figure or select a correct image for given caption.
ability to match image caption
Roberts al.
2024
Li 2024b
MMSCI
742k ligures
scientific figure and cuption pairs. (aim: alignment of figure-caption pairs)
scientific figure understanding, figure-caption alignment
ChartMimic
Shi al. 2024
1000
chart image instroction Python code
chart image to Python code conversion
Wino-ground
text-image paits
reasoning of objeats relationship difference by swapping words in 2 captions
Thrush al.
2022
1600
(aim: match the correct text est-image pains given two captions and two images)
Table 7: Summary of challenging benchmarks in visual reasoning and text-to-image generation.
15

---

Published as conference paper at ICLR 2025
B GENERATION INSTRUCTION
Generation Mode
Final Prompt
Texl-image
"Please generate a scientific figure according to the following requirements: (generation query
Text-code-image
"Please generate scientific figure according the following requirements: (generation query).
Your output should he in [Python/Tikz]] code. Do not include any text other than the [Python/Tikz] code."
Table 8: Auxiliary instruction for constraining the model to generate scientific graphs in corresponding mode.
C GENERATION QUERY
Object Type
Query Template
Attribution
Generation Query
Understanding Dimension
A/An (object) with adan (color] bonder.
circk, blu
circle blue bonder.
Attribute
Two [coloc] spaces divide alan [ID-ohject] into four parts.
blue, pyramid
Two blue spazes divide pyramid into four parts.
Numeric
In bur chart, sám [soke--] }) bar to the right of olan [colos-2] har, leftooot the (alles//chetess) in chart.
Chart
bluc, orange, tallest
a bas chart, biue bar the right orange hor, and the lefheoot har the tallest the
Attribute, Numeric, Sputial
Graph theory reprosmtation
[number] podes.
12
hinary
Matrix
A [number 1)-by-[mambber
six-by-this manix.
Numeri
Real-iife object
There are Immberr- boxes on [number-2] 2| degree stope.
30
There are 4 boses on 30 degres slope.
Sputial
with [minher-2] polaaans, marked in the fins column.
index
3.5
tbble with three mws and five columns, The index is markedd in the flost
Attribute, Numeric, Sputial
Amnttziin
The English text (the name of the object) preposation) the (object].
the left of, triangle
The English text tthe nume of the object)
to the left of the trinnel.
Sponial
Function
[funcioon its
3"x
Numeric
Table 9: Illustration of constructing generation queries.
16

---

Published as conference paper at ICLR 2025
D
CODE OUTPUT
Llama Tikz Code
GPT-4o Tikz Code
doccmenntcl ass (standalone lone]
usepackage (tikz] b
documentccass standalone) )
\ usepackage tikk)
(beginddocusen)
b [scale2]]
[ilo[[[l11)rreen (0,
(begin(tikapicturg
-- (1,0) are (0:180:1) cycle;
$ Green cone
(11=wwE11 (1,0)
\fill [green] (0, 0) (-1,-2) (1, cycle;
(1.5.0.5) (1,1)
eyele;
\begin \begin [shiptt 0)
8 Brown ellipsoid \shade [ball color=brown]
2111=b=i1brr (0,
(0.5,0)
(0.5,0.5)
(3,-1) ellipse (1.5 and 1)
sycle;
\end[scope|
Nend (tikapiettur))
Nend ( tikzpicture
\end (ddccmenttt)
eend ( (document)
Figure TikZ code snippets of GPT-4o and Llama models for generating a cone and ellipsoid.
17

---

Published as conference paper at ICLR 2025
GPT-4o Python Code
Llama Python Code
matplotlib pyplet import numpy as up
hak.ktts.mllot..artt.. Import Poly3DCollection
mpl_ioolkits.
Import matplotlib pyplot
from mpl_toolkits www.kkitssmppls...... mplob3d import Axes3D import numpy ap
Create figure fig - p.....iuure.......
=igg.ade_aibpltt (111,
8 Create rigure and axia -igt. (figsize- 61) ax - su..add_uuppttt111
projestion='2d")
$ Derfng a funatian to create cone cceale_cone fax, radius, height, color):
projection-13d")
# Define the parsmeters of the cone cone_height
up ip.liispace 10, mp.pi, 100) radius np- lu)
cone_radius
y radlus nppsintu)
8 Define the parametera of the ellipsoid ellipaoid_x_rade e ell:paolo_y_radiin - 1,3
np. likeo_
verta - [1ist Y, 2111 aptts::ppedd:(00 C, height) Apex of the cone Poly@nCallection (verts,, color=color, alpha=0.6)
elLipsoid___radius
# Generate the vertices of the cone ip.1inspace
(verts,
np.pi, 100; 1p cone_height, 1001 x (cone_radius up.ourer tap, (G), sp.sin
ax co.ed_cclllec
Create function to plet ellipsoid def ozeate_elLipsoid
IX, fyr 12,
(cone_cadius op.ourer
color):
sp.sin
mp.linspace (0, 2 pi. 100)
(cone_height
np. (np. size (v))
np. mp.linspace 10, pt 100)
(cone_radivn
outeuter cop (u),
anp.alze (1)),
ap.sin
ap.cos(v])]
y optouter (i).
Plot the ax pl.t__e_urrrceel 2, color-1 green',,
ap.sin
ap.outer lap inp.cces_lceeeee.. (u),
sipha=0.5)
col
8 Cenerate the vertices of the ellipsoid 1p.
ax.plot_surfaee Ix, y. 2,
np.pt, 100;
color-color, 61pha-0.6)
sp..inepacee0,, 2 up.pi. 100:
Paramecezs for the shapes cone_height. ellipsoid_rx ellipsoid_ry 1.5
x
x_eilipsoid (elppixxeeell
(pp.outeer(ppccoo.tt))) ap. mp.
[pp.outer(ap.ciinll)) ap.sin
>_ellipseid (pp=oidd~[llllpp3000 (o11ppoot__pp__adddll
ellipaoid_r2
mp.cuter((p..oneee ap.eos(v)))
# Create and plot the create_cone (ax, ceme_radius, some_height, 'green')
Piot the ellipsoid
wttppecttaracaa_eellle_____ y_lllipsoid =_ellipsoid, holor-'browe alpha-0.3)
* Create and plet the ellipsoid create_allipooiddat
ellipsoid_
ellipaa'd_ry. ellipaotd_rz, 'brown')
8 Set the rraita of the axes ax ax.aet_xlim
Set piot 1imits ex.se:_xlim 51)
l- ex.set_alim 6)
ex.sor_ylimi 51)
ax.sat_2lim([e. 10])
i pisplay the piot
# Show the plot pht.show
plt.ahow
Figure 7: Python code snippets of GPT-4o and Llama models for generating cone and ellipsoid.
18

---

Published as conference paper at ICLR 2025
E
IMAGE EXAMPLES
Prompt
Tike
Elana_
Python
DALLE
A graat cons and brsnn allpsiid
Some circles are of different sizes and the
lerpod
stnighht a conriinntc system symmserica paiss,
matti.
into
blue, prom.
Them i sylinder on the inble. The sylin- der and the table are on the left side the
image. The up view and lett view the sylindle
image.
linglish
shapes
miangle interset.
panobelaa upening upwzeds.
Table 10: Comparison of Generated Images by Different Models from Various Prompts: Each column in this table presents the side-by-side comparison of images generated by the different models in response to the corresponding prompts. Results for each prompt are shown in each row, demonstrating the diversity of model approaches and styles. Image outputs for the first four columns are generated through the models' code, while the others are generated directly by prompting the models.
19

---

Published as conference paper at ICLR 2025
GPT-4o.Tikk Tike
Llams, Tika
Gyt--o_Python
Python
Autematike
Stable Diffisinn
DALLE
A uod tute iniik the inceuring cup, and petri dol ousede.
shudow which
panallel seaments endpoims clrcl. crdlpiinss back
C:
C: Coinkiann
R:
R:
There are ficylindes oz the right side of the
carnae and ltbes un the lover lefl sile.
R:
A small bell talls from sable 1.6 meters ligh and bounces to beight
Do
meten. The injestors of the balls fall and rebound is reprsemee by a light grorn dotted line:
gentagonal jeium 'sep image
pentagoral
their
Table 11: Failure Cases of Different Models: The examples give selected cases where both GpT4o models have an average correctness score < 2.5. The last four rows depict instances where all models have correctness scores 2.5.
Scores across Correctness (C), Relevance (R), and Scientific Style (S) are illustrated as star ratings.
20

---

Published as conference paper at ICLR 2025
Prompt
Llama_
GPt--4_Prtton
Lluma_ Python
Ausomatikz
DALLE
A ciangle with sides of equal leagth.
tnikkink R: tnkoklnk
tokkokkkk R: tnnooink
tninniinn R: dntooiin
tninoinn
tonnoinn
teiniin
R: tniioiink
A clicle with blue booder.
C: Cokkkokkk
CoAkkokkk R: R:ARR S:1e
C=ARkok R: EERAiinn SchRRROe
Co R: R= Sctiira
CoARbAik R: REARAA ScROOTT
CHArhhi R: #A S: SaRa
R: # S:tdhniiee
R: Rnniiintt
A rnd irangle and grore rindn
kokkiik STRREER
RaRniinn S.REAE
S:RRARR
black
Ca kokkkink
tninkinn 2:
*
SRERER
Table 12: Gold Standard Examples: The examples give selected instances where models have an average correctness score in the majority of cases. Scores across Correctness (C), Relevance (R), and Scientific Style (S) are illustrated as star ratings.
21

---

Published as conference paper at ICLR 2025
F EVALUATION GUIDELINE
F.1. CORRECTNESS
Score
Description
5
The image fully meets all the requirements with no mistakes.
4
The image meets the key requirements, with only minor mistakes.
3
The image meets some or half the requirements, with some mistakes.
The image meets only few of the text's ' requirements and contains serious mistakes.
1
The image fails to meet the requirements of the text.
0
No image content or compile error.
Table 13: Correctness Scoring Guideline
Score: 5
Score: 4.2
Score:
Score: 2.4
Score: 1.4
Prompt:
Prompt:
Prompt:
Prompt:
Prompt:
There 4 line seg- ments inside circle.
The diagonal of the square red and very thick.
There 5 squares the left side of the can- vas 3 circles on the right side.
The intersection of the circle and the square is filled with red.
The intersection of the circle and the square is filled with red.
Explanation:
Explanation:
Explanation:
Explanation:
Explanation:
There are 4 line seg- ments and they in- side circle.
The diagonal of the out- put image not visibly very thick. Other than that it's correct.
partially correct: The left graph shows rectangle instead squares. The right-side graph is correct.
The red-marked inter- section does not in- volve square. How- ever, there is red in- tersection and a circle.
There is square and intersection of two graphs. (There a cle and some red, but it's overall incorrectly displayed).
Table 14: Correctness Guideline Examples (Scores are averaged across multiple annotators)
22

---

Published as conference paper at ICLR 2025
F.2 RELEVANCE
Score
Description
5
The image contains redundant objects features.
image contains few redundant objects or features but remains highly relevant the text's requirements. The image contains some redundant objects and some required elements.
3
2
The image contains more redundant objects than required elements.
The overall image is not relevant to the requirements.
No image content or compile error.
Table 15: Relevance Scoring Guideline
Score: 5
Score: 4.6
Score: 3.4
Score: 2.2
Score: 1.4
Prompt:
Prompt:
Prompt:
Prompt:
Prompt:
The diagonal the A binary tree with to- tal of 7 nodes.
diagonal of the square is red and very thick.
There are tangent lines to a circle.
There are tangent lines to circle.
square is red.
Explanation:
Explanation:
Explanation:
Explanation:
Explanation:
Fully relevant, Additionally marked dundant objects or fea- tures.
One redundant diago- nal line square.
few redundant cir- cles and two dotted straight lines. Addi- tionally, there a red- dish color.
Redundant curves in- side circle. Over- all irrelevant image to the text. Also irrele- vant coloring.
blue color for tree nodes.
Table 16: Relevence Guideline Examples (Scores are averaged across multiple annotators)
23

---

Published as conference paper at ICLR 2025
F.3 SCIENTIFICNESS
Score
Description
5
The image be presented in textbook academic paper without any change.
4
The image minor issues and requires minimal adjustments presented lextbook academic paper.
3
The image has serious issues in scientific style, including mismatched sizes, unsuitable positions, overlapping graphs or text, incomplete graphs, etc.
2
The image's style not common in scientific settings.
The overall image more suitable a lifestyle context and not appropriate for scientific demonstration.
0
No image content or compile error.
Table 17: Scientificness Quality Scoring Guideline
Score: 5
Score: 4.2
Score: 3.2
Score: 1.8
Score: 1.4
Prompt:
Prompt:
Prompt:
Prompt:
Prompt:
A binary tree with a to- tal of 7 nodes.
M triangles and N cir- cles, where M is the same as N, and M and N are both larger than 2.
Three triangles of the same size but different shapes.
A binary tree with to- tal of 7 nodes.
M triangles and N cir- cles, where M is the N, and M and both larger than
Explanation:
Explanation:
Explanation:
Explanation:
Explanation:
perfect scientific im-
A scientific image, but the triangles and cir- cles are not arranged properly.
There many redun- dant lines around the triangles, which nega- tively affects its demon- strative purpose.
The representation of nodes and colorful edges are uncommon scientific images. Also, the quality (in terms of resolution) of the image is not what one would expect modern science.
The shapes are not strictly triangles and circles purposes. The image is closer to lifestyle. Also, the quality (in terms of resolution) of image not what one would expect in modern science.
age.
scientific
Table 18: Scientifieness Guideline Examples (Scores averaged across multiple annotators)
24

---

Published as conference paper at ICLR 2025
G EVALUATION OF SUCCESSFULLY COMPILED IMAGES
Scientific Style
Compile
Models
Correctness
Relevance
Error Rate
Automatikz
2.14
2.41
3.50
0.04
Llama_tik tikz
2.49
2.72
3.66
0.29
GPT-4o tikz
3.84
4.02
4.10
0.09
Llama python
2.92
3.52
4.42
0.28
GPT-4o python
3.76
3.64
4.21
0.07
Stable Diffusion
2.19
2.09
1.96
DALL-E
2.16
2.00
1.55
Table 19: Overall Model Performance, averaged across all generated images (not including compile error cases), and
two annotators.
Criteria
Correctness
Relevance
Scientific style
Language
EN
DE
ZH
FA
EN
DE
ZH
FA
EN
DE
ZH
FA
Llama tikz
2.34
2.68
2.14
1.75
2.72
3.23
3.00
2.39
3.47
4.05
4.00
4.14
GPT-4o tikz
4.05
4.24
4.18
4.32
4.24
4.45
4.84
4.68
4.32
4.66
4.63
4.68
GPT-ol tikz
4.66
4.59
4.50
4.76
4.68
4.75
4.82
4.91
4.63
4.84
4.74
4.76
Alama_pythoo
3.88
2.08
2.19
1.97
4.15
2.35
2.50
2.11
4.92
3.85
3.88
3.67
GPT-4o _eThhooppython
3.75
4.15
4.13
4.09
3.72
4.18
4.23
3.94
4.31
4.50
4.83
4.53
GPT-ol_pythhn
4.28
3.83
4.32
4.00
4.10
3.83
4.13
4.00
4.50
4.53
4.53
4.50
Qwen2.5_pyther
3.10
2.30
2.05
2.40
3.08
2.48
2.25
2.53
3.70
3.43
3,28
3.68
DALL-E
1.98
2.15
1.83
1.93
1.88
2.03
2.03
2.00
1.40
1.58
1.53
1.50
Average
3.50
3.25
3.17
3.15
3.57
3.41
3.47
3.32
3.91
3.93
3.92
3.93
Table 20: Multilingual performance of compiled images (compile errors from generation are included) The highest values across languages are highlighted in bold.
Attribute & Numerical & Spatial
Attribute & Numerical
Attribute & Spatial
Numerical & Spatial
Model Attribute
Numerical
Spatial
Automatikz
2.70
2.01
1,80
2.35
2.04
2.17
1.88
Llama_tike
2.83
2.75
2.15
2.56
2,45
2.39
2.33
GPT-4o tikz
4.11
3.99
3.54
3.9
4.03
3.77
3.54
Alama_python
2.99
3.72
2.5
3.31
2.94
2.36
2.56
GPT-4o GPT--4oppython
4.12
4.05
3.81
3.74
3,93
3,44
3.33
Stable Diffusion
2.75
1.73
2,06
2.41
2.46
1.96
2.11
DALLL-E
2.68
1.77
2.13
2.36
2.31
1.94
2.07
Table 21: Correctness evaluation within each understanding category (compile errors are not included).
Function&
Model
shape
3D shape
Chart
Graph theory
Matrix
Real-life object
Table
Annotation
Coordinate
Automatikz
2.60
1.60
1.85
2.53
2.07
1.88
2.00
1.50
2.00
IJama__ikk
3,10
1.93
1.96
1.00
2.75
1.71
1.75
2,00
1.94
GPT-4o tikz
4.13
3.40
4.11
4.26
4.00
3.28
3,56
3.88
3.63
Llamaapython python
3.01
2.21
3.95
0.00
2.50
2.90
1.50
3.33
2.50
GPT-4o python
4.28
3.47
3,31
3.63
3.57
3,42
3.25
3.88
3.20
Stable Diffusion
2.08
2.43
2.11
1.43
1.56
3.12
1.94
1.63
1.75
DALL-E
2.08
2.47
1.86
1.25
1.50
3.24
2.13
2.13
1.58
Table 22: Correctness Evaluation within each object type (compile errors not included). The column-wiss highest is highlighted bold.
25

---

Published as conference paper at ICLR 2025
H AUTOMATIC EVALUATION METHODS
We also test how well recent automatic text-to-image evaluation metrics correlate with our human judgements. In specific, we test the metrics CLIPScore (vit-base-patchl 66 and vit-large-patch (Hessel et al. (02T). ALIGNScore (Saxon et al. 2024) and PickScore (Kirstain et al. 2023 , as well as the multimodel LLM based metric DSG (Cho et al. 2024) with Gemma-2-9B-Shnn shows the resulting Kendall correlations on a per-image granularity. The highest correlation is reached by PickScore that was trained on large-scale dataset of human preference labels for generated images. However, the human Kendall correlations for this task, with 0.75 (correctness), 0.68 (relevance) and 0.62 (scientificness) are much higher. This suggests the need for more suitable evaluation metries in the domain of scientific text-to-image generation.
(Team et 2024 Meng et al. 2024) for question generation. Table 23
Correctness
Relevance
Scientificness
DSG (Gemma2)
0.18
0.15
0.02
CL.PPcore
-0.00
0.01
0.04
CI CI.PSScoreLaree
0.03
0.03
0.04
AlignScore
0.23
0.21
0.09
PickScore
0.26
0.23
0.15
Table 23: Kendall correlation between automatic metries and human scores.
LIMITATIONS
We that AUTOMATIKZ may have been bad especially because it was trained on textual descriptions taken from captions of scientific papers, which may look substantially different from the instructions used in ScImage. For consistency, did not develop model specific prompts, however. This holds more generally: while prompting is known to have (sometimes substantial) effect on model performances (Mizrahi et al.) 2024 Leiter et al. 2023 our study used one and the same prompt across all models.
One interesting avenue to explore in future work is the combination of heterogeneous LLMs, as we saw that models have complementary strengths and limitations. We finally note that sample sizes for some the scientific objects considered and for the multilingual evaluation were comparatively small. This means the corresponding results need to be interpreted with caution.
Ethically, there is that risk that naive scientific users may place unwarranted trust in the output generated by some of the models, e.g., GPT-40, without assessing whether the generated output confirms with their expectations or their prompted input. The human user has to take full responsibility for the outputs created by the models explored in our work.
26


Reviews:
[{"review_id": "0Q0C5dwNbr", "reviewer": "Reviewer_hHH2", "summary": "This paper introduced a benchmark, ScImage, for evaluating models for generating scientific images from textual descriptions. This paper thoroughly investigated the existing datasets and benchmarks for scientific image generation, and pointed out there is no systematic evaluations for scientific text-to-image models. This paper evaluated 5 state-of-art models with 2 generation modes: text-image and text-code-image, while text-code-image includes TikZ and Python as coding language. The benchmark emphasizes comprehension along three dimensions: numeric, spatial and attribute. It contains a 404 hundreds queries in 101 templates. Human evaluation is based on correctiveness, relevance and scientific style. It is good including agreements analysis of the human annotations to make sure human evaluation is reliable. The 5 models are compared over these criteria, and compilation failure is pointed out as a drawback of text-code-image models. Detailed analysis is made on types of understanding, object categories, multilingual prompts, and different programming languages. A qualitative analysis is conducted and highlighted some typical failures about understanding physical world. The findings underscore the further exploration on generating scientific images with LLMs. The paper summarized some limitations, such as not utilizing chain-of-thought, not developing automatic metrics and potential ethic issues. \nOverall, the paper is well written, and has the potential to contributes a new benchmark for evaluating models capability to generate scientific image, which enables more comprehensive analysis towards models, especially LLMs.", "questions": "Data Construction:\n1. Can you provide a detailed explanation of the rationale behind constructing the dataset solely from generated prompts based on templates? What considerations were made regarding the lack of ground truth in this dataset? Additionally, have you considered using real-world scientific data to construct the prompts, which would provide a more robust basis for evaluation?\n2. Given that the dataset is relatively small, with only around 400 entries, how do you justify its representativeness and reliability compared to other related datasets that typically contain tens of thousands of examples?\n3. You mentioned that you are only using the most commonly encountered types listed in the taxonomy. How did you decide which ones are the most commonly encountered?\n\nHuman Evaluation:\nYou mentioned that automated evaluations can be unreliable, particularly in recognizing precise directions in text and images. Can you elaborate on the specific limitations of current automated metrics, especially in the context of evaluating scientific graphs? Why do you believe that a single bad case is sufficient to justify the exclusive reliance on human evaluation?\n\nEvaluation Guideline:\nI suggest revising the evaluation guidelines to recommend using the same prompt for generating images with different scores, rather than using different prompts for different scores. How would you justify this approach, and do you believe it could lead to a more reliable assessment of the generated images?\n\nResult Analysis: \nIn the results section, there is a description of the experimental data but a lack of in-depth analysis of the results. Could you clarify the reasoning behind the evaluation results? How do you believe that this level of analysis contributes to the significance of your findings and their applicability to future research in this area?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Originality: The paper presents a potentially valuable contribution by introducing a new benchmark, ScImage, specifically focused on evaluating scientific image generation from text descriptions. While benchmarks for general text-to-image generation exist, few are tailored for scientific image generation, where precision in spatial, numeric, and attribute comprehension is critical. This benchmark addresses a gap in the current evaluation landscape, particularly for multimodal LLMs.\n\n2. Quality: The paper presents a clear structure and coherent organization. While it contains several commendable elements, there are areas where improvement is needed to enhance the overall quality of the research. Addressing these aspects could significantly strengthen the findings and their implications.\n\n3. Clarity: The paper is clearly structured, with each section flowing logically into the next. The figures and examples enhance comprehension by illustrating key points effectively.\n\n4. Significance: The benchmark’s potential to foster progress in scientific image generation could be significant, as this area has notable applications in research, education, and communication. By evaluating models on the specific demands of scientific image generation, such as spatial accuracy and numeric precision, this benchmark could aid in developing more capable multimodal models tailored to scientific needs.", "weaknesses": "There are some problems, which must be solved before it is considered for publication. If the following problems are well-addressed, I believe that the essential contribution of this paper are important for evaluating LLMs' capability of text to image generation.\n1. Your manuscript needs careful editing and particular attention to spelling, for example, in Figure 7, \"Llame\" should be \"Llama\".\n2. Concerns with Dataset Construction:  I question the methodology used in constructing the dataset. The dataset appears to be generated from templates, producing a variety of prompts but lacking ground truth. Additionally, the dataset's size is relatively small, with only around 400 examples, which contrasts starkly with related datasets mentioned in the paper that typically consist of tens of thousands of examples. This raises concerns about the quality of the generated prompts and whether the dataset is sufficient to adequately evaluate the model. I suggest to add a quality analysis to the dataset. For example, evaluating a subset of the dataset with domain experts to make sure the generates prompts make sense, and performing some analysis to ensure the diversity and representativeness despite of the small size of dataset.\n3. Concerns with Evaluation Approach:  I am also skeptical about the evaluation methodology, as it relies solely on human assessments. The authors justify this choice by referencing limitations of automated evaluation metrics, particularly in the context of scientific figures where metrics like CLIP Score and FID Score may fall short. For example, they mention issues noted by the MMVP benchmark with automated evaluations, especially in accurately identifying specific directions like \"up\" and \"down.\" However, citing one bad case is insufficient to conclude that automated evaluation is unreliable. I would suggest to provide more detailed and comprehensive reasoning for the necessity for human evaluation over automated metrics in this specific context. \n4. Lack of Depth in Result Analysis:  The results section primarily describes the scores obtained by each model but lacks a deeper analysis of these results. I would suggest to add the possible explanation to the observed phenomenon, the underlying problem of the models, and how the results indicates the directions for improvement. That could offer valuable insights for future research, enhancing the paper’s overall impact and utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sTryorvJyZ", "reviewer": "Reviewer_4MC8", "summary": "This work presents a novel benchmark dataset designed to evaluate the multimodal capabilities of large language models (LLMs) in generating scientific images from textual descriptions. The dataset encompasses a wide variety of formats, including 2D/3D shapes, charts, graph theory representations, matrices, tables, and more, with a focus on assessing spatial, numerical, and attribute comprehension. The evaluation process involved 11 scientists who reviewed the performance of models such as GPT-4, LLaMA, and Stable Diffusion.\n\nHowever, the dataset primarily consists of query templates, which limits its applicability to other models, as the evaluation heavily relies on human (scientist) assessments. Furthermore, the technical details provided are not sufficiently rigorous, with multimodal models and Stable Diffusion collectively categorized as large language models, resulting in imprecision and lack of clarity. Additionally, several relevant prior works are inadequately cited or discussed, which detracts from the comprehensiveness of the study.", "questions": "1. How does this benchmark dataset differ from ChartMimic, and why should it be discussed extensively in the main body?\n2. How can this benchmark be used to evaluate other models? Is evaluating only GPT-4 and LLaMA sufficient, or should more multimodal LLMs be tested?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This dataset includes a wider variety of image types compared to previous work that primarily focused on charts and bar graphs.\n2. The evaluation was conducted by 11 scientists, ensuring high-quality assessment. However, this dependence on human evaluation limits the benchmark dataset's scalability for testing additional models.\n3. The study also evaluates various textual description languages and coding languages/packages, making the analysis comprehensive.", "weaknesses": "1. The paper lacks a thorough introduction to related work on the interpretation of scientific images, such as [1, 2, 3]. Furthermore, the distinctions between this work and ChartMimic remain unclear. To enhance clarity, Table 8 in the appendix should be included in the main body to better illustrate how this study compares to previous research.\n\n2. The dataset primarily consists of query templates, which limits its generalizability for evaluating other models due to the heavy reliance on human (scientist) assessments. \n\n3. Given the focus on scientific images, it is crucial to evaluate the model’s understanding of domain-specific scientific knowledge embedded within these images, rather than solely assessing spatial, numerical, or attribute-based features (e.g., color, size, shape). Images from domains such as biology, chemistry, physics, and materials science contain nuanced, domain-specific information that should be considered to provide a more complete assessment.\n\n[1] Roberts, Jonathan, et al. \"SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation.\" arXiv preprint arXiv:2405.08807 (2024).\n[2] Li, Zekun, et al. \"Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension.\" arXiv preprint arXiv:2407.04903 (2024).\n[3] Wang, Zirui, et al. \"Charxiv: Charting gaps in realistic chart understanding in multimodal llms.\" arXiv preprint arXiv:2406.18521 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SSv6b8uzp5", "reviewer": "Reviewer_prga", "summary": "This paper introduces a new benchmark, ScImage, which evaluate models abilities of scientific text-to-image generation, in both code-based and multimodal modes. This benchmark is multilingual, containing 4 languages. This benchmark emphasizes models' spatial understanding, numeric understanding, and attribute binding abilities. The authors use human evaluation for images generated by 5 models, where gpt-4o outperformed all other models.", "questions": "- How should people evaluate other models' performances on this benchmark without needing human evaluators? For example, using image-image similarity between the generated model output with a oracle image for each task could provide a basic idea on how well the model output is. Additionally, llm-as-a-judge could potentially evaluate the model outputs as well.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors introduce ScImage, which contributes to evaluation of models' scientific image generation capabilities.\n- ScImage includes a wide range of tasks, which could provide comprehensive evaluation of models' image generation abilities.\n- The authors provide detailed query construction process and templates, which could help creation of new queries.\n- The authors provide example queries and example outputs of models, which facilitates understanding the benchmark.", "weaknesses": "- The authors use human evaluation to assess models' performances on this benchmark. It is good to know how humans rate the outputs, but it is less scalable if using human evaluation. If other people would like to test their models on this benchmark, they would need to also hire professional human evaluators, which would be expensive given that this benchmark contain more than 400 instances. It would be good if the authors could explore automated metrics that have high correlation with human evaluation results, or provide a crowdsource platform where human eval could be less costly, or use llm-as-a-judge methods to automatically evaluate model outputs.\n- The authors provide limited number of models' results on this benchmark. It would be good to see how other open source models perform on this benchmark, such as", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)