MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science
Authors: Erle Zhu
Yadi Liu
Zhe Zhang
Xujun Li
JinZhou
Xinjie Yu
Minlie Huang
Hongning Wang
Venue: ICLR 2025 Poster (2025)
Keywords: multi-modal reasoning
scientific reasoning
physical simulation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: improving multi-modal scientific reasoning capability with physics perception model and simulation assistance

Abstract: Pre-trained on extensive text and image corpora, current Multi-Modal Large Language Models (MLLM) have shown strong capabilities in general visual reasoning tasks. 
However, their performance is still lacking in physical domains that require understanding diagrams with complex physical structures and quantitative analysis based on multi-modal information. 
To address this, we develop a new framework, named **M**ulti-Modal Scientific Re**A**soning with **P**hysics Perception and **S**imulation (**MAPS**) based on an MLLM. 
MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. 
The PPM module is obtained by fine-tuning a visual language model using carefully designed synthetic data with paired physical diagrams and corresponding simulation language descriptions. 
At the inference stage, MAPS integrates the simulation language description of the input diagram provided by PPM and results obtained through a Chain-of-Simulation process with MLLM to derive the underlying rationale and the final answer. 
Validated using our collected college-level circuit analysis problems, MAPS significantly improves reasoning accuracy of MLLM and outperforms all existing models. 
The results confirm MAPS offers a promising direction for enhancing multi-modal scientific reasoning ability of MLLMs. 
We will release our code, model and dataset used for our experiments upon publishing of this paper.

Published as a conference paper at ICLR 2025
MAPS: ADVANCING MULTI-MODAL REASONING IN EXPERT-LEVEL PHYSICAL SCIENCE
Erle Zhu! Yadi Liu", Zhe Zhang?, Xujun Li Jin Zhou?,
Xinjie Yu², Minlie Huang!.2 Hongning Wang 'The Conversational AI (CoAl) Group, *Deparmeent of Computer Science Technology Department of Electrical Engineering
Tsinghua University
se12 @mailmml ils tsinghua edu. en, hw-aid wwwttiinnuuua...mmmmm.. edu. cn
ABSTRACT
Pre-trained on extensive text and image corpora, current Multi-Modal Large Lan- guage Models (MLLM) have shown strong capabilities in general visual reason- ing tasks. However, their performance still lacking in physical domains that require understanding diagrams with complex physical structures and quantitative analysis based on multi-modal information. To address this, we develop a new framework, named Multi-Modal Scientific ReAsoning with Physics Perception and Simulation (MAPS) based on an MIL.M. MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simula- tor. The PPM module is obtained by fine-tuning visual language model using carefully designed synthetic data with paired physical diagrams and correspond- ing simulation language descriptions. At the inference stage, MAPS integrates the simulation language description of the input diagram provided by PPM and Ohaii-offSSmmulatinn underlying rationale and the final answer. Validated using our collected college- level circuit analysis problems, MAPS significantly improves reasoning accuracy of MLLM and outperforms all existing models. The results confirm MAPS of- fers promising direction for enhancing multi-modal scientific reasoning ability of MLLMs. Our code is available at https://giluub.com//uuccoo
results obtained through Chain-of- Simulation process with MLLM to derive the
INTRODUCTION
Pre-trained on large-scale text and image corpora, Multi-Modal Large Language Models (MLLM) exhibit strong capabilities in general visual reasoning tasks, including image captioning and visual question-answering (Li et al., 2022: Team et al., 2023; AI; Liu et al., 2024). Through elaborated pre-training and post-training, the proficiency of LLMs in text-only mathematical reasoning and programming has significantly improved (Hendrycks et al., 2021; Lu et al., 2022: Lightman et al., 2023), broadening their applications to more scientific and professional tasks. However, for scien- tific disciplines that require understanding complex physical structures images and mathematical reasoning based scientific knowledge from multi-modal information, the capabilities of MILMs remain weak (Yue et al., 2023). This limitation hinders their further application in educational, aca- demic, and industrial scenarios. Thus, enhancing the multi-modal reasoning abilities of MLLMs in expert-level physical sciences while extending their application scenarios is valuable yet challeng- ing research direction.
The current methods in mudti-modal reasoning (Zhang et al., 2023b: Zheng et al.. 2023; Mitra et al., 2024) primarily concentrate on generating rationale that integrates multi-modal information, al- lowing the model to derive the final answer from this intermediate result. This process is commonly referred to as Chain-of -hain-hf-Thouggt (CoT) (Wei et al., 2022) reasoning. Another commonly adopted pathway is to integrate LLMs with external tools, including small-sized specialized multi-modal models, as well as software such as code interpreter (Gao et al., 2023; Wang et al., 2024a). However,
ccorresponding author

---

Published as a conference paper at ICLR 2025
Emsages
Issues in Multi-Modal Reasoning for Scientific Scenarios
dimppe
edberua poluf
Perception of Physics Scenario
Quantitative Analysis
ineaeaee, di (ebiee))s Crreakr 190 (N
de renater
Touur Aedlysti
Dx. 5 tewetie
Drace Peroption
broelcdge Mothenation
Three aptinnga moss are atocche a rigl,, gesss cimage 2o. rind the insquancy system.
Image:
Benogrihen
44g bediaaste PC pakre
Semariic
Recogrttonn
Ratefionshi
Inkerie pular pelrt
C wair at 20nC Soea
Emsage:
Assist
Precise
langh betnean pe
Complex
pfrysical
Description exing Language
Language
Model Physics Scasarte
Mathernticaal
rearame craiee comeecee
Physics Perception Model
Physical Sinulator
lighes Wnd in
Solutions
Two Issues in Multi-Modal Reasoning for Scientific Scenarios and Our Solutions. The location of the model error is highlighted in red. The scientific questions are sampled from MMMU (Yue al., 2023).
Figure 1:
these methods mainly focus on general images or diagrams containing simple physical information, making it difficult to directly transfer to scientific scenarios that involve complex physical diagrams and require precise numerical analysis.
To address the aforementioned limitations, we proposed Multi-Modal Scientific ReAsoning with Physics Perception and Simulation (MAPS), a novel framework for solving complex multi-modal reasoning problems physical disciplines. And in this paper we verified the effectiveness of MAPS in electrical discipline, which typically involves multiple circuit diagrams and is representative of expert-level physical science requiring reasoning on complex physical diagrams. The core idea of MAPS is to decompose expert- reasoning problems into two sub-tasks: understanding the phys- ical diagram and reasoning based on this comprehension and related physical knowledge. MAPS realizes physical diagram understanding by fine-tuning a visual language model using carefully de- signed synthetic data, resulting in what we term the Physies Perception Model (PPM). The role of PPM is translate physical diagrams into simulation language descriptions that can be executed by simulator. At the inference stage, MAPS integrates the converted simulation language de- scription and their respective simulation results, obtained through derive intermediate rationales and ultimately the final answer to the question. Experiment results on college-level circuit analysis problems demonstrate that framework successfully address the challenges in complex multi-modal reasoning tasks in physical science. Most importantly, it sig- nificantly reduces the occurrence of hallucination when using and solving physical equations. This advancement creates new avenues for precise multi-modal scientific reasoning using MLLMs.
process, to
To summarize, our main contributions in this work follows:
We introduce a novel multi-mode modal reasoning framework MAPS to address the current lim- itations of MILMs in solving expert-level scientific problems involving complex physical diagrams. MAPS incorporates MLLMs with a finetuned perception model and physical simulator improve the precision of its reasoning steps and results.
Through experiments college-level circuit analysis problems, we demonstrate that MAPS significantly outperforms existing methods, offering a viable pathway to build multi-modal solutions for expert-level scientific problems.
We devise an automated pipeline to synthesize diverse paired training data for finetuning an MLLM. By leveraging intrinsic generalization ability of pre-trained models, the pipeline helps MLLMs effectively adapts complex real-world problems, alleviating the issue of data scarcity in scientific domains.
2

---

Published as a conference paper at ICLR 2025
2 MOTIVATION
Following the human approach to solving science problems with diagrams, we break down the problem into two steps: understanding the physical context in the multi-modal input (Perception) and using scientific knowledge and mathematical deduction to derive the answer (Analysis). Based on these two steps, we summarize the limitations of current MLLM-based solutions for solving such problems into main categories:
Issues in Perception. Based on observations reported in the MMMU benchmark (Yue et al., 2023) and our empirical studies, we found that current general purpsse MLLMs, including the most power- ful ones such as GPT-4V and Claude-3.5, exhibit poor perception abilities in understanding diagrams related to physical sciences (e.g., circuit diagrams). This corresponds to the perceptual error ident-- fied in the error analysis in (Yue et al., 2023). This significantly limits their application in the field of scientific reasoning with multi-modal input.
Issues in Analysis. Although MLLMS can sometimes correctly understand diagrams, their domain knowledge and mathematical reasoning abilities can still be lacking. This often leads hallucina- tions during further reasoning steps, resulting in misleading answers.
We offer some specific cases in Figure illustrating how an off-the-shelff MILM makes mistakes in perception and analysis steps. Based on these observations, we decide to decompose this complex multi-modal reasoning task into sub-tasks and leverage expert models and domain-specine tools to solve the sub-tasks that are infeasible for current MLLMs. Concretely, as shown in Figure 1, our proposed two solutions to the two issues mentioned above are:
Solution to Issue in Perception: Translate physical diagrams into simulation language descrip- tions. We adopt simulation language for two reasons. First, it describes the physical scene of the diagram using formal language, enabling the language model to directly access the fundamental structure behind the question. Second, with parameters of physical objects provided, we can directly use simulator to obtain all states and observation values of the physical scene. In the context of cir- cuit analysis, we use SPICE (Nagel, 1975) as our simulation language. For other scenarios, there are corresponding choices, such ANSYS APDL (Kohnke, 1982) in mechanical disciplines and ZPL (Laikin, 2018) in optics domains. Specifically, we develop an expert visual language model to com- plete this conversion. Since there is no large-scale available dataset or existing model for this task, we devise a data synthesis pipeline to generate abundant physical diagrams and their corresponding simulation languages for our visual language model training.
Solution to Issue in Analysis: Reasoning under the assistance of simulation. Although current MLLMS can perform mathematical reasoning using external tools (Chen et al., 2022; Zhou et al., 2023), recent research found it still challenging to prompt LLMs to write programs for solving sci- entific problems (Tian et al., 2024). In the benchmark evaluating real-world scientific programming tasks, even the best model achieves accuracy of less than 10% in completing main problem. To address the issue of hallucination when MILMS perform mathematical derivations and synthe- size scientific programs, tool, namely a physical simulator. The simulator comprises domain-specifie knowledge and thus is guaranteed to be precise in its output with respect the given input.
delegate the main quantitative reasoning tasks to Nomain-specifer
Combining the two solutions above, we design a Chain-of-Simulation process that obtains sim- ulation language description and simulation results utilizing the fine-tuned perception model and simulator, and prompt an MI.LM to compute the answer under the assistance of simulation lan- guage description and simulation results at the inference stage.
METHODOLOOGY
Our proposed MAPS framework, illustrated in Figure 2, consists of two phases: the Physics Per- ception Model (PPM) construction phase and the Inference phase.
The core components of our framework are follows:
Physics Perception Model (PPM). It serves as an expert perception model that translates given physical diagram into a simulation language (SL) description. This model is fine- tuned from pre-trained Visual Language Model (VLM) using a synthetic dataset designed for the diagram-to-SI conversion.
3

---

Published as a conference paper at ICLR 2025
PPM Construction Phose
(b) Inference Phose
Eirea lemobiter
PPM
PPN
Synthesis
Samp
Morting Languago
Step Chain-of Simulation
Apple Prongt Troplate
Diggian Layost
Ar andrctes came Bi pepeetenn wor in the citul Mayaom SPICE wehuoredd
ne and pown Eh of dhe curet sorm f.lr mnendrgl
0"
Phycica
diiudattion -idid Aided
PAW Tratsing
Ressoring
Figure 2: Our proposed MAPS framework is integrated with Multi-modal Large Language Model (MLLM), a Physics Perception Model (PPM) and a Physical Simulator. (a) At PPM Construction Phase, we fine-tune pre-trained VLM with carefully designed synthetic data obtain PPM which can convert physical diagram into simulation language descriptions.
(b) At Inference Phase, we acquire simulation language description and simulation results which assist MLLM with the further reasoning to obtain final answer of original problem.
apply Chain-of-Simulation
Physical Simulator. The simulator is used to perform numerical simulations and obtain the state and observations about the physical scene carried in the diagram.
Multi-modal Large Language Model (MLLM). The MILM primarily handles semantic understanding and basic mathematical reasoning, based on the results provided by PPM and the simulator. When solving physical problems with diagrams, the MLLM parses the target problem from the textual question, refines the simulation language description generated by the PPM, extracts useful simulation results, and performs final reasoning based on the original question and the added simulation information.
3.1 INFERENCE PIAASE
We first introduce the inference phase because it conveys the main philosophy of our solution. As -f-imultinm and Simulation-Aidet
shown in Figure 2(b), this stage includes two steps: Reasoning. Suppose we have scientific problem with a physical diagram X in pixel format and textual description Xr., our model required to infer the answer Yr.-
3.1.1 CHAIN-OF-SIMULATION
The first step in the Chain-of-Simulatie (CoS) process is to use the PPM to con- vert the pixel schematic diagram Xv into an initial SL description Z. Since the problem involves multi-modal informa- tion, the initial SL description produced by the PPM may lack completeness de- picting the full physical scene. For exam- ple, circuit diagram, a resistor might be labeled as R1 but its value might be provided in the accompanying textual de- scription in the question X,.- To address this, we employ the MILM to refine ini- tial SL description based on textual input Xt. The MLLM incorporates additional information from the accompanying text, resulting in a comprehensive and accurate SL text that fully describes the physical scene.
Algorithm 1 MAPS: Inference Phase
1: Input: Xv, X,,, PPM, Simulator, MLLM
2: Output: Yr.
Chain-of-Simllatie
3: Obtain SL description Z PPM(Xv)
4: Refine SL description
Z t- MLLM(XX, Z, ww.porpt.rrffeee
5: Obtain simulation result R + Simulator Z))
Simulation-Ai
Reasoning
6: if valid(()) then
7: YL < MLLM(X L, Xv. Z, R. prompt.sar)
8: else
Yg <- MLLM(X,, Xv, prompt s1)
10: end if
11: return Yz
Once the comprehensive SL description Z is generated, it is fed into the physical simulator to per- form physical simulations. This process produces simulation result R, including state values and observation values of the physical scene. This approach effectively mitigates mathematical reason-
4

---

Published as a conference paper at ICLR 2025
ing errors that may arise from the model's hallucinations in scientific computation, ensuring accurate and reliable results.
3.1.2 SIMULATION-ALE AIDED REASONING
After the CoS process, MAPS will apply the question information (Xz, Xv), SL. description Z, and simulation result R. to well-designed prompt template. This template prompts the MLLM to generate further rationale and infer the final answer Yz. We consider the simulation language and simulation results as intermediate rationale in the model's reasoning process, similar to various Chain-of- Thought (CoT) mechanisms in existing prompting methods (Mitra et al., 2024; Zhang al., 2023b; Zheng al., 2023). The entire process is illustrated using pseudo-code in Algorithm 1. Experiments show that under the assistance of CoS, the MLLM can be prompted to accurately answer questions, effectively narrowing the gap between current MLLM capability and expert-level performance on scientific problems.
3.2 PPM CONSTRUCTION PHASE
Accurate conversion from pixel schematics to simulation language descriptions is crucial for the CoS to function effectively. We highlight this importance with red star in Figure 2, emphasiz- ing the significance of PPM in our framework. Due the scarcity of real-world paired data that maps physical diagrams to simulation language descriptions, which is crucial for training Vision- Language Models (VLMs) recognize the physical diagrams of interest, we choose synthesize the paired data. To achieve this, we craft rules to generate large dataset comprising diverse circuit diagrams and their corresponding simulation language descriptions. These synthetic data are then used to fine-tune the pre-trained VLM, ultimately producing the PPM.
3.2.1 DATA SYNTHESIS
The data synthesis process is depicted in Figure 2(a). And the detailed steps of a generation process are described as follows.
The diagram layout is our data structure designed to correspond to the plotting language, encompass- ing all the physical objects, their displayed positions and annotations in the diagram. Subsequently, the pipeline synthesizes the diagram and the corresponding SL description through two paths: the diagram synthesis path and the simulation language (SL) synthesis path.
Diagram synthesis path. As shown in the upper branch of Figure 2(a), the diagram layout is first converted to plotting language. There are various plotting languages available, such as LaTeX (TikZ) and Graphviz, which use formal syntax describe diagrams and can be compiled into pixel images. The design of diagram layout allows for a straightforward transformation from diagram layout plotting language. Finally, we compile the generated plotting language using its designated compiler to generate the diagram in pixel format.
SL synthesis path. This path focuses distilling the physical structure from the diagram layout ing physical knowledge. Operationally, we apply physical rules to the diagram layouts to derive the intrinsic physical model, which contains only abstract physical objects and the functional relation- ships between them. For example, in circuit diagrams the physical structure can be formulated using netlist model (Nagel, 1975), which includes all components along with their types, parameters, and topological connections. In mechanical scenarios, the physical structure can be described using a FEM (Rao, 2010) model represent the mechanical system. Eventually, the physical structure formatted into simulation language.
This process produces both physical diagram and its corresponding simulation language descrip- tion. Since each step of the generation procedure involves random sampling, large number of diagram with different objects, spatial relationships and annotations can be generated through suffi- cient sampling.
3.2.2 PPM TRAINING
The training goal of the Physics Perception Model (PPM) is to generate the corresponding SL de- scription from a given diagram. We use a decoder-only pre-trained visual language model as the base model for the PPM. In practice, the training loss during PPM fine-tuning is the average neg- ative Maximum Likelihood Estimation (MLE) loss (Bishop & Nasrabadi, 2006) over the synthetic data.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We evaluate our MAPS framework through extensive experimentations on real-world scientific prob- lems. Given the substantial workload involved in constructing and validating the entire pipeline, we have limited our initial verification to the circuit analysis scenario, which is generally believed very difficult for state-of-the-art MLLMs (Yue et al., 2023).
4.1 IMPLEMENTATION
In this section, we describe our implementation of MAPS framework in the circuit analysis scenario.
Synthesis of Training Data for PPM. In the context of circuit diagrams, the diagram layout is de- fined as the planar grid and components connected between the grid nodes, with the values or labels annotated alongside the component symbols. The grid structures of synthetic data are randomly sampled from a predefined hierarchical distribution, ensuring the diversity of shapes, components, and annotations in the generated circuits. We use CircuitTikz as our plotting language to draw the circuit diagram using a LaTeX compiler. Since the component annotations in the real-world dia- grams can be in a numerical format (e.g. 1022) or a label format (e.g. Rí), we generate two types of circuit diagrams to cope with this variation accordingly: (1) Numerical -ttere eircuit, where the value is annotated on the diagram. The PPM is required to infer both the type and value of the components; (2) Label-type circuit, where only the labels of the components are provided in the diagram. The PPM predicts the type and label of the components, with an <Empty> token in the value position.
The physical structure of a circuit diagram can be represented using by netlist model (Nagel, 1975; Tao et al., 2024), which is a directed graph where each node represents an equipotential point, and each edge represents a circuit component. The SL synthesis step involves writing rules to automatically identify equivalent circuit nodes using basic physical properties and converting grid information into a netlist. We utilize SPICE (Nagel, 1975) as our simulation language for circuit analysis problems. The syntax of SPICE is based the netlist model into SPICE (Nagel, 1975) program at the end of each generation process. Please refer Appendix C.1 for our design of the hierarchical distribution and detailed illustration of the synthesis process.
netlist model, allowing us directly translated
We name our synthetic data ppm-synnlppp as our current data synthesis process only supports the generation of Linear Pure Resistive Circuits (LPRC) (Svoboda & Dorf, 2013). contains 20k pairs of synthetic circuit diagrams and their simulation descriptions, divided into train- ing, validation, and test sets in a ratio of 8:1:1.
PPM Training, For the training of PPM, we adopt CogVLM-I7B (Wang et al., 2023a) as the base model of PPM. The PPM is fine-tuned to generate the SPICE description for given the circuit dia- gram. For our detailed settings, please refer to Appendix C. Based on our preliminary experiments, the base model is largely unable to accurately perform the conversion task for most circuit diagrams when using prompting methods. Therefore, the training stage is essential for the development the MAPS pipeline.
Inference. In our main experiments, we use GPT-4V as our MILMM and NgSPICE (Nenzi Vogt, 2011) as physical simulator to execute circuit simulation. Given circuit analysis problem with a diagram and textual description, our framework infers the answer to the problem following the process described in Algorithm 1. For more implementation details, please refer to Appendix B.
Evaluation Dataset. To evaluate the entire MAPS framework on real-world physical prob- lems, collected 79 high-quality circuit analysis problems from related textbooks and name it SuppleCircuit.in
S SimpleCii reeeirrcuitvva is constrcuted based on exercise problems pri- marily collected Chinese circuit analysis text books, but since current MILMMs are primarily multi- lingual and the linguistic type is not an influencing factor in our framework, this should not affect the evaluation of different MLLMS on this dataset. As each question in Simples has an exact golden answer, we can directly compare the answer produced by the candidate model with the golden answer to compute the accuracy. For a fair evaluation of our proposed solution framework, we only retained questions that involve LPRC type questions, which are covered in the first four chapters of the textbook.
EcuiiEvaa

---

Published as a conference paper at ICLR 2025
4.2 EVALUATION OF PPM
We first assess the quality of PPM in translating circuit diagram into SPICE language. We adopt metrics to measure its quality:
Component Quantity Accuracy (ACCco). This metric measures the accuracy of PMM's predic- tion in terms of the number of eircuit components. The prediction is marked correct only when the number of different types of components are all correct. This measures the object recognition quality of PPM and is necessary condition for correct conversion from circuit diagram to its simulation language description.
Component Value Accuracy (ACCey). Based ACCco. ACCcv requires the model to predict the correct value of each component. This is also a necessary condition and is only applicable for Numerical-Typp Type Circuits. ACCcv reflects both the object recognition quality for circuit components and the PPM's ability to recognize numerical values in the diagram.
Simulation Accuracy (ACCim). This metric measures correctness of PPM's conversion results by comparing the consistency of simulation results between the generated SPICE code and the label code. Although ACCco is necessary condition for PPM to useful MAPS, in practice, achiev- ing the same simulation results indicates the same physical circuit with high probability. For the specific examples of these metrics, please refer to Appendix C.2.
We first evaluate PPM the test split ppm-syn syn syn-lprc. Then, we integrate PPM into the inference framework for fur- ther evaluation on world diagrams. The evaluation result of PPM is shown Table 1. Through training, our PPM can successfully convert most of the synthetic diagrams. For the conversion of real-world schematics, our PPM only has around 50% simulation accuracy, which leaves big room for further improvement. We pro- vide more in-depth discussions about PPM in Section 5.2.
Table 1: Conversion efficacy of PPM on synthetic dataset ppm- peon-syn-lprettese and 20 diagrams on real-world dataset SimpleCircuitEvaa indicates Numerical-Type diagram while "Lab." indi- cates Label-Type.
of ppm-
"Num."
ppm-syn 1pre-test
ppa-eyn-lpre-test
Simplecircuittval
Metrics
Num.
Lab.
Num.
Lab.
ACCcof
99.2
98.5
87.0
80.0
ACCevt
95.5
87.0
ACCai
85.4
53.3
4.3 EVALUATION OF MAPS FRAMEWORK
To verify the effectiveness of the MAPS framework. we implemented it using existing advanced MLLMs, including GPT-4V (Achiam et al., 2023), Claude-3.5 (Anthropic, 2024) and GLM-4V (GLM al., 2024). We compared our method with directly prompting these MILLMS to generate the results. Additionally, we implemented the Multimodal-Cof (Zhang et al., 2023b), which prompts the model to generate detailed descriptions and analyses of the given circuit diagram and then infer the result based on the generated multi-modal thought for comparison.
Our main results are reported in Table 2, which demonstrates that MAPS significantly improves the MLLM's multi-modal reasoning capability circuit-analysis problems and help it outperform existing models and methods. For example, the state-of-the-arr GPT-4V only achieved less than 7.6% accuracy on the real-world circuit analysis problems, while solution raised bar more than 3 times to 32.9%. Through our case studies, we found MAPS effectively alleviates the issues on physical diagrams understanding and complex mathematical reasoning of current MLLM mentioned in Section 2.
We found that our framework and baseline methods all fail solving problems collected from the Chapter of the textbook, which mainly focuses on the Equivalent Transformation of Resistance and mostly cover the circuits that could not be directly executed in simulator. When the problem is not simulatable, the MLLM can leverage the additional information from simulation language to reason the final answer. Please refer to our Appendix D.2 for more specific case studies. However, how to improve MLLM's general scientific reasoning ability through interaction with a physical simulator still a challenging problem and remains for our future work.
7

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results of MAPS and other baselines Si leci reuiitvall MAPS significantly surpass existing models and method on complex circuit analysis problems.
t Acc.(%)
Chapter
Chap1 25
Chap2 24
Chap3
Chap4
All
#Problem
19
11
79
GPT-4V
16.0
4.17
5.26
9.09
7.6
GPT-4V MMCoT
8.0
4.17
5.26
0.0
5.1
GPT-VV + MAPS (Ours)
52.0
7.14
36.8
45.5
32.9(x4.3)
Claude-3.5
16.0
4.17
0.0
0.0
6,33
Claude-3.5 MAPS (Ours)
44.0
12.5
21.1
36,4
25.3(x4.0)
GL.M-VV
8.0
4.17
5.26
0.0
6.33
GLM-4V MAPS (Ours)
32.0
0.0
15.8
36.4
x9.0x
Gemini-1.5
8.0
12.5
0.0
0.0
6.33
GPT-4o
20.0
4.17
5.26
9.09
10.1
ANALYSIS
5.1 ANALYSIS ON INFERENCE PHASE DESIGN OF MAPS
We perform in-depth analysis of our framework and investigate the contribution of its different com- ponents. Our ablation study was performed using a sample of 20 randomly selected problems from EimpleCicciitEve
We analyze our MAPS framework by answering a series of questions.
Q: Can we directly prompt MLLM generate simulation language descriptions of given circuit diagrams, instead of training the expert model PPM?
A: Despite being pre-trained on large-scale corpora, we found that even the most advanced MILMs, such GPT-4V, often struggle to generate accurate simulation language descriptions for relatively complex circuit diagrams. Specifically, GPT-4V refused to generate SPICE code in 8 out of 10 instances in our evaluation set.
Q: Is the simulator necessary for MAPS framework?
Table 3: Ablation study of MAPS framework on Problems sampled from The results show high reliance of the MAPS framework on the physical simulator.
A: We ablation analysis to answer this question and report the results in Table 3. We found that MAPS does work without the assistance of the simulator when we remove the simulation results from the final query (i.e., MAPS w.o. Simulator). We also ver- ified the necessity of simulator by prompting MILMM to write Python programs to infer the answer (Chen al., 2022) given the simula- tion language and problem description (i.e.. MAPS w.o. Simulator + PoT). Notably, MAPS w.o. Simulator and MAPS w.o. Simulator + PoT both achieved only 15% accuracy the evaluation set. This underscores the importance of incor- porating a professional simulator when addressing problems with complex physical backgrounds.
Method
t Acc.
MAPS (Ours)
55.0
MAPS w.o. SL
45.0
MAPS w.o. Simulator
15.0
MAPS w.o. Simulator + PoT
15.0
Q: Is the simulation language description helpful for the final reasoning?
A: We found that when the problem is not simulatable, the simulation language can still be helpful the final reasoning of framework. The structural information provided by the simulation language significant reduces hallucination of MLLM when understanding the diagram, akin to the role of scene graph in general multi-modal reasoning (Mitra et al., 2024). Appendix D.2 presents a detailed example showing how simulation description in MAPS alleviate the MLLM's hallucination problem when the physical scene not simulatable.
We also investigate whether the simulation language description is necessary simulation-aides reasoning step when simulation results are given, denoted as MAPS w.o. SL in Table 3. The result shows that the SL plays vital role in final reasoning when the simulation results given, bridging the gap between the diagram information and numerical simulation results.
8

---

Published as a conference paper at ICLR 2025
5.2 ANALYSIS OF PPM CONSTRUCTION
Philosophy of PPM Construction. Although we focus solely on circuit disciplines in our evalua- tion, the philosophy of constructing a PPM is universal across all physical disciplines. The target of PPM is to convert physical diagram to its formal and simulatable language description, which quires paired training data in the form of physical diagrams and corresponding simulation language descriptions.
Since there is no available open-source data in such a format and human annotations on a large corpus is quite costly, we devised an automated data synthesis solution enhance the VLM's per- ception ability on real-world diagrams. The assumption behind our data synthesis pipeline is that the potential space of physical diagrams can be effectively covered by a human-desige Physical diagrams are often composed of dots, lines, and symbols with specific physical meanings, and are primarily designed to abstract real-world scenarios. By distilling the core patterns of these diagrams, we can establish a distribution to generate representative training data for PPM. For ex- ample, in circuit diagrams, we have observed that most inputs are formed with planar grids, with components placed the edges of these grids. In mechanical diagrams, the pattern could be com- position and positional relationship of mechanical objects (pole, ball, box etc.). Since our work exploratory, designing a universal generator for physical diagrams obtaining comprehensive physical perception model remains an open problem.
distribution.
Using VLM to implement PPM. Converting a physical diagram into its simulation language de- scription can be viewed as comprehensive vision task which involves the recognition of physical objects and the OCR of its detached labels, along with the complex topology about the components' connection. In terms of circuit schematics. some previous works (Bayer et al., 2023; Bailey et al., 1995: Tao et al., 2024) investigate multi-step process to convert a pixel-level eircuit into digital structure, but their methods are expensive to implement and not scalable to diagrams of new styles.
By using VLM as our perception model, we obtaín an end-to-end physical diagram recognition so- lution whose capability can be extended through expanding the data distribution during training. Besides, we also observe that pre-trained VLMs exhibit promising generalization ability after train- ing on our synthetic data, e.g., its OCR ability on float number although our synthetic data only contains integer values.
Scaling the conversion task. Through development set based on synthetic Numerical-Tyy Circuits, we also found that the conversion accuracy (ACCsim) decreases as circuit's complexity increases. Figure 3 illustrates that the number of nodes and components increases in our synthetic data, the simulation accuracy of PPM's predictions shows a downward trend. This result is intuitive since the smaller circuits with simpler physical structures show higher accuracy during test.
Simulation Accuraty vs &Cumponent
Simulation Accuraty vs. &Node
1.0
100
170
1.5
90
0.R
0.8
-63
0.5
603
-0
12.4
0.2
20
0.24
0.0
0.0
#Comcorer
4Roode
Figure 3: With the increase circuit scale seale-speciffcallyy the number of components and electrical nodes the accuracy of PPM decreases. The colorbar display the sample amount in each scale.
DISCUSSION & CONCLUSION
In this work, introduce the MAPS framework to address the inability of existing MLLMs in understanding complex physical diagrams and to solve such problems analytically. Our framework, which trains Physics Perception Model (PPM) interpret physical diagrams and applies Chain-

---

Published as a conference paper at ICLR 2025
of of-Simulation and Simulation-Aider Reasoning during inference, successfully solves the circuits analysis problem, a typical and important type of real-world physical problems.
MAPS excels in deriving final answers when the physical scenario is directly simulatable. However, a key limitation is its static workflow, which lacks feedback interaction with the physical simula- tor. To address this, dynamic workflow where the simulator acts feedback is necessary. In this setting, PPM still serves an important role in connecting multi-modal information with the physical simulator. This improvement would significantly enhance the versa- tility of our physical agent and is an important focus for future work.
external environment for
As the first attempt of this kind, this work only tested MAPS LPRC circuit analysis problems. Extending MAPS to other scientific disciplines with complex illustrative schematics is an important next step. requires developing a universal and accurate PPM for the Chain-of-Simulatiinn This is challenging task in computer vision that remains for future work. Additionally, simulators are currently domain-specifer making effective organization across simulators of different domains or the development of universal simulator crucial for MAPS's broader application.
As shown our experiment results, our work presents a solid path towards building multi-modal agents capable of solving expert-level scientific problems, contributing to the progress towards achieving AGI in scientific domains.
7
LEDNONTEDGEMENT
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and Tsinghua University Initiative Scientific Research Program.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0872. 2023.
Anthropic AL Introducing the next generation of Claude anthropic.com https://www:
anthropic. wwt.co/////ooa/ wtt.c//////////////e/eoooooasaaa
family. [Accessed 04-06-2024).
Anthropic. Introducing Claude 3.5 Sonnet, 2024. URL news/claude 3 5-5--onneed sonnet. [Accessed 01-10-2024).
Donald Bailey, Andrew Norman, Giovanni Moretti, and P North. Electronic schematic recognition. Massey University, Wellington, New Zealand, 1995.
Johannes Bayer, Shabi Haider Turabi, and Andreas Dengel. Text extraction for handwritten circuit diagram images. In Mickael Coustaty and Alicia Fornés (eds.), Document Analysis and Recogni- fion ICDAR 2023 Workshops, pp. 192-198, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-44198..
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint drXiv:2211.12588 2022.
Nathan C Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Con- nor W Coley, and Vijay Gadepally. Neural scaling of deep chemical models. Nature Machine Intelligence, 5(11):1297-1335 2023.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint 2336v:2306. 20..0400000 2023.
10

---

Published as a conference paper at ICLR 2025
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family large language models from glm- glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12799 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujuu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint arXiv:2309.17433 2023.
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. In The Twelfih International Conference on Learning Representations, 2024.
URL /ttp::s///ppereveeww net/ htr///orummm// iidiidd--o 9d=
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. Structgpt: A general framework for large language model reason over structured data. arXiv preprint arXiv:2305.0964. 2023.
PC Kohnke. Ansys. In Finite Element Systems: Handbook, pp. 19-25. Springer, 1982.
Milton Laikin. Lens design. Cre Press, 2018.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Yuesen Li, Chengyi Gao, Xin Song, Xiangyu Wang, Yungang Xu, and Suxia Han. Druggpt: A gpt- based strategy for designing potential ligands targeting specific proteins. bioRxiv, pp. 2023-06, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 233.202305.20050. 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuiigg Advances in neural information processing systems, 36, 2024.
Shilong Liu, Hao Cheng. Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint ar3 1.05437, 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioin- formaties, bbbbabbb, 2022.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models, 2024.
Laurence W Nagel. Spice2: A computer program to simulate semiconductor circuits. College of Engineering, University of California, Berkeley, 1975.
Paolo Nenzi and Holger Vogt. Ngspice users manual version 23. Experiimentsnnspice23-mmm pdf, 2011.
11

---

Published as a conference paper at ICLR 2025
Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. Mp5: A multi-modal open-ended embodied system in minecraft via active perception, 2024. URL https::///..m. wtw.i.iiivv.o.co org/ abs aba//b//333/ /2312 07472.
Singiresu S Rao. The finite element method in engineering. Elsevier, 2010.
James A Svoboda and Richard C Dorf. Introduction electric circuits. John Wiley & Sons, 2013.
Zhuofu Tao, Yichen Shi, Yiru Huo, Rui Ye, Zonghang Li, Li Huang, Chen Wu, Na Bai, Zhiping Yu, Ting-Jung Lin, and Lei He. Amsnet: Netlist dataset for ams circuits, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1105,, 2023.
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. arXiv preprint arXiv:2407. 11777.1168.. 2024.
Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Zhengxin Li, Lin Ma, Shenghua Gao, et al. Tool-Imm: A large multi-modal model for tool agent learning. arXiv preprint arXiv:2401.1077 2024a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Ji- tao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception, 2024b. URL https:://///. /tttpp:://axiv..oo wwwaaaaivvv//// org/ /2011 16158.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhiu Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogvim: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 2023a.
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mlims): A comprehensive survey on emerging trends in multimodal rea- soning, 2024c. URL https:://// //axxvivvvv../////// org/ abs / 2401 06805.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He. Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. Jarvis-l: Open- world multi-task agents with memory-augmented multimodal language models, 2023b. URL https: / org/ abss 2ttp///11...9......
Irene Weber. Large language models as software components: taxonomy for llm-integrated ap- plications. arXiv preprint (00i::2400.10000) 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 0224824-248877 2022.
Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. Droidbot-gpt: Got-powered ui automa- tion for android, 2024. URL https: /tttps::/azzee /tt////axvvvv//e////e/eeeeeaaavr www.iiii....aaao...oo .rg. absp//bss///2// 07066.
Douglas Brent West et al. Introduction to graph theory, volume Prentice hall Upper Saddle River, 2001.
Chenfei Wu. Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models, 2023, URL https:// arxiv. www.vorg/aae abs 2.303.04772
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: org/
survey, 2024. URL
Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1381, 2023.
12

---

Published as a conference paper at ICLR 2025
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive musti-disciplime multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:231 2023.
Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu. Geogpt: Understanding and processing geospatial tasks through an autonomous gpt, 2023a. URL https://arxiv. org/abs/ / 2307. 07930.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00092, 2023b.
Ge Zheng, Bin Yang. Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5166 18-5119,, 2023.
Yiwu Zhong, Zi- Yuan Hu, Michael R Lyu, and Liwei Wang. Beyond embeddings: The promise of visual table in multi-modal models. arXiv preprint arXiv:2403. 18252, 2024.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using gpt-4 code interpreter with code-based velffverriication, 2023. URL https: /ttp//.rxx.//////////..//.../..t abs/ abs/2308. 07921.
Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of-thought prompting for visual reasoning refinement in multimodal large language models, 2024.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Improving Multi-modal Reasoning Ability of MLLM. Reasoning ability is foundational for build- ing agent that assist human to solve complex real-world tasks. There are many studies focusing on improving the reasoning ablility of MLLMs. There have been three main directions to boost the rea- soning capability of MIL.Ms, including instruction-tuning prompt engineering and tool use (Wang et al., 2024c). As instruction tuning requires high-quality multi-modal training corpus which is seare in scientific domains, most studies focus on how to improve the scientific reasoning ability of MLLMs via prompting methods and tool utilization.
The multi-modal prompting methods involve designing effective prompts to fully activate the model's image understanding and language reasoning abilities based on the carefully crafted text instructions. Specifically, existing methods (Zhang et al., 2023b; Mitra et al., 2024; Zheng et al., 2023: Zhou et al., 2024; Zhong et al., 2024: Yang et al., 2023) focus how to enable the model to generate intermediate rationales, or chain-of-thought (CoT) (Wei et al., 2022), for parsing image information and then further reason based these intermediate results (Zhang al., 2023b). Some variants of Multimodal-CoT focus on the form of CoT, for example, Zheng et al. (2023) format the CoT decomposition of original problems and use the answers of sub-problems generate the result, while Mitra et al. (2024) adopt Scene Graph (SG) as the rationale to assist the inference of final answer.
On the other hand, integrating LLMs with external tools exhibits significant improvement in sci- entific reasoning (Chen et al., 2022; Gou et al., 2023). For the reasoning problems containing data from different modalities, the category of available tools extends beyond traditional external software (e.g., calculators, calendars, search engines, code execution environments) to specialized vision models (e.g., object detection models, OCR models, image semantic segmentation models, etc.). Research this direction seeks to build useful tool sets and design mechanisms for MLLMs to interact with these tools, thereby completing designated reasoning tasks more effectively (Liu al., 2023; Wang et al., 2024a; Gao et al., 2023).
Multi-Modal Agent in Scientific Scenarios. A multi- modal AI agent is system designed to real- ize users' general general-purpose purpose requirements by perceiving its environment with molli- information and making decisions based its observations (Xie et al., 2024). Multi-modal agents have been de- veloped for many important domains, including GUI automation (Gur et al., 2024; Wen et al., 2024; Wang et al., 2024b), embodied AI (Qin et al., 2024; Wang et al., 2023b) and general understanding, generation and editing of images, videos and audios (Wu et al., 2023; Gao et al., 2023; Liu et al., 2023: Yang et al.. 2023). For example, Gur et al. (2024) devise an LLM-based agent that learns from interactive experiences to follow human instructions and complete tasks on real websites, such as click, type or making selection.
To construct multi-modal agents in scientific domains, an important research direction involves how to perceive multi- modal information encompassing diverse scientific concepts. With the develop- ment of LLMs, lot of efforts have been made to build multi-modal foundation models tailored for specific scientific scenarios. These models are capable of perceiving or generating chemical for- mulas, protein sequences, geographical information, graphs, and more (Luo et al., 2022; Li et al., 2023; Frey et al., 2023; Jiang et al., 2023; Zhang et al., 2023a). However, most previous works focus solely multi-modal information in text format, neglecting the pixel-fommat information of physical diagrams that are prevalent in human knowledge bases.
B ADDITIONAL IMPLEMENTATION DETAILS OF MAPS INFERENCE PHASE
B.1
SIMULATION
For the simulation of circuit problems, we use NgSPICE (Nenzi & Vogt, 2011) developed by the UC Berkeley CAD Group as simulator. The core arguments we for the simulation are listing on Table
We store the simulation results in dicticnary format, which is commonly used data structure in programming as well as the conversation with MLLM (Mitra et al., 2024; Weber, 2024). Figure 4 shows an example of the post-processing of simulation result.
14

---

Published as a conference paper at ICLR 2025
Table 4: Parameters of NgSPICR simulator
Param.
Setting
Temperature
27
Nominal Temperature
27
.itle Active DC Circuit Vs
SPICE
2 VI 20
Raw Simulation Result
No compatibility mode selected!
op print ((V)) measurement of print v(2. .endc .end
Circuit: Active Circuit
Doing analysis TEMP 27,000000 TNOM 27.000000
Data
v(1) 5.00000e+00 v(2) 05000000++00 I(v)) 0.000000e+00
Simulation result: "1": *-5V", "2": "-5V*, "!":
v(2,1) 0.000000e+00
-.plot", "pprint', or
lines) no simulations
"u": "DV"
Dictionary Format Result
Figure 4: Post-processing of simulation results.
B.2 PROMPT TEMPLATES
In this section, we will showcase the prompt templates that we used Inference Stage.
At the Chain-of-Simulation step of MAPS inference, since our training PPM is merely an image- to-text model, the component values of circuit in textual description is merged the simulation language by MLLM in Refine process.
The prompt used for this process shown at Figure This prompt will only be applied when we detect <Empty y> token in generated simulation language of PPM, which is special token design for the component with missing value in the diagram.
Prompt Template MAPS Inference CoS Refine
You are an expert in circuit principles and need to solve dircuit problem, consisting of circuit diagram and question. We have converted the circuit diagram into SPICE netlist language the
values might missing (marked <Empty>> complete SPICE information provided in the problem, such filling resistance values from question into the netlist, Your output should be the completed SPICE code, and please place your answer between SPICE and
circuit recognition madel. However, circuit diagram.
Question (question) (subq)
# SPICE Netlist Converted from Circuit Diagram (spice)
Figure 5: Prompt template of refine process at Siain-of-Simmlation step
In the Simulation-Aider Reasoning(SAR) step, the MLLM infers the answer based on the informa- tion provided by Chain-of- Sfiiooaaii.imoinn...mmmmm Figure 6 shows the prompt template used for SAR in circuit disciplines.
15

---

Published as a conference paper at ICLR 2025
Prompt Template MAPS Inference SAR
You are an expert arcut péiciples and need to solve dioult problem, consisting of diruut deagram and question. We have convorted the diruut diagram into SPICE retlat language using a dirout recogrilion model. Mearwhille, we have imported the SPICE oode into cicuut simulstion software and obtsined si simulatio reeults, Besed the circuit diagram, the question, the SPICE netist, and the simulation results, please intor the answer to the probiem.
Piease joon
final numerical value
JSON format follow::
"Values": "Value?": "Resuli22
Y.
If the simulation results directly contain the answer, you should oulput the answer obtained from the simulation neults. the simuration recults show anomalies or do nor contain the anower the question, need further infer the final result based netlist and simulation results.
(questoo](subo)
SPICE Nedist Converted from Cirouit Diagram (spioe)
Simuration Results (sim ref)
Figure 6: Prompt template of Simulation-Aided Reasoning step
If the simulation results are not obtained (due to incorrect simulation language) in SAR step. we use special prompt that allows the MILM infer the final result based on the information provided in the problem and the simulation language. Figure 7 shows the special prompt.
Prompt Template MAPS Inference SAR w.o. simulation result
You are an expert circut principles and need to salve a circuit problem, consisting of dircuit diagram and a question. We have converted the cirouit diagram into SPICE netlist language using arcult recognition model. Based on the question, the ctrcuit diagram, and the SPICE netlist, please infer the answer the problem.
represent final Json
"Value1": "Result1",
"Value2": "Result2"
(question) (subq)
SPICE Netlist Converted from Cirouit Diagram (spice)
Figure 7: Prompt template of Simulation-Aidee Reasoning step (No Simulation Result)
Figure 8 shows the prompt template that we prompt MLLM to directly infer the answer. Since current MILMs have been trained on CoT data, they will apply automated CoT to infer the answer.
MMCoT (Zhang al., 2023b) decomposes the multi-modal reasoning process into paradigm: Rationale Genetraion and Answer Inference. Following the similar idea, we define the rationale in our setting as the language description of the physical diagrams. Figure 9 and 10 display our prompt templates for the two step generation.
step
16

---

Published as a conference paper at ICLR 2025
Prompt Template MLLM Direct
Please solve the following dircuit probiem, and represent the final numerical values expressions JSON format, formatted
Json
"Valuel": "Result"",
"Value2:: "Result2"
# Question
(question\(subq)
Accompanying Diagram
(diagram)
Figure
Prompt template MLLM directly inference
Prompt Template MLLM MMCoT Step
You will receive circuit problem with circuit diegram. You need generate a descriptive language for the dircuit diagram to facilitate furthar inference. pestra_prompt)
Question {question><sutel
Accompanying Diagram
(diagram)
Figure 9: Prompt Template for Step of MMCoT
Prompt Template MLLM MMCoT Step
You need solve cirouit problem cirouit diagram, you can to the descriptive language circuit diagrem. Pleese represent numerical values or expressions JSON format, formatted as:
json
"Value1". "Result1", "Value2": "Result2"
Question (nuestion((subq)
# Circuit Diagram Descriptive Language (desc)
Accompanying Diagrem (diagram)
Figure 10: Prompt Template for Step 2 of MMCoT
17

---

Published as a conference paper at ICLR 2025
C ADDITIONAL IMPLEMENTATION DETAILS OF PPM CONSTRUCTION PHASE
In this section, will delve into the implementation details involved in the construction phase of the Physics Perception Model (PPM).
C.I. DATA SYNTHESIS
The data synthetic pipeline has been shown in the left side of Figure 2(a). We have introduced the general process of data generation in Section 3.2.1. In this section, we will introduce our synthesis pipeline in circuit discipline with specific example.
In the first step, we sample an diagram layout from the manual distribution. As discussed in Section 5.2, the key property of our designed generation distribution is cover the distribution of real-world diagrams as comprehensively
possible.
Our implementation diagram layout sampling in synthesizing PPM's training data for Linear Pure Resistive Circuit (LPRC) (Svoboda & Dorf, 2013) diagrams is shown in Algorithm 2. where D. l in the pseudo code represent Discrete Probability Distribution and Uniform Distribution respectively. We only show our main idea in the pseudo code due to its tedium. Since we use hierarchical sampling process, we can sample diverse circuits with different shapes, components and annotations. The hyperparameters of the sampling process are set by human experiences.
Algorithm 2 Diagram Layout Sampling for LPRC
1: Input: dmax. dmin; n. Pa. F. Pt,
2: Output: Diagram Layout I
& Determine the scale of the grid
3: number of grid: (m x n): m,n m
4: horizontal scale: d = (d) ,.,")) where dt ~ Uld (dminn dmax)
5: vertical scale: d (di,... d where di U(diins dmax)
& Determine the component's type direction in each edge
6: horizontal component: T = [T mx((n1)) where Th ~ D(t,pt)
7: vertical component: T (m.)lmm-ijiit where TH, p()
8: direction of horizontal component: D* [D [D where D D(0, 1), (0.5, (0.)).
9: direction of vertical component: D = [DY, (Pi](mm-)xx where D D((0, 1), (0.5,
Determine the component's s value & unit in each edge
10:
Determine the component's label in each edge
11:
Determine the observation's label direction in each edge
12:
Assign the observation label to controlled source
13:
14: I -CrreuiDDiagammmm n, d' d. ,.....
15: return I
In our illustrative case, we sampled a 4 x grid and assign each edge with specific components, as shown in Figure 11.
After the sampling of diagram layout, there are two synthesis paths that respectively generate pixel- format diagram and simulation language(SI)) description.
The diagram synthesis path involves converting the grid into LaTeX language that can describe a circuit diagram. We use the LaTeX package circuitiks to plot the circuit. After compiling the LaTeX code, pixel-level circuit diagram can be generated. Each edge in the grid is converted into line of LaTeX drawing language. The drawing statement includes the start and end positions of
18

---

Published as a conference paper at ICLR 2025
sa
Figure 11: Data Synthesis: Diagram Layout Sampling
the element/wire, the shape of the element, the label of the element (number or string), the type of measurement, and its label.
The SL synthesis path primarily focuses distilling the physical structure from the diagram layout using human prior knowledge. The physical structure of a circuit can be represented by a netlist (Nagel, 1975; Tao al., 2024) model, which is directed graph (West et al., 2001) where each node represents an equipotential point and each edge represents a component. The netlist model includes all components along with their types, parameters, and topological connections of circuit, while filtering out position and scale noise when plotting the diagram. We write rules to automatically identify equivalent electrical nodes using basic physical properties and convert grid information into a netlist. Figure 12 illustrates the physical structure extraction process of our example case.
110
Remove Components
JEV
DFS
No
No
N1
No
3uo
NG
Figure 12: Data Synthesis: Extracting physical structure from diagram layout using physical rules
Finally, the netlisti is translated into SPICE code, ie. our simulation language as the training label of PPM. The SPICE language can be mainly divided into parts: the first part is the description of circuit elements (Element Card), and the second part consists control commands that determine the simulation type and output results (Control Card). Since each edge in the netlist corresponds
19

---

Published as a conference paper at ICLR 2025
directly circuit element description line in SPICE, the conversion of the first part is merely formatting process. For the second part, which involves control commands, we set up the simulation for steady-state analysis of Linear Purely Resistive Circuits(LRPC), This is done by using the .OP (Operating Point) command to define the simulation type as a DC operating point analysis and the .PRINT command to specify the circuit state quantities to be measured.
We also counted the amount of electrical nodes and components our synthetic dataset ppm ppm-syn- weyn-lpre-tere sestt. The statistic results are shown on Table 5, where "#X" represents the number of objects of type X.
syn-
Table 5: Statistics of ppm-syn- -ertt-esrc-ttttttt
Parameter
Mean
Std
Max
Min
#Nodes
7.876
3.137
26.0
1.0
#Branches
11.088
5,364
45.0
0.0
#Resistors
6,566
3,452
25.0
0.0
#Voltage Sources
1.340
1.268
9.0
0.0
#Current Sources
1.517
1.340
12.0
0.0
#Controlled Sources
1,411
1,457
11.0
0.0
#Shorts
0.255
0.508
0.0
#Voltage Measurements
1.192
0.834
7.0
0.0
#Current Measurements
0.503
0.713
6.0
0.0
Figure 13 shows some cases of synthetic circuit diagrams.
12V
71f?
Rr
Nu Ro
69mV
69k1
R
99kΩ
49H
97kV
Rio
Sk?
641
580
9V
37kF2
72mf
lo
24Ω
6012
720
U
44kA
14kA
5mf2
Ug
94mff
980
51Ω
95kf
11A
52k?
68kΩ
Figure 13: Examples of synthetic diagrams
The synthetic paired data used for the training of physics perception model.
20

---

Published as a conference paper at ICLR 2025
C.2 PPM TRAINING
We will introduce our PPM training process for our main experiments in detail in this section.
The training objective of PPM is to predict the simulation language given the visual diagram input. Let the diagram input be Xv, and the output text sequence be Yz = (yI.1,,... VI,r) ) with a length of T. The model parameters are denoted as 0. The probability distribution for predicting the next token under the model can be represented by (v YL,1a). . The MLE fine-tuning loss can therefore be written in the form L (Xv; YL.
E
log pogpoly-o y1.c+1|xy, Yr,1:), where
YL,1= (VL.1.-., )L,,)"
Let the training dataset be = (x y he=:,, containing N samples. The training process involves minimizing the negative MLE loss for all training samples:
0- = min
MLL (x
(XLV v
vv())
X
DD
In our experiments, we adopt CogVLM-177 our base model to train the PPM. The model version for main experiment is cogagent -aa-tt---aa1110
We primarily train the visual modules and the image-text cross-attention part, while the parameters of the text generation part remain mostly unchanged. This is because the main challenge of this task lies in image understanding, and the text generation aspect has already been adequately learned through pre-training in the language model part of CogVLM. We control the trainable parameters of the model as follows: the visual encoder, the VIT, the visual multi-layer perceptron and rotary encoding module, the BOI token and the EOI token, resulting in total of 11.6B parameters that need updating. The remaining parameters are kept freezed.
We employ Low-Rank Adaptation (LoRA) (Hu et al., 2021) as the fine-tuning strategy to train the VLM. Using the LoRA algorithm to train the VLM significantly reduces the number of parame- ters for which gradients need to be computed, thereby greatly decreasing the memory overhead of training.
We list main hyperparameters used for PPM training Table 6.
Table 6: Main Hyper-parameters of PPM Training
Param.
Setting
lora-rank
50
max-length
2000
batch-size
32
train-iters
2000
optimizer
Adam
learning-rate le-5
Ir-decay-style
cosine
warmup 0.2
After the training process, evaluated the PPM using the metrics introduced in Section 4.2. To illustrate how these metrics work, we present three cases Figure 14.
21

---

Published as a conference paper at ICLR 2025
SL descriptlon(Labee)) .tilee Active DC Circuit
Image:
USZ
G 0.02
20
O 1 20
R 30
Simulator
ZE.00
I
eQuivalance
.ontroll
print -v(3) measurement
.endc
.end
ACCcg=0
ACCoy
Simulator
PPM Prediction 1:
PPM Prediction 2:
PPM Prediction 3:
1011
1011
E 1 0.4 G 1 0.02 2
02014004 GO3 1 0 0.02 2
E20100 004
GO3 1.0 0.02
20
R.. 20
R 1
50
.control
.control
.control
v(1) measurement
print v(1) meesurement .endc .and
v(1) measurement .endc .end
.endc .and
Figure 14: Cases of evaluating PPM.
D ADDITIONAL DETAILS OF EVALUATION ON SIMPLECIRCUITEAA
D.I DETAILS SIMPLECIRCUITEVAL
To evaluate the performance of MAPS on Linear Pure Resistive Circuits (LPRC), we selected ques- tions involving LPRC from the first four chapters of circuit course texthook. To ensure question diversity and coverage, we consulted domain experts to remove redundant questions, resulting in final of 79 questions. The characteristics of the problems from these four chapters and their details are summarized in the table below:
Table 7: Summarization of S SimpleCircittEva
Chapter
Content
#Components(Avgg)
Characteristics
Circuit elements and circuit laws
4.7
Basics circuits. No controlled sources.
2 Analysis method of simple
6.8
Resistance circuits. Not di- rectly simulable, require cal- culation of equivalent resis- tance.
resistance circuit
3 General method of analysis of linear resistance circuits
7.6
LPRC circuits. topologies.
Normal
Some theorems electric
6.2
LPRC circuits. topologies.
Complex
circuits
In Figure 15 offer example question for each chapter in Simpl e rcui tEval.
22

---

Published as a conference paper at ICLR 2025
Example questions 4 chapters SimpleCircwidEEval
Chapter
Chapter 2
@: voltage and current marked in each dircutt as shown Figure 1-14.
qe Find input resistnnce (1mathrmm(4B)
IV
figure
Chapter 3
Chapter
the drouit shown in Foure 3-15,
are stwan U_S 2- 5--50egeo.
1010maga
Orelen O
Do
Thic the curmets in nech branch and the powerr
emitiad branch cirult
oelivered by two controlled sources.
4-18.
Figure 15: Example questions in SimplecircuitEval.
D.2 SUCCESSFUL EXAMPLES
We found simulation language can reduce the hallucination of MILM on understanding the physical diagram, as shown on Figure 16. These results are consistent with those in a concurrent work (Tao al., 2024), which uses SPICE descriptions as auxiliary information to guide MI.LM's decisions in the context of IC design.
Case: The role Simulation Language
MAPS Prompt: You are an expert in diruut principles and need to solv a circltt problem,
each crout shown figure
Find input resistance (L_(aattrm(AA)) 1) for each diruit anown figure Peeae complete part (t).
60
Netlis: Converted from Chrutt Diegram
spice
R 0
R100
2212
In order calcuaat the input redistance (Am)thi((AB) we can examine the droult dagram and analyge
GPTVV:
MAPS Solution:
insut resistance (vnathrm/AB)) U the circat: shown in figure 2-3(b),
Before start, let's label the reinters, starting from point ciocknate d reetior well name the two resistrn on the top
comprises bridge structure.
200mega 10 V. 210mege the ddle resitor 410mega
this oridge circut,, we can see two dividing restetors (20 on the eft and 2n on the right) paralici top, and dividing resisters
on rightt
61Omega
the formula for panallel neastors.
For the ownsies lei resistons on the top (20 and 20), the total resistance V. R. (vmethmm((p)) is calcuaaed as: (ymothi((uo)) Vrac[R_ tooot [R_1 R_2) VracE2 lodot 2) [2
series, overall RL(134)
R. (Urrathmm:A)) AB3) 1) is the rum of VI (133))))) 10 and (235)
for panalaa lel nevators the buttom (4D and 60), tutal resistance "'_''mettmmm (down)) 33 at:
yi VR. ('mathim (A3)) (134) + (233) 310mmgg
1.5/Gmoog 4.510mega
I.R ('mmtthrm down)) VraatR lodot R. 2) + 2) =n Visot4 lodot 61(4 6) 10mega'
(110)
The final 2S0N formet
Finally,
equivalent resistances
overall
json ("R_ (AB)":"0 "4.50" y
R_ ('mathim (AB1) 1)
R. ('yaateem(aa))) RL, (ummathrm(uu))) (demmm - 2.4
Vömega
'jon ("w_(VV//ahtrmmmmmeemmm
Case Study: The vital role of simulation The positions marked in red are the hallucina- tion positions of the MILM.
Figure 16:
Figure 17 illustrates how MAPS effectively overcomes the challenge of MLLM's inability to com- prehend complex physical diagrams by employing formal simulation descriptions and executing precise simulations. In another case, as shown Figure 18, we found that MAPS successfully addresses the issue of MILM's inability perform derivations of complex equations.
23

---

Published as a conference paper at ICLR 2025
Case: MAPS solve the Diagram Understanding Issue in MLLM
the drout shown
MAPS Prompt: You
circuit principees need solve
equur. 5u,5-3,, Ls=1 A. 1_Iiel RL2mm1 US voltage source su St. and the current source $t.56.
drcult problem,...
Question
pictured
mathrm
mathim mathrm (A) Omega su mathim (S))
Omaga
mathim (5)) the
(U)
GPT4V: Frst, we need to desermine the electicc power provided
by V. U. ('mathmmS)) and V. (1mathimm (5))
Cirouit Circult
SPICE
From and voltage source are in perslle,, which meens that the current floiigg thouuhh the vatage source It (yntthrm(S) 53) ( the (Vmathmm(5)) which 1A. Therefore, the U. (mmathmm(5) Qj
3V
3A Sw
(1mathrm()))
0.3
it the voltage across the current source octerminod solely the resistons V. R_2 1 and V2 with the total resista ice ('math mttoaaal))
150mega
Zi0mega 310moga
* Simulation results
Thus,
(11: -3V, 2: 44V, '3': '2V 'no1': "3V,, Vo 'DAA, 'ul: NBV")
Therefore, (((athimm))) ls:
MAPS Solution:
unceratand problem
circut clagram, netal arrubakoon
1. The vellige source BU, (enadom (ED cutsut SP (u, farmula gP sriza
We represent the answers 250N format as follows:
(U) (mnathm lines o4-
iimes
"p_ur:
Thus. [0) iddnes BV lames 14 Bivs.
joon
Figure 17: Case Study: MAPS addresses the diagram understanding issue of MLLM.
Simulation
Derivation Issue MLLM
Q: Use the leap ourrent method to
MAPS Prompt: You are an expert in diroult principles and need solve circult
problom,
Rguure 5-16.
Question
Use the loop current method to find the power produced the 1A current the droukt shown Figure 3-15.
Netlist converted circult diagram
GPT4V: to solve this cinout problem and find the power produced by the curnet source, need determine formula, the voltape, to carculate the power.
C
6
0.02
is
1
3
can see creult
R 3
Assume thas the current through the s0-ohm resistor is V. the current through the loop cansizting the 20-0hm resistor vottage source cordisting 30-ohm resistor source is
.control
through
Simulation
Please use physical calcullttor softwaae to solve this problem ISON formst. connot perform actual dineutt equetion solving to provide numerica resuts, this step done manuelly.
(11: '33.33333V' 22: 113,33333V, '3:: '33.33333V,, A: 333333333')
MAPS Solution:
We can directly compute the power:
"rootintrttPPowe produced by the 1A
current sourcel: 1-3.3311133333991111
Figure 18:
Case Study: MAPS solves the math derivation issue of MLLM.
24

---

Published as a conference paper at ICLR 2025
D.3 ERROR ANALYSIS
Through the analysis of erroneous samples, we identified two primary causes of errors in MAPS:
1. Incorrect simulation description conversion during the Chain-of-SSmulattion step. Due to the relatively precise solutions produced by the physical simulator, the errors in Chain-of-Simulation (CoS) process can only occur during the translation step of the simulation language description (SLD). These errors specifically manifest in the incorrect recognition of components by PPM, errors in the identification of circuit topology by PPM, and mistakes in the MLLM refinement process of the SLD. Upon our observation, these types of errors constitute the majority, accounting for 18 out of 20 errors.
2. Hallucination during the Simulation-Aiin Aided Reasoning step. We also found that even when the PPM generated a correct simulation language description, the final inference result was still incorrect. This is primarily due to the limited mathematical reasoning capability of the MIL.M.
We present two typical cases, shown in Figure 19 (Error in CoS) and Figure 20 (Hallucination in SAR step), where our MAPS framework fails to solve the problem.
Case: Error in Chain-of-Simulatie directly cause wrong answer
MAPS Solution:
Qu In the diruit shown in Figure 3-16, given
Baxed on diout dagram, SPICE neti et and the g ven simuaation neaulls, we
1_2-- syaaaaaaa 1_5-00ymmgaa, hind the currens in each brunch and the power the contro
R. 4ipmmgg...
The almu adin roults provide the to tolwing curmen usse.
112 03.5577144 1540 (close 2010
104 42147477A)
After Chain-ot- Saatt-ot--imulation
Notally simukatio
caloulated
MAPS Prompt: You are expert drout prind and need to zohe tircut
This indiotees that the surrent 19 is astually vary cmall and thue the current in the RS branch can bo considored nog hio Marrowex, because the vattage across is equal email
(V1)) direction
the crcal shown Figure 3-10 oiean that (B1-r (natemmms) cumenta each branch and umillled by contnnl scurcae..
Queerton
through Rl
conclude that
power by
SPICE
For the cotageecnntn let curren ecurce i' (101)) sence controi direction te opponite the panser be catculitedd Lsang the formule (P_(P") Iju darmau 1129y.
V11 4:
3ile Astive DC Circut
Green that the semulatiion result caea chose (P_(P134 shouuld ckow zero
Vi8 0 N20
VM4 K:0
R4. N50
Wrong directin current measure,
should be 'V12 N42 2.0
For
1e12 liies 9j.
vallage
81 1442
ling mulaaioo recutt gives power V1. 5500
2 NJ2
Organtring thwoe reoults into JSCW formon
vn N
"p""-1 -1 142200
pritt (V80) messuroment of
+22) meseunmment US
mesoanemen
14" 142857,
16,
X
par (VIT): messrrmmnt o1
enco
(1t MDV: 15..2217--11 N6.7148VV MI-5 448857V h64: 4380TV
Simutstiin IeeLis
zoro con offostvvly be treated as 2010. This pa risuuiarryy common simulations invoning idcal controlled courcee.
numerioal limits
142857A 1-0.38667A (42000)))))
Figure 19: Case Study: Error in Chain-of-Simullee
25

---

Published as a conference paper at ICLR 2025
Case: Hallucination Simulation-Aider Reasoning
Q:
theorem
Ind the cureet branch Stat:S and the power
Oon
branch, we noed lind equvalen vollage source BBV 2220688 the ve across tarminab 55ubd5 when they are opent and the eculva internal mosatnnce SPL_
enitted by the SeatSS branch in the cirultt shown Figure 4-18.
Equivalont
2008
Since the simulaaionn result not provide By (h18, noad caloulaae it from the
orginal drou diagram, When the $6ands branch open. tage S&V on (8 the toge point "nd 201 (sicc there are no components between 1211 and por So8), Hewnwer do not directty have the vatage point Sos.
After Chain-of- -iihu--f-Smuaaiion
Prompt: expert princprie
Eqpinatent interna repietance SR, itsle
Gunetto on
The equi eegaivdentt interna recirtance found by shorting tage sources and lociing into the ferminat SubS. Obeervin crcuul disgrem resicamnes from paint S5o45
antitied SoandS
cincali shown 4-1...
SINCE droutl
dince the original crout contanaa tac voltigge scurcsss and theme unioreen oument BSVi8s fowing
directly _RhBS 58Rt L0088 circatt
V 20
V 3/0 20
Hovover wo con directly extract the oument size from simukatoo results, the feld is indkorinng
lowiing through the
opposto
B 1 40
4-22 90
The power SSP38 is given by the formula:
2 2-33300 20
P vodel
prsves need recertence between points 58665 $pa0sOmeged 5321Crmeged
contral
(1L2)))) 2irCrrage 22CDragge 2 42C0rmgg
and
PP - 2224 lokol 420mmage 0.24412 lodio 420Omage 10.5/1
Siruuadoon HOGUTSS
The final JEON json
200 2: 40V. St. 2017 SOV. ner: 10: 10.5555
(1 tounone branch ab): 0.5. spower branch ab/.105)
Figure 20: Case Study: Hallucination in Simulation-Aide Reasoning step.
26


Reviews:
[{"review_id": "VhX3fOLhWI", "reviewer": "Reviewer_xCVY", "summary": "This paper proposes an innovative framework named MAPS, which relies on multimodal large language models (MLLMs). During the inference phase, it integrates the simulated language descriptions of input diagrams generated by PPM and data obtained through the Chain-of-Simulation process, aiming to address complex structural and quantitative analysis issues in the field of physics. Through empirical studies on university-level circuit analysis problems, the authors demonstrate the significant effect of MAPS in enhancing the inference accuracy of MLLMs, outperforming all existing models. These results reveal the potential of MAPS in strengthening the multimodal scientific reasoning capabilities of MLLMs, providing a promising direction for the development of this field.", "questions": "Incorrect spelling and expression:\n- Line 243 - diagram diagram -> diagram\n- Appendix A.2, Figure 5, Row2 - Chnese -> Chinese\n\nSome Confusions:\nThe research idea proposed in this paper is highly innovative, but during my reading, I encountered several questions that I hope the authors can provide answers to:\n- Table 1 presents the conversion effectiveness of PPM on the synthetic dataset ppm-syn-lprc-test and the real-world dataset SimpleCircuitEval. It is noted that the recognition accuracy of Label-Type diagrams is lower than that of Numerical-Type diagrams, which seems contrary to common belief since label-type diagrams are generally considered easier to identify than numerical-type diagrams. What is the reason for this difference?\n- The test dataset SimpleCircuitEval comprises only 79 questions. Is this sample size sufficient to support a convincing evaluation result? Does it adequately represent the universality of LPRC question types?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The advantages of this paper are mainly reflected in the following aspects:\n- This paper proposes an innovative process framework that can combine physical perception models (PPMs) with simulator outcomes to infer answers to physical problems. The framework integrates the understanding of physical diagrams with the reasoning of physical knowledge, and its effectiveness has been validated through experiments;\n- The paper designs and introduces a synthetic dataset named ppm-syn-lprc, which is used for fine-tuning the visual-language model PPM. The generation process of the synthetic dataset is elaborated in detail, and the scalability of this method is demonstrated. For instance, this approach can be applied to other fields such as mechanical systems, providing clear directions and ideas for improving the model's reasoning capabilities in other areas of physics;\n- The structure of the paper is well-organized and logical, making it easy for readers to understand and follow.", "weaknesses": "Writing:\n- There are some typos in the article, and it is recommended that the author carefully proofread and corrected them to enhance the professionalism and readability of the paper.\n\nExperimental Design:\n- Generalization Issues: The results of this study have only been validated on the GPT-4V model, which may not be sufficient to demonstrate the applicability of the framework to other model architectures. It is suggested that the authors extend the evaluation of the framework to different model architectures to strengthen the generalizability and credibility of the experimental conclusions.\n- Evaluation of Synthetic Data Effectiveness: The paper mentions using CogVLM-17B as the base model for PPM and fine-tuning it on the synthetic dataset. To clarify whether the improvement in the model’s ability to understand physical diagrams is due to the inherent capabilities of the base model or the introduction of synthetic data, the authors are advised to add comparative experiments before and after fine-tuning. This will help verify the effectiveness of synthetic data and reinforce the persuasiveness of the research findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jDkL5i0lw4", "reviewer": "Reviewer_tWjp", "summary": "This paper presents the framework of Multi-Modal Scientific ReAsoning with Physics Perception and Simulation (MAPS) based on an MLLM. It tries to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator.  The authors show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. Towards that end, the authors craft rules to generate a large dataset of diverse circuit diagrams and their corresponding simulation language descriptions. This synthetic data is then used to fine-tune a pre-trained VLM, ultimately producing the PPM.", "questions": "See the weakness section.\n\nThe prompts in the Appendix mention \"Chinese circuit problem\".  What is meant by \"Chinese circuit problem\"? Is it a specific term? If not, perhaps you can revise its usage to avoid potential confusion.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes several contributions. It creates a large dataset of diverse circuit diagrams. \nIt proposes to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. (This is a good approach.)\nThe paper show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. \nThe authors point out that all the information about a circuit may not be in the diagram and may be in the surrounding text and their approach addresses this issue.", "weaknesses": "It is not clear where errors happen. An error analysis of a sample of the evaluation data would be helpful. In particular,  please provide breakdown of errors by type (e.g., perception errors vs. reasoning errors), and give examples of common failure cases.\n\nWhat are the different kinds of questions addressed in the paper? Do they need simulation or just a solver. (With simulation capability many more complex questions can be answered using multiple simulations.) Please provide a categorization of the question types in your dataset, along with examples of each category. In your current dataset what percentage of questions requires a single simulation, multiple simulations, and ones that can be solved with simpler methods.\n\nWhat kind of training data is used with respect to the questions and Simulation language specification? More information on that would be helpful. In particular, please provide information such as the distribution of question types, and the complexity of circuits represented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxAGvn7not", "reviewer": "Reviewer_1DpV", "summary": "This paper presents MAPS (Multi-Modal Scientific Reasoning with Physics Perception and Simulation), a novel framework designed to improve the performance of Multi-Modal Large Language Models (MLLMs) in expert-level scientific reasoning tasks, specifically in physical sciences. MAPS enhances the comprehension and analytical processes by integrating a Physics Perception Model (PPM) with a simulator for interpreting complex physical diagrams and quantitative reasoning.", "questions": "The paper demonstrates the effectiveness of MAPS using circuit analysis problems. Can the authors provide evidence or insights into how MAPS could be adapted for other physical sciences, such as mechanics or optics, which involve different types of diagrams and simulation languages?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1: The integration of perception and simulation for multi-modal scientific reasoning is well-conceived and leverages MLLM strengths while mitigating their weaknesses in handling complex diagrams.\n\nS2: Results from circuit analysis problems highlight a notable increase in accuracy, showcasing MAPS' ability to outperform current state-of-the-art methods.\n\nS3: The paper provides comprehensive explanations for data synthesis, PPM training, and the inference process, enhancing reproducibility.", "weaknesses": "W1: The framework is tested primarily on circuit analysis, which may not fully capture its adaptability across different physical sciences.\n\nW2: The multi-step process involving diagram conversion, SL generation, and simulation may introduce cumulative errors, which could affect real-world applicability.\n\nW3: The reliance on synthetic data poses a challenge for real-world accuracy, as unseen or complex diagrams might not align with generated examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mr4mbKP8h1", "reviewer": "Reviewer_b9qd", "summary": "The paper proposes to use improve Multi-Modal LLM's reasoning capability by fine-tuning with domain-specific formal languages and using external physics simulator.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. There is some novelty in augmenting MLLM with the formal languages describing the circuits and using an external physics simulator.\n2. The proposed method works well in improving the performance.\n3. The paper is written clearly and easy to follow", "weaknesses": "1. The paper only proposed a solution for a very specific domain of physical reasoning, namely circuit analysis. This implies that for every other sub-domain, we need to use another set of formal language and physics simulator, which is not scalable. This also implies that this philosophy cannot be extended to physical reasoning problems that don't have any formal languages describing it, which reduces its potential in the frontiers of science research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)