MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science

Authors: Erle Zhu, Yadi Liu, Zhe Zhang, Xujun Li, JinZhou, Xinjie Yu, Minlie Huang, Hongning Wang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: multi-modal reasoning, scientific reasoning, physical simulation

Abstract:
Pre-trained on extensive text and image corpora, current Multi-Modal Large Language Models (MLLM) have shown strong capabilities in general visual reasoning tasks. 
However, their performance is still lacking in physical domains that require understanding diagrams with complex physical structures and quantitative analysis based on multi-modal information. 
To address this, we develop a new framework, named **M**ulti-Modal Scientific Re**A**soning with **P**hysics Perception and **S**imulation (**MAPS**) based on an MLLM. 
MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. 
The PPM module is obtained by fine-tuning a visual language model using carefully designed synthetic data with paired physical diagrams and corresponding simulation language descriptions. 
At the inference stage, MAPS integrates the simulation language description of the input diagram provided by PPM and results obtained through a Chain-of-Simulation process with MLLM to derive the underlying rationale and the final answer. 
Validated using our collected college-level circuit analysis problems, MAPS significantly improves reasoning accuracy of MLLM and outperforms all existing models. 
The results confirm MAPS offers a promising direction for enhancing multi-modal scientific reasoning ability of MLLMs. 
We will release our code, model and dataset used for our experiments upon publishing of this paper.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729937364172

thread_type:
official_review

<thread_object>
creator: reviewer_b9qd

<content>
summary:
The paper proposes to use improve Multi-Modal LLM's reasoning capability by fine-tuning with domain-specific formal languages and using external physics simulator.

strengths:
1. There is some novelty in augmenting MLLM with the formal languages describing the circuits and using an external physics simulator.
2. The proposed method works well in improving the performance.
3. The paper is written clearly and easy to follow

weaknesses:
1. The paper only proposed a solution for a very specific domain of physical reasoning, namely circuit analysis. This implies that for every other sub-domain, we need to use another set of formal language and physics simulator, which is not scalable. This also implies that this philosophy cannot be extended to physical reasoning problems that don't have any formal languages describing it, which reduces its potential in the frontiers of science research.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534982124

thread_type:
official_review

<thread_object>
creator: reviewer_1dpv

<content>
summary:
This paper presents MAPS (Multi-Modal Scientific Reasoning with Physics Perception and Simulation), a novel framework designed to improve the performance of Multi-Modal Large Language Models (MLLMs) in expert-level scientific reasoning tasks, specifically in physical sciences. MAPS enhances the comprehension and analytical processes by integrating a Physics Perception Model (PPM) with a simulator for interpreting complex physical diagrams and quantitative reasoning.

strengths:
S1: The integration of perception and simulation for multi-modal scientific reasoning is well-conceived and leverages MLLM strengths while mitigating their weaknesses in handling complex diagrams.

S2: Results from circuit analysis problems highlight a notable increase in accuracy, showcasing MAPS' ability to outperform current state-of-the-art methods.

S3: The paper provides comprehensive explanations for data synthesis, PPM training, and the inference process, enhancing reproducibility.

weaknesses:
W1: The framework is tested primarily on circuit analysis, which may not fully capture its adaptability across different physical sciences.

W2: The multi-step process involving diagram conversion, SL generation, and simulation may introduce cumulative errors, which could affect real-world applicability.

W3: The reliance on synthetic data poses a challenge for real-world accuracy, as unseen or complex diagrams might not align with generated examples.

questions:
The paper demonstrates the effectiveness of MAPS using circuit analysis problems. Can the authors provide evidence or insights into how MAPS could be adapted for other physical sciences, such as mechanics or optics, which involve different types of diagrams and simulation languages?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706243332

thread_type:
official_review

<thread_object>
creator: reviewer_twjp

<content>
summary:
This paper presents the framework of Multi-Modal Scientific ReAsoning with Physics Perception and Simulation (MAPS) based on an MLLM. It tries to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator.  The authors show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. Towards that end, the authors craft rules to generate a large dataset of diverse circuit diagrams and their corresponding simulation language descriptions. This synthetic data is then used to fine-tune a pre-trained VLM, ultimately producing the PPM.

strengths:
The paper makes several contributions. It creates a large dataset of diverse circuit diagrams. 
It proposes to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. (This is a good approach.)
The paper show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. 
The authors point out that all the information about a circuit may not be in the diagram and may be in the surrounding text and their approach addresses this issue.

weaknesses:
It is not clear where errors happen. An error analysis of a sample of the evaluation data would be helpful. In particular,  please provide breakdown of errors by type (e.g., perception errors vs. reasoning errors), and give examples of common failure cases.

What are the different kinds of questions addressed in the paper? Do they need simulation or just a solver. (With simulation capability many more complex questions can be answered using multiple simulations.) Please provide a categorization of the question types in your dataset, along with examples of each category. In your current dataset what percentage of questions requires a single simulation, multiple simulations, and ones that can be solved with simpler methods.

What kind of training data is used with respect to the questions and Simulation language specification? More information on that would be helpful. In particular, please provide information such as the distribution of question types, and the complexity of circuits represented.

questions:
See the weakness section.

The prompts in the Appendix mention "Chinese circuit problem".  What is meant by "Chinese circuit problem"? Is it a specific term? If not, perhaps you can revise its usage to avoid potential confusion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709975145

thread_type:
official_review

<thread_object>
creator: reviewer_xcvy

<content>
summary:
This paper proposes an innovative framework named MAPS, which relies on multimodal large language models (MLLMs). During the inference phase, it integrates the simulated language descriptions of input diagrams generated by PPM and data obtained through the Chain-of-Simulation process, aiming to address complex structural and quantitative analysis issues in the field of physics. Through empirical studies on university-level circuit analysis problems, the authors demonstrate the significant effect of MAPS in enhancing the inference accuracy of MLLMs, outperforming all existing models. These results reveal the potential of MAPS in strengthening the multimodal scientific reasoning capabilities of MLLMs, providing a promising direction for the development of this field.

strengths:
The advantages of this paper are mainly reflected in the following aspects:
- This paper proposes an innovative process framework that can combine physical perception models (PPMs) with simulator outcomes to infer answers to physical problems. The framework integrates the understanding of physical diagrams with the reasoning of physical knowledge, and its effectiveness has been validated through experiments;
- The paper designs and introduces a synthetic dataset named ppm-syn-lprc, which is used for fine-tuning the visual-language model PPM. The generation process of the synthetic dataset is elaborated in detail, and the scalability of this method is demonstrated. For instance, this approach can be applied to other fields such as mechanical systems, providing clear directions and ideas for improving the model's reasoning capabilities in other areas of physics;
- The structure of the paper is well-organized and logical, making it easy for readers to understand and follow.

weaknesses:
Writing:
- There are some typos in the article, and it is recommended that the author carefully proofread and corrected them to enhance the professionalism and readability of the paper.

Experimental Design:
- Generalization Issues: The results of this study have only been validated on the GPT-4V model, which may not be sufficient to demonstrate the applicability of the framework to other model architectures. It is suggested that the authors extend the evaluation of the framework to different model architectures to strengthen the generalizability and credibility of the experimental conclusions.
- Evaluation of Synthetic Data Effectiveness: The paper mentions using CogVLM-17B as the base model for PPM and fine-tuning it on the synthetic dataset. To clarify whether the improvement in the model’s ability to understand physical diagrams is due to the inherent capabilities of the base model or the introduction of synthetic data, the authors are advised to add comparative experiments before and after fine-tuning. This will help verify the effectiveness of synthetic data and reinforce the persuasiveness of the research findings.

questions:
Incorrect spelling and expression:
- Line 243 - diagram diagram -> diagram
- Appendix A.2, Figure 5, Row2 - Chnese -> Chinese

Some Confusions:
The research idea proposed in this paper is highly innovative, but during my reading, I encountered several questions that I hope the authors can provide answers to:
- Table 1 presents the conversion effectiveness of PPM on the synthetic dataset ppm-syn-lprc-test and the real-world dataset SimpleCircuitEval. It is noted that the recognition accuracy of Label-Type diagrams is lower than that of Numerical-Type diagrams, which seems contrary to common belief since label-type diagrams are generally considered easier to identify than numerical-type diagrams. What is the reason for this difference?
- The test dataset SimpleCircuitEval comprises only 79 questions. Is this sample size sufficient to support a convincing evaluation result? Does it adequately represent the universality of LPRC question types?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732183162235

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to All Reviewers

comment:
We sincerely thank all reviewers for their constructive feedback and detailed comments. We are encouraged to find that our reviewers appreciate the novelty of our proposed idea (Reviewers xCVY, b9qd) with remarkable performance in real-world scenarios (Reviewers xCVY, tWjp, 1DpV, b9qd), and clear presentation (Reviewers b9qd, xCVY, 1DpV).

It is worth noting that there are some shared questions regarding the writing of our paper and domain transferability across different physical disciplines. Next, we provide our clarifications to these common questions (CQs) in the general response, followed by individual responses to each reviewer's specific questions.

**[CQ1] Typos & Confusing wordings**

**[CA1]** Despite our careful revisions before the submission, there were still a few typos and confusing wordings remained in our paper. We're grateful for Reviewer xCVY and tWjp to bring these to our attention. We've made careful corrections in our updated manuscript. Specifically, we've fixed the typo "diagram diagram" and removed the confusing term "Chinese Circuit Problem" from our prompt (Appendix A.2, Figures 5, 6, 7, and 8). We sincerely appreciate the reviewers' suggestions, which have significantly enhanced the clarity and quality of our paper.

**[CQ2] Adaptability across different science domains**

**[CA2]** This is a very important question. As mentioned in Section 7, because the MAPS framework requires a robust understanding model of physical schematics, i.e. PPM, which in turn counts on a sophisticated data synthesis process incorporating domain-specific knowledge for model training, our current work has conducted experiments only in the circuit analysis domain. Technically, the MAPS framework can be naturally extended to many other physical disciplines: as long as we can describe the schematic's underlying physical dynamics using formal simulation language and leverage corresponding simulators to carry out the corresponding physical system, we can apply the proposed solutions. 

The key to applying the MAPS framework across different disciplines lies in effective engineering implementations. Based on our circuit analysis experience, we would like to provide the following insights for extending MAPS to other disciplines:

1. We need to design the synthesis of PPM training data tailored to the characteristics of schematics in different disciplines to train the corresponding PPM. In Section 5.2, "Philosophy of PPM Construction," we outlined the fundamental principles. First, we need to  establish the components and layouts specific to each discipline. For example, in a statics system, components might include blocks, rods, ropes, hinges, and spheres, with positional relations such as connections and tangency. To generate a <schematic, simulation language>  pair, we can exhaust the combination of the components and their layouts, apply physical rules and simulation language syntax to generate the corresponding simulation language descriptions and diagram generation instructions (such as latex), and subsequently create the corresponding image.
2. We need to configure different physical simulators tailored to different scenarios. For instance, for a statics system, we might use the ANSYS simulator to model the scenario, while for an optical system, we could use Zemax OpticStudio for simulation. This step primarily involves designing a simulation result format that is easily understandable and inferable by LLMs. In this paper, we represent simulation results within the circuit analysis domain using JSON dictionaries to facilitate model comprehension.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732507745228

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Gentle Reminder to the Reviewers

comment:
Dear reviewers,

We hope this message finds you well. We understand that the author reviewer discussion is a critical component of the ICLR review process, and we would like to remind you that the rebuttal period is scheduled to **conclude on November 26th**. Given there are only 2 days left before the deadline, we would like to call for your attention to our provided responses.

Thank you again for your valuable time and thoughtful comments. We have provided comprehensive responses and additional results in light of your feedback. As we are nearing the end of the discussion stage, we would appreciate it if you could review our responses and update your scores if your concerns have been adequately addressed. We remain open to further discussion should there be any issues that you feel have not been fully resolved. Thank you for your consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**(a) Summarize the scientific claims and findings of the paper.**

The paper presents the Multi-Modal Scientific Reasoning with Physics Perception and Simulation (MAPS) framework, designed to enhance the reasoning capabilities of Multi-Modal Large Language Models (MLLMs) in the physical sciences, specifically for circuit analysis. MAPS integrates two components: a Physics Perception Model (PPM) trained on synthetic data to interpret physical diagrams and a simulator to reason about physical knowledge via a Chain-of-Simulation process. The approach demonstrates significant improvements over existing models in reasoning tasks for circuit analysis, validating its effectiveness in expert-level multi-modal reasoning tasks.

**(b) Strengths**

1. **Novel Framework**: MAPS combines physical perception with simulation-based reasoning, addressing limitations in MLLMs for domain-specific tasks.
2. **Performance Gains**: Significant improvements in circuit analysis problems showcase the potential of MAPS in scientific reasoning tasks.
3. **Synthetic Data Generation**: The paper details a scalable method for generating synthetic datasets to fine-tune the PPM.
4. **Cross-Disciplinary Potential**: While demonstrated in circuit analysis, MAPS is conceptually extensible to other physical sciences with formalized rules and simulation tools.
5. **Clear Presentation**: The paper is well-organized and provides comprehensive explanations of the framework and its components.

**(c) Weaknesses**

1. **Limited Domain Coverage**: The evaluation focuses solely on circuit analysis, raising questions about generalizability to other disciplines.
2. **Error Analysis**: While errors are identified, the breakdown and mitigation strategies could be more detailed, particularly regarding perception and reasoning failures.
3. **Synthetic Data Dependence**: The reliance on synthetic data limits real-world applicability, as unseen or complex diagrams might diverge from the generated examples.
4. **Scalability Concerns**: Extending MAPS to disciplines without formal language descriptions or widely used simulators remains a challenge.

**(d) Reasons for the decision.**

MAPS is a novel and promising method that significantly improves multi-modal reasoning in circuit analysis and has the potential for broader scientific applications. While there are limitations in domain coverage and scalability, the strengths in innovation, methodology, and performance gains justify acceptance with a spotlight recommendation, and inspire the community to improve visual reasoning capabilities with a similar approach.