FEABench: Evaluating Language Models on Real World Physics Reasoning Ability

Authors: nayantara  mudur
hao  cui
subhashini  venugopalan
paul  raccuglia
michael  brenner
peter  norgaard
google  research
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-21 01:53:32

Abstract: Building precise simulations of the real world and using numerical methods to solve quantitative problems is an essential task in engineering and physics. We present FEABench, a benchmark to evaluate the ability of language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA) software. We introduce a multipronged evaluation scheme to investigate the ability of LLMs to solve these problems using COMSOL Multiphysics$^\textregistered$. We further design an LLM agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solution over several iterations. Our best performing strategy generates executable API calls 88\% of the time. However, this benchmark still proves to be challenging enough that the LLMs and agents we tested were not able to completely and correctly solve any problem. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would push the frontiers of automation in engineering. Acquiring this capability would augment LLMs' reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world.

Main paper: 
 introduction: 
 Several works have demonstrated the significant potential of large language models (LLMs) on analytical mathematical and scientific reasoning [1, 2, 3, 4, 5, 6] , and on programming tasks in generalpurpose languages like Python [7, 8, 9] . However, many quantitative tasks that are the cornerstone of engineering and scientific workflows require numerical analysis performed with sophisticated computational modeling software. For example, the development of a modern smartphone requires detailed modeling of the mechanical, thermal, and electrical behaviors of its many subcomponents. Finite element analysis (FEA) [10] is the approach typically used. This involves approximating the partial differential equations that describe the physical behavior of a system by building discretizations (or meshes) over geometries. This is then solved until convergence using numerical solvers. The vast relevance of FEA to domains like mechanical, biomedical and aerospace engineering, optics, and fluid dynamics has given rise to software such as Ansys ® [11], Abaqus ® FEA [12] , and COMSOL Multiphysics ® [13, 14] , that are indispensable to modelling complex systems with the interplay of non-trivial geometries, and multiple physical phenomena.In this paper, we begin to bridge this gap by examining the ability of LLMs to solve problems using finite element analysis (FEA) by generating Java API calls. This task requires reasoning over the problem and the ability to successfully operate the software. We selected COMSOL Multiphysics ® because it supports a wide range of physics models and can be applied to applied mathematics problems involving optimization or differential equations, to problems involving complex geometries. Because the FEA workflow is relatively canonical, the reasoning approach for modeling is similar to other FEA software, and all problems typically share a conceptual breakdown into a sequence of blocks that involve defining (1) Geometry, (2) Material properties, (3) Physics, (4) Meshing (5) Numerical Analysis and Solver settings, and ( 6) Postprocessing (details in Appendix C.2). COMSOL Multiphysics ® Version 6.1 is used in this work.Our contributions are the following: We create a benchmark of 15 quantitatively verifiable problems. These problems require numerical methods to solve, and the target values are expected to be largely independent of the modeling software. We measure different state-of-the-art LLMs on their performance with a multi-score evaluation strategy. Finally, we design a multi-turn LLM agent that interacts with the API and improves upon its answers over several iterations and examine its performance. 
dataset: 
 The benchmark problems are derived from tutorials in the COMSOL Multiphysics ® Application Gallery [15] which are often based on established validation problems or other sources [16, 17] . The input is a problem description describing a problem with a specific target quantity that needs to be computed (Figure 1 ). The problems span a range of real world / mathematical systems including the dynamics of a Lorenz attractor, heat transfer, and a Bragg reflector. Each entry consists of the following main fields: (1) Model Specifications: A condensed yet fully-specified description of the problem, and the physics / boundary conditions. This description is unambiguous in terms of details such as material properties or dimensions. ( 2) Selection Information: An engineer would rely on visual information from the Graphical User Interface (GUI) (Figure 3 ) for spatial information, particularly the numerical representation of geometrical entities, like points or boundaries. Since the agents do not have visual access to the GUI, we augment the model specifications with this information, to supply the model with the necessary representation of spatial information in the GUI. This information will be valid as long as the agent chooses to construct the geometry in a manner reasonably similar to how the ground truth (GT) geometry was created. (3) Plan: An explicit step-by-step description in natural language of the steps to be followed to solve the problem using COMSOL Multiphysics ® . Either of the following comprise self-sufficient tasks for the agent to solve: a) Model Specifications + Selection Information (designated ModelSpecs), or b) Plan (Plan). ( 4) Target Description: A brief phrase describing the quantity that needs to be computed. ( 5) Target Value: The correct value of the target physical quantity. Note, the target values are expected to be largely independent of the choice of modeling software. ( 6) Ground Truth Code: Ground truth code, that if executed, is able to successfully solve the problem and export the target value to a file. ( 7) Model Tree: Executing COMSOL Multiphysics ® calls essentially build a model that can be regarded as modifying a tree with certain predefined 'branches' such as 'geometry' and 'physics'.The model generated by executing code can thus be represented in a condensed form as a model tree (see Appendix A.3 for an example). Converting a tutorial to a benchmark problem requires ensuring that a verifiable artifact (the target) can be derived from it, generating inputs and ensuring they have all the information needed to solve the problem, generating the GT solution and verifying that it exports the correct target value (Appendix A). 
evaluation metrics: 
 Our evaluation metrics measure the correctness of the solution, even when a target value could not be exported, (Table 1 ). These additional metrics offer the advantage of being continuous, unlike the relative error, which can only be computed if the solution exported a valid target value. Metrics denoted by † require execution. We delineate the metrics, and the facets they probe here: (1) Executability † : 'Correct' (executable) lines as a fraction of parsed API calls in an LLM solution.(2) Model Tree Score † : Similarity score between the LLM code's model tree and the GT tree. This normalized so that a solution with no lines of code is scored 0. If equivalent to the GT code, it would be 1. This measures the alignment of the model's solution path with a successful path.(3) Code Similarity Score: Similarity score between the solution and the GT code. While this also measures alignment with the GT solution, two different code blocks could generate equivalent model subtrees. We mainly report this metric for completeness, and to motivate the need for the domain-specific metrics we introduce here. ( 4) Physics Metrics: The code is a basis to represent the actions the LM takes to model the problem. Since the physics block is the most challenging (Figure 2 ), we additionally analyze the LLM's physics action sequence by parsing and evaluating the physics commands. The most basic physics action sequence involves: Create Interface (eg: HeatTransfer) → Create Feature under Interface (eg: TemperatureBoundary) → Modify Feature Properties (eg: T0, to set a temperature). Our Physics Metrics include (a) Interface Factuality: How many defined interfaces were not hallucinated? (b) Interface / Feature / Feature Property Recall: How many interfaces / features / feature properties created / modified by the GT solution were also in the LLM solution? (c) Feature Dimension: For features created by both, does the feature's spatial dimension match? Some nested physics metrics, such as 'Feature Dimension' might not be valid for a specific problem, in case there was no matching feature between the ground truth and the LLM code: we mask out these problems while computing the means for that specific metric. ( 5) Target Relative Error † : There are various ways in which exporting the correct value may fail: a) the code fails to export b) it exports an 'empty' table or, c) it exports an incorrect table, such as a default value or the wrong quantity. We ask an LLM to check that the exported table matches the target description and that the exported quantity is not a default or a value reproduced as is from the problem description, and to parse the response, if so. Valid Target is the number of problems in the benchmark for which the LLM successfully judges the exported table to be valid. We then compute the relative error between the last value in the exported table and the GT answer. Comparison across LLMs Our baseline experiment queries different frontier LLMs with a oneshot prompt (see Appendix G). Table 2 records the means and standard errors on the means across problems. Mean executability is in the range 0.60-0.79, implying that the LLMs are familiar with the higher-level grammar and syntax of COMSOL Multiphysics ® API code. However, they are prone to hallucinating the choice of interface (factuality between [0.54 -0.85]). This is likely a significant contributor to the non-executable lines because an invalid interface renders all physics lines of code under this interface invalid. Claude 3.5-Sonnet has the best performance in the baseline setting. See Appendix F for a qualitative analysis.Comparison across Tasks and Agent Settings We fix the LLM to Gemini-1.5-Pro, and compare performance with the list of physics interfaces and features (PhyDoc In-Context) added to the prompt and in an interactive multi-turn agent setting. The agent is equipped with an 'evaluator' that both executes the code and records API replies and when executability crosses 90%, examines the code and the output table, and returns feedback, in an 'LLM-as-Judge' fashion. It thus computes a 'fitness' for each solution without having access to the GT target value. The fitness is used to track the best solutions. The CONTROLLERAGENT calls a CORRECTORSUBAGENT that calls tools that aid it in correcting the code given the 'current' code and feedback, execution history and the result of tool calls. An initial set of 20 samples are generated using PhyDoc In-Context and the best solutions are corrected for 20 steps (see Appendix C.2 for details). The best of all solutions is then identified and evaluated. Both PhyDoc In-Context and the agentic setting significantly reduce interface hallucinations, and the agent's mean executability is 0.88 (vs 0.62). Thus, grounding the LLM with information about or interaction with the API boosts performance. This indicates that at least some of the difficulty the LLMs encounter is due to their lack of knowledge of valid arguments to the API commands. We further examine whether the Plan task is easier. The comparison between task versions is of interest since both demand slightly different skills. For a person attempting to solve this task, ModelSpecs requires the individual to both infer implicit engineering and physical reasoning decisions to be made (e.g.: the LLM needs to infer that the correct representation of a cylinder's 2D cross-section is a rectangle) and further translate this to valid API calls. Plan explicitly describes all steps to be followed in natural language and requires the LLM to only translate the steps describing interactions with the GUI to valid calls. The comparison between the two tasks offers one way to decouple the complexity of making correct modelling decisions from the difficulty of translating decisions into syntactically correct API calls. If an LLM fared poorly at making the right modelling decisions but could reliably translate natural language instructions to API calls, it would find Plan an easier task. However, we find that a more explicit plan doesn't consistently boost performance.Although Relative Error | Strict is the principal metric one would ideally want to optimize for, we do not report means over that metric here since the LLM was only able to compute a Valid Target that was also within 10% of the correct answer for a single problem in the Multi-Turn Agent and ModelSpecs + PhyDoc experiments.For this problem, the correct target value is 18.3 • Celsius, and the value exported by the LLM is 20 • Celsius, which is a default temperature in COMSOL Multiphysics ® : this is an indicator of the solution not being solved correctly. While a stricter relative error threshold would filter out such serendipitous matches, this risks filtering out problems in which a solution might be conceptually correct but differs from the target because of say, differences in solver and mesh sizes. In Figure 2 , we used the initial population of samples generated by the LLMs to analyze executability within different blocks of code. The physics block has the lowest executability, thus motivating our focus on metrics that seek to measure performance on the physics decisions (API calls) made by the LLM.We also excluded any problems whose ground truth code took over a minute to solve.Without any modification, the tutorials might export a single value, a table, or not export any target quantity at all, with the final output being qualitative in nature, such as in the form of plots or figures.For our benchmark, however, we specifically wanted every problem to have a numerically verifiable target value, in order for there to be an absolute notion of correctness (i.e. if the code was fully correct, and aligned with the intent of the problem, it should be able to export this value). This also enables easier evaluation of the problems. The following procedure and guidelines were adopted to curate the benchmark:• For an initial set of 2-3 problems, model specifications and plans were annotated by hand, by an expert user of COMSOL Multiphysics ® .Figure 3 : Screenshot of the graphical user interface for the correctly solved problem in Figure 1 .• For subsequent problems, we speed up the benchmark generation procedure by following an initial LLM-assisted data generation process, with the final verification steps involving humans. An LLM is provided with a tutorial, as well as a two-shot prompt with the expert annotated model specifications. • The LLM is entasked with returning a model specification for the tutorial that has the same format. This requires the LLM to identify an appropriate target value from the tutorial which it does from either the text or the figures, and returning a model specification for computing this target value. • The LLM is then asked to create a plan corresponding to the model specifications, using a two-shot prompt with two plans. The utility of the tutorials are that the plan is closest to the GUI instructions listed in the tutorial, while model specifications is more concise. • A ground truth code that can compute the correct value is then generated for the problem.We manually verify that the ground code when run, exports the desired target value. This step also involves simultaneously ensuring that all information required to build the model is contained in the plan, and in the model specifications by editing the LLM-generated drafts and ensuring that no Translation Errors are encountered when parsing and executing the ground truth code in COMSOL Multiphysics ® using the bridge described in Appendix C.1 or that any errors if encountered are in non-crucial lines and do not prevent the solution from being computed. Any missing or incorrect information is fixed, and the selection_information field, that contains numerical identities of boundaries and points is also created. • We add an instruction to export the output to OUTPUT_PATH/output.txt in the model specifications and plan.A.3 Fields from an example entry:Here is an example of the information saved for one of the problems, comsol_453 ... model . component (" comp1 ") . physics () . create (" ht " , " HeatTransfer " , " geom1 ") ; ... model . component (" comp1 ") . physics (" ht ") . create (" temp1 " , "T emp era tur eBo un dar y " , 1) ; model . component (" comp1 ") . physics (" ht ") . feature (" temp1 ") . set (" T0 " , "273.15[ K ]") ; ... model . result () . table (" tbl1 ") . comments (" Point Evaluation 1") ; model . result () . numerical (" pev1 ") . set (" table " , " tbl1 ") ; model . result () . numerical (" pev1 ") . setResult () ; model . result () . table (" tbl1 ") . save (" OUTPUT_PATH / output . txt ") ; The LM output is first parsed to identify the block with Java API calls, and further parsed to pythonize the lines (Appendix C.1). This filters out lines that are not code or cannot be pythonized and results in a sequence of COMSOL Multiphysics ® API calls and their 'pythonized' counterparts, all of which start with model. and end with ';'.The pythonized lines are then passed to the MPh client, and replies for each line are received. We parse API replies using the following patterns. A reply containing any of the following ['Messages', 'has no attribute', 'No matching overloads', 'invalid syntax', 'Exception', 'is not defined'] are considered Syntax Errors. Replies with ['Ambiguous', 'comma', 'No Model set'] are Translation errors. The last category category is rare in our experiments and are occasionally encountered when we tested adding new problems to the benchmark that contained lines that weren't translated correctly in the query: the first two flag errors in the query to COMSOL Multiphysics ® via Mph, while the last indicates that an action is being done on a non-existent model, which is inconsistent with the setup of the code. All other replies are designated Correct.(1)We use the difflib [22] package, that computes a score between 0 and 1 as a measure of string similarity, using the ratio of the lengths of the longest matched subsequences to the ratio of the lengths of strings being compared. Code Similarity reflects this score between the generated code and the ground truth code. It is not surprising that this metric has the least change since significantly different blocks of code might yield the same answer. As a specific example, a model.study("std1").run(); will leverage COMSOL Multiphysics ® 's default numerical solver for the problem. However, this could also be represented explicitly using large blocks of model.sol("sol1")... lines in the Ground Truth Code field.The model tree representation of the model built by the language model can be extracted, and one can use the same similarity score as above to compute a similarity score relative to the target tree. We expect this to be a more reliable measure of alignment since different blocks of code that build the same model will have the same model tree (addressing the case described in Code Similarity). Using the formula below, the score will be 1.0 if the trees are identical, and 0.0 if the trees are equivalent to a tree before any code is run.M odelT reeScore = Score(LM, GT) -Score(Empty, GT) 1.0 -Score(Empty, GT)The following is an empty tree, corresponding to a model that has only been initialized, before any code is run. There are various ways in which computing the correct value and exporting it to a table may fail: a) the LLM's code forgets the export command to the API and no table is exported b) an empty table is exported or, c) a table containing an incorrect value is exported, such as a default value or the wrong quantity (eg: time instead of temperature). Failure modes b) and c) are far more common than a) and occur when the code is not fully correct and the partially constructed COMSOL Multiphysics ® model exports nothing or an incorrect value. For instance, a partially solved model that was asked to compute the temperature at time=190s might export a table where the last value was 190 but because of errors in model construction, no temperature was exported. In such a case if the ground truth answer is say, 185 • C, without verifying the physical quantity, one would mistakenly evaluate the algorithmically parsed figure 190 to be quite close to the target. In other cases, the software might export a default such as 293.15 K if the solver did not solve correctly.If a table containing the target quantity is exported, it is first read and parsed. The last value in the table is algorithmically extracted. To address this problem, we ask an LLM (Gemini-1.5-Pro), to extract the exported value and units from the table, if it is a match for the target description, and minimize the chances of incorrectly evaluating these failure modes as valid solutions.You are provided with a table that was exported by a model built in COMSOL. The table * should * contain the EXPECTED TARGET QUANTITY. The following failure modes may occur when the model is not built correctly: 1. The table might be empty or might export a physical quantity that is different from the expected target quantity. 2. The table might export the same physical quantity, but the quantity is just an initial or boundary condition, or a default value that was exported, instead of the result of genuinely numerically solving the problem. You can find numbers already in the problem description in 'PROBLEM'. Default values include 20degreesCelsius, 293.15 K, 0 etc. Carefully examine the 'TABLE' and compare it with the units and description of the expected target quantity and the numbers in 'PROBLEM' to assess whether the table exported a value that was the result of genuinely numerically solving the problem. You must return TARGET VALUE and TARGET UNITS in json format if the table was the result of genuinely solving the model, computing a solution and exporting it. Return 'N/A' for both fields if the table suffers from either of the failure modes described above. We then compute the number of problems for which the LM was able to parse the reply and convert it to a JSON. This fraction is the number we report as Valid Target.Our strict filter for whether a model has truly solved the problem is to take the subset of problems for which the problem was judged to be a valid export by the LLM, and to consider the algorithmically parsed last value. We then compute the relative error of this value against the ground truth target value. If this value is less than 10%, we consider it valid.The interface lines are parsed from the ground truth code by finding lines that fit the regex pattern for interface creation. Likewise for the feature creation and feature property modification lines. Each of these lines of codes can be considered as an "Action" consisting of an Action Type (eg: Create Interface) with corresponding Arguments (eg: Interface tag, Name of the Interface, Geometry).model.component("comp1").physics().create("Interface_tag", "InterfaceName", "Geometry_tag");model.component("comp1").physics().create("ht", "HeatTransfer", "geom1"); Create Feature: model.component("comp1").physics("Interface_tag").create(" Feature_tag", "FeatureName", Dimension);model.component("comp1").physics("ht").create("temp1", "TemperatureBoundary", 1);Modify Feature Property: model.component("comp1").physics("Interface_tag") .feature("Feature_tag").set("Param", "Value");model.component("comp1").physics("ht").feature("temp1").set("T0", "1000[degC]");We check whether the Interface name exists in a list of known COMSOL Multiphysics ® interfaces. If it exists in this list, we assign it a factuality of 1, else 0.How many GT interface creation actions (ignoring Interface_tag) were also in the LM code? This checks whether the same interface was defined on the same geometry. 'nan' if there are no interfaces in the GT (not encountered in our dataset).Since multiple features may be created under the same interface (eg: 2 Boundary Conditions with different temperatures), we compute the occurrences of each GT feature name in the GT code and in the LM code, and a recall for each GT feature name, and then average over all GT features. In our implementation, if no GT features are defined, a) AND no LM features are defined the recall is 1, b) but LM features are defined, the recall is 0.Let F c be all the GT features that are also created by the LM solution. Let Dim c be the set of F c such that the LM feature has the same dimension as the GT feature. Feature Dimension = |Dimc| |Fc| This is a correctness and physics reasoning metric as opposed to an alignment-focused metric since creating a TemperatureBoundary with dimension 2 attempts to create a 2D temperature boundary condition. Creating a TemperatureBoundary with dimension 1 attempts to create a temperature on an edge. Thus this measures the LM's ability to correct deduce the spatial dimension of boundary conditions or other features from the context of the problem.This compares the modify feature property actions. It computes how many GT modify feature property actions were also in the ground truth, ignoring differences in Interface_tag and Feature_tag. If no GT properties are modified, a) AND no LM features are modified the recall is 1, b) but LM features are modified, the recall is 0.The raw output of the LLM is a string containing COMSOL Multiphysics ® API commands in Java. An interface between Python and COMSOL Multiphysics ® is needed to execute this code and interact in other ways with the API. We use the Python package MPh [23] and Rpyc for this. MPh is a scripting interface built on JPype [24] that enables a Python program to communicate with and build a model in COMSOL Multiphysics ® . Each Java API command in the LM's output can be 'pythonized' algorithmically. In most cases, the pythonized line is near identical to the Java line. However, due to differences in Java and Python syntax there exist some corner cases that need to be handled separately. Eg: 'new String[]' is exclusively a Java construction, while the notation for booleans in Python is True / False as opposed to true / false in Java. Thus a 'pythonizer' is constructed that parses and translates Java API calls to their Python counterparts.The setup involves the following assumptions: an MPh client object is created. This behaves like a stateful 'sandbox', where models can be built by LLMs, code can be evaluated, or information such as the current state of the model tree, properties under a node and the exported table can be queried and retrieved. Although multiple models can be created and set under the client, for simplicity we work with settings that involve a single model. Before running a new solution, the existing model is deleted and a new blank model is created. The LLM actions will modify this blank model. Thus, by design, all lines of code the LLM outputs, should start with 'model.' and end with ';'.1. Geometry, if any: This involves identifying the dimensionality of the problem, and constructing a representation of the object being modelled, say a cup, by creating and composing primitive shapes such as ellipses or rectangles to build the object. While already constructed geometries can also be imported from other software such as CAD, in our benchmark, we currently restrict ourselves to models for which we construct the geometry from scratch in COMSOL. This typically starts with a 'model.component ("comp1") .geom' pattern. 2. Physics: This will include specifying all the physical conditions for the problem, including initial or boundary conditions, forces, properties or in the case of mathematics problems, the differential equation. This typically starts with a 'model.component ("comp1") .physics' pattern. Some problems may additionally have lines that begin with 'model.component ("comp') .multiphysics', and set up the coupling between different kinds of physical phenomena. We categorize these lines, if any as 'physics' in Figure 2 . 3. Material: Creating materials and assigning them to domains. One can either assign known materials such as 'Copper' and the object will inherit the default properties of that material, or define a blank material and its properties such as conductivity from scratch. This typically starts with a 'model.component ("comp1") .material' pattern. 4. Mesh: Usually a shorter step that involves meshing the surfaces of the geometry to set up elements. This typically starts with a 'model.component ("comp1") .mesh' pattern. 5. Study / Solver: This involves specifying the conditions of the analysis and solver, such as the number of timesteps. While the solver code can be modified to override defaults, COMSOL also has the ability to automatically populate the model with the default solvers most apt for a given problem. This typically starts with a 'model.study' or 'model.sol' pattern respectively. In Figure 2 , we categorize both patterns as 'solver'. 6. Results: Once the numerical solver has completed the analysis, one will likely postprocess the problem, in order to generate desired plots or tables. This typically starts with a 'model.result' pattern.Our single-turn results, particularly on executability and hallucinated interfaces, highlight the need to ground the LLM's responses with feedback from and knowledge about how to use the API. Given a single try, the LLM is likely to make some errors, and we hypothesize that receiving feedback informative of the nature and location of the errors will allow it to correct those errors in a directed fashion. We thus build an environment in which the LLM can interact with the API, receive feedback and attempt to correct its solution. We design a nested multi-agent system that interacts with the COMSOL Multiphysics ® API, as well as tools (or specialized functions) that contain information relevant to correcting a problem. Each agent has a specific role and input context.Design Principles (1) To minimize failures or longer-than-desired chains of calls, we adopt an algorithmic sequence of agent calls except when we call tools. ( 2) High executability does not guarantee alignment or correctness, since API messages alone are not informative about inconsistencies with the problem description, such as incorrect units. We thus combine an LLM's feedback with the API messages to flag such inconsistencies. (3) Calling tools may fail in cases where the argument to the tool is invalid or the call sequence could not be parsed. In such cases we ensure the agent system can still attempt to correct the code, without the relevant information from tool calls.CONTROLLERAGENT: The main agent that tries to solve the problem description by generating solutions, interacting with the API and calling subagents.Input Context: Problem description.Working: This samples an initial population of N(=20) solutions using PhyDoc In-Context. Each solution is evaluated by the Evaluator. A fitness score, between 0 and 2, is computed for each solution, using the following formula: Executability + ExportSuccessful where ExportSuccessful is 1 if (the solution computed a value AND had executability above 90%) and 0 if not. The controller agent tracks a set of best replies using their fitness. The set of best replies stores at least B(=1) solution, as well as all solutions that successfully computed a value. This agent also determines the context to be sent to the CorrectorSubAgent, using the following algorithm:• Solution to iterate on: We use an iteration criterion inspired by the Markov Chain Monte Carlo (MCMC) acceptance criterion. The solution to iterate on (rendered in the prompt to the CorrectorSubAgent as "CURRENT CODE") is (a) the last solution if the last solution has equal fitness as the best solution, and (b) the last solution if a random float between [0, 1] is less than α = Last_F itness Best_F itness , else the best solution. • ExecutionHistory: The best solutions, if not already used in context upto a maximum of 3 best solutions, in addition to the last N_bad(=1) replies, if not already in context.EVALUATOR: This returns the feedback for a solution in a 'score' dictionary (Left panel, Figure 5 ) Input Context: A solution.Working: The evaluator combines execution feedback with subjective feedback from an LLM. The evaluator executes the parsed solution in the API, and stores the API messages as well as the Executability. If Executability exceeds 90%, the VerifierLLM examines the code as well as the computed value in 'output.txt' and returns subjective feedback on changes that need to be made. To facilitate this, the VerifierLLM sets an analytical guess for what the solution should be at the onset of the problem, and compares the computed value with this guess when it evaluates a solution. The API feedback provides a signal on syntactical correctness and the VerifierLLM provides a signal on completeness. Note, this evaluator is not aware of the GT target value.CORRECTORSUBAGENT: This returns an updated solution.Input Context: Problem description, Current Code and Feedback, Execution History Components: ToolLookupAgent Working: This calls the ToolLookupAgent and retrieves its reply. It then includes this reply to the rest of the context received from the ControllerAgent to propose the next solution.TOOLLOOKUPAGENT: This calls tools and returns the information retrieved from them.Input Context: Feedback Components: ToolRegistry Working: The LLM is shown tool descriptions and the input context and must return a list of tool calls, as structured classes using the Langfun [25] package consisting of the tool name and its arguments. If successfully parsed, each tool is called with its arguments and the replies are concatenated (see Figure 5 for the feedback and reply for a single step). The tools in the registry are:1. QUERYPHYSICSINTERFACES: This returns a list of valid physics interfaces. 2. QUERYPHYSICSFEATURES: This returns the features under an argument interface or a list of known features under interfaces. 3. QUERYMODELTREEPROPERTIES: The LLM must call this tool with a path argument ('/physics/Heat Transfer in Solids/Solid 1' in Figure 5 ) to receive the properties under the node corresponding to path. 4. RETRIEVEANNOTATEDSNIPPETS: To call this tool, the LLM must specify a branch -one of the conceptual blocks such as physics or geometry -and a query -a brief natural language description of a specific step. In Figure 5 , the LLM first called this tool with the branch 'geometry' and the query 'Create a 2D axisymmetric geometry in...'. A retriever then looks up the annotated library and retrieves 3 annotations along with their code snippets, most similar to the query made. Thus, this allows the LLM to search a library of code snippets to find the correct ways to express certain steps in code, simulating how a human unfamiliar with a coding language would look up similar examples of code.At the end of this experiment, the CONTROLLERAGENT saves its best solutions as well as other intermediate states. In order to help the LLM learn how to appropriately format a valid path, say to the 'Solid' feature, the current state of the model tree is shown to the ToolLookupAgent LLM. It also has a history of unsuccessful (incorrectly formatted) paths in previous queries to this tool, in order to minimize the chances of incorrectly calling this tool with an invalid path.We use the Discovery Engine API [26] with the model name 'semantic-ranker-512-003' to rank and retrieve the top 3 annotations most similar to the query snippet. The annotation library was generated by taking tutorials and splitting them into code blocks using the patterns described in C.2. There are 768 pairs of annotations and snippets across all branches of code. Here is an example of an annotation 'summary' and its snippet:Summary: Defining a transient study with a time range from 0 to 0.025 seconds with a step of 1 second. The study will solve for the "spf" physics interface, and a relative tolerance of 0.001 will be used. The number of solver iterations will be automatically determined based on the time step.model . study () . create (" std1 ") ; model . study (" std1 ") . create (" time " , " Transient ") ; model . study (" std1 ") . feature (" time ") . setSolveFor ("/ physics / spf " , true ) ; model . study (" std1 ") . feature (" time ") . set (" tlist " , " range (0 ,0.025 ,1) ") ; ... model . study (" std1 ") . feature (" time ") . set (" solnum " , " auto ") ; Figure 6 depicts the blockwise executability in the initial sample relative to the best solution across problems. The standard deviations in the best case are higher since we have 1 best solution for each problem, and 20 samples per problem in the initial population. Figure 7 plots the Executability as well as the number of errors over solution iteration. The evolution of the metrics isn't monotonic and in some cases the agent gets stuck on the same solution for some iterations, or takes an incorrect turn. We added the acceptance criterion to minimize the number of iterations required to "escape" an incorrect turn.In Figure 8 , we delve into the differences between the LLM-generated code for the ModelSpecs task in the baseline (one-shot) setting with Gemini-1.5-Pro, relative to the ground truth code, for the problem in Appendix A.3. At a high-level, the LLM's solution consists of API calls that qualitatively posess the same structure and grammar as in the GT code. This is a problem that requires the LLM to represent the cylindrical cross-section of a cylinder as a rectangle in 2D, with the axisymmetric condition applied for rotational symmetry about the cylinder's axis. The LLM instead creates a 3D geometry and attempts to create a rectangle. This doesn't work as is indicated by the error message, since the rectangle is a 2D construct and cannot be directly created in 3D. Since the rectangle creation action fails, no 'r1' node is created, and any subsequent actions that act on the 'r1' node cannot be executed. This pattern of non-executability is also observed downstream, where all actions on the 'ht' node are rendered invalid because the 'ht' node could not be created in the first place. Note, if the LLM had chosen a 2D geometry, or a 2D axisymmetric geometry, the geometry lines of code would have been correct. The reason they fail is because of an incorrect (3D geometry) decision taken first.Next, the LLM chooses to set the thermal conductivity under the materials node. These lines of code are executable and this may be a valid choice, if the physics node is properly able to query properties redefined under the materials node.The LLM tries to create a 'HeatTransferinSolids' interface. This is a subtle error. Heat Transfer in Solids is indeed the correct natural language name for this interface under COMSOL Multiphysics ® and is often referred to as such in documentation on the internet. However, this is not the correct syntactical name for the interface, which, as can be seen in the GT code, is 'Heat-Transfer'. Errors like these are likely why the adding the list of physics interfaces and features to the prompt (PhyDoc In-Context) improve performance on both tasks. Since the LLM's chosen interface and features differ from the ground truth in this example, the Interface Recall and Feature Recall metrics are both 0, as is the Interface Factuality metric (since 'HeatTransferinSolids' does not exist). The GT code modifies 5 features, of which the LLM only modifies 1 (setting T 0 to 273.15 K). Thus the Modify Feature Property score is 0.2.In the results section, the model incorrectly attempts to set the properties 'geom', 'x' and 'y; under the point evaluation node. All three lines trigger 'Unknown property' exceptions.Figure 8 : Comparing a Ground Truth code with the LLM-generated code. The ochre-colored lines or arguments in the GT code demarcate lines or arguments that were modified or absent in the LLM generated code. The green (red) lines in the LLM-generated code denote lines that were (were not) executable. The arrows against the red lines show the abbreviated API message returned for each non-executable line.G. model . component (" comp1 ") . material () . create (" mat1 " , " Common ") ; model . component (" comp1 ") . material (" mat1 ") . propertyGroup () . create (" Enu " , " Young ' s modulus and Poisson ' s ratio ") ; model . component (" comp1 ") . material (" mat1 ") . propertyGroup (" Enu ") .set (" E " , "2.1 e11 ") ; model . component (" comp1 ") . material (" mat1 ") . propertyGroup (" Enu ") .set (" nu " , "0.3") ; model . component (" comp1 ") . physics () . create (" solid " , " SolidMechanics " , " geom1 ") ; model . component (" comp1 ") . physics (" solid ") . create (" sym1 " , "SymmetrySolid " , 1) ; model . component (" comp1 ") . physics (" solid ") . feature (" sym1 ") . selection () . set (2) ; model . component (" comp1 ") . physics (" solid ") . create (" sym2 " , "SymmetrySolid " , 1) ; model . component (" comp1 ") . physics (" solid ") . feature (" sym2 ") . selection () . set (1) ; model . component (" comp1 ") . physics (" solid ") . create (" bndl1 " , "BoundaryLoad " , 1) ; model . component (" comp1 ") . physics (" solid ") . feature (" bndl1 ") . selection () . set (4) ; model . component (" comp1 ") . physics (" solid ") . prop (" Type2D ") . set (" Type2D " , " PlaneStress ") ; model . component (" comp1 ") . physics (" solid ") . prop (" d ") . set (" d " , ".1") ; model . component (" comp1 ") . physics (" solid ") . feature (" bndl1 ") . set (" LoadType " , " FollowerPressure ") ; model . component (" comp1 ") . physics (" solid ") . feature (" bndl1 ") . set (" FollowerPressure " , " -10 [ MPa ]") ; model . component (" comp1 ") . mesh (" mesh1 ") . create (" map1 " , " Map ") ; model . component (" comp1 ") . mesh (" mesh1 ") . feature (" size ") . set (" hauto " , 4) ; model . component (" comp1 ") . mesh (" mesh1 ") . run () ; model . study () . create (" std1 ") ; model . study (" std1 ") . create (" stat " , " Stationary ") ; model . study (" std1 ") . feature (" stat ") . setSolveFor ("/ physics / solid " , true ) ; model . study (" std1 ") . run () ; model . result () . numerical () . create (" pev1 " , " EvalPoint ") ; model . result () . numerical (" pev1 ") . selection () . named (" sel1 ") ; model . result () . numerical (" pev1 ") . set (" probetag " , " none ") ; model . result () . table () . create (" tbl1 " , " Table ") ; model . result () . table (" tbl1 ") . comments (" Point Evaluation 1") ; model . result () . numerical (" pev1 ") . set (" table " , " tbl1 ") ; model . result () . numerical (" pev1 ") . set (" expr " , new String []{" solid . syy "}) ; model . result () . numerical (" pev1 ") . set (" unit " , new String []{" N / m ^2"}) ; model . result () . numerical (" pev1 ") . set (" descr " , new String []{" Stress tensor , yy -component "}) ; model . result () . numerical (" pev1 ") . set (" const " , new String [][]{{" solid . refpntx " , "0" , " Reference point for moment computation , x -coordinate "} , {" solid . refpnty " , "0" , " Reference point for moment computation , y -coordinate "} , {" solid . refpntz " , "0" , " === Now generate the JAVA API code to compute the target quantity for the problem below, by following the plan described. Export the table containing the target quantity to OUT-PUT_PATH/output.txt. java '' ' model . component () . create (" comp1 " , true ) ; model . component(" comp1 ") . geom () . create (" geom1 " , 2) ; model . component(" comp1 ") . mesh () . create (" mesh1 ") ; model . component (" comp1 ") . geom (" geom1 ") . create (" e1 " , " Ellipse ") ; model . component (" comp1 ") . geom (" geom1 ") . feature (" e1 ") . set (" semiaxes " , new double []{3.25 , 2.75}) ; model . component (" comp1 ") . geom (" geom1 ") . feature (" e1 ") . set (" angle " , 90) ; model . component (" comp1 ") . geom (" geom1 ") . create (" e2 " , " Ellipse ") ; model . component (" comp1 ") . geom (" geom1 ") . feature (" e2 ") . set (" semiaxes " , new int []{2 , 1}) ; model . component (" comp1 ") . geom (" geom1 ") . feature (" e2 ") . set (" angle " , 90) ; model . component (" comp1 ") . geom (" geom1 ") . create (" dif1 " , "Difference ") ; model . component (" comp1 ") . geom (" geom1 ") . feature (" dif1 ") . selection (" input ") . set (" e1 ") ; model . component (" comp1 ") . geom (" geom1 ") . feature (" dif1 ") . selection (" input2 ") . set (" e2 ") ; model . component (" comp1 ") . geom (" geom1 ") . run () ; model . component (" comp1 ") . geom (" geom1 ") . run (" fin ") ; model . component(" comp1 ") . selection () . create (" sel1 " , " Explicit ") ; model . component (" comp1 ") . selection (" sel1 ") . geom (" geom1 " , 0) ; model . component(" comp1 ") . selection (" sel1 ") . set(3) ; model . component (" comp1 ") . selection (" sel1 ") . label (" O ") ; 
discussion: 
 Our benchmark seeks to inspire rigorous evaluations of the capabilities of LLMs on solving problems that require simulating physical phenomena in the real world and performing numerical analysis. Such problems are ubiquitous in science and engineering, and solving them requires synthesizing reasoning over the physics domain with the ability to leverage numerical analysis software such as FEA. Our dataset serves as a novel testbed to evaluate the ability of LLMs to interact with feedback from an execution environment, correctly analyze and attribute errors and correct them. Future directions include devising agent strategies that boost performance, adding more benchmark problems, increasing problem complexity such as working with more complex geometries, and examining whether performance improves by leveraging GUI visual information. An LLM system that can successfully master FEA software would unlock the ability to solve a range of real-world engineering and physics problems.1. Simpler Geometry: COMSOL Multiphysics ® can be used to analyze the physics of systems involving intricate geometries such as microwaves or transformers. In these cases, in practice, most problems involve importing a pre-built geometry object that might have been built externally using Computer-Aided Design (CAD) software and to then perform the remaining analysis. Since we wanted to explore the ability to solve the problem end-to-end and without requiring imports of derived objects, we restrict ourselves to problems that did not require imports of geometry, or any other files.2. Tutorial / Code Simplicity: We additionally chose problems that did not involve multiple 'Model' JAVA classes and restricted ourselves to tutorial documents with fewer than 20 pages. The first requirement is a consequence of how our connection to the COMSOL Multiphysics ® sandbox is set up, and to make the problem easier for the models to attempt to solve. We additionally ensured that the problems were amenable to computing a numerical artifact. 
 
 Appendix:  



Reviews: review: title: title: Review of "FEABench: Evaluating Language Models on Real World Physics Reasoning Ability"
main_review: main_review: # Summary
"FEABench: Evaluating Language Models on Real World Physics Reasoning Ability" presents an innovative approach to assessing the capabilities of Large Language Models (LLMs) in solving real-world physics problems using the finite element method, a sophisticated technique often taught at the graduate level in math and physics courses. The paper makes several significant contributions to the field:
1. Development of a New Benchmark: The authors introduce FEABench, a benchmark consisting of 13 quantitatively verifiable problems derived from tutorials in the COMSOL Multiphysics⃝R Application Gallery. This benchmark is specifically designed to test the application of LLMs to complex physics problems that require substantial computational and analytical skills.
2. Multi-score Evaluation Strategy: To systematically measure the performance of LLMs on this benchmark, the paper proposes a multi-score evaluation strategy. This strategy employs a series of metrics to provide a comprehensive assessment of the models' capabilities.
3. Complexity Analysis through Prompt Engineering: The paper also explores the complexity of the dataset by employing prompt engineering techniques. The findings indicate that state-of-the-art LLMs still struggle to perform well on this challenging dataset, highlighting the need for further advancements in the field.

# Evaluation
The paper is well-organized and introduces a novel problem area in the application of LLMs to the Finite Element Method (FEM). The contributions are clearly defined and address a real gap in the current evaluation strategies for LLMs in the domain of physics.
However, the paper could be strengthened by providing a deeper connection with the broader field of numerical analysis using LLMs. A more detailed discussion on how these methods integrate with existing numerical techniques could enhance the relevance and applicability of the research.

# Recommendation
Overall, the paper is of high quality and addresses an innovative and challenging application of LLMs. The introduction of FEABench provides a valuable resource for the community, and the multi-score evaluation strategy offers a nuanced approach to assessing model performance.
Given the novelty of the problem addressed and the contributions made, I recommend the acceptance of this paper. However, I would encourage the authors to consider expanding the discussion on the integration of LLMs with traditional numerical analysis techniques to provide a more comprehensive overview of the field.
title: title: A novel idea but need more details, analysis and comparisons
main_review: main_review: Pros:

1.The paper addresses an important and practical problem of evaluating the ability of language models to simulate and solve physics problems using FEA software.
2. The benchmark dataset is well-curated and includes a range of problems from different domains, providing a comprehensive evaluation of the LMs' performance.
3. The multipronged evaluation scheme and the introduction of the LM agent with API interaction are innovative and provide valuable insights into the LMs' capabilities and limitations.

Cons:

1. Could you provide more details on the selection criteria and generation procedure for the benchmark dataset? How did you ensure that the problems are representative of real-world physics and engineering problems?
2. What are the potential limitations and sources of error in the evaluation scheme? How did you handle cases where the LM solutions were not executable or did not export valid target values?
3. The paper lacks a detailed analysis of the empirical findings and performance of the LMs and LM agents on the benchmark problems. The authors only provide summary statistics and do not delve into the specific challenges faced by the models or the reasons for their performance.
4. The paper does not provide a comparison with existing multi-agent methods or benchmarks in the field. It would be valuable to understand how the proposed benchmark and evaluation scheme compare to other approaches in evaluating the performance of LMs in physics reasoning tasks.
5. Can you discuss the potential biases or limitations of the benchmark? Are there any specific types of problems or scenarios that may be more challenging for LMs? How do you plan to address these limitations in future work?
title: title: Evaluation of Language Models for Physics Problem-Solving Using FEA Software
main_review: main_review: This paper introduces FEABench, a novel benchmark for evaluating the ability of large language models (LLMs) to solve real-world physics and engineering problems using finite element analysis (FEA) software. The authors present a multi-faceted evaluation scheme to assess LLMs' capabilities in interacting with COMSOL Multiphysics software through its API to solve a range of physics and engineering problems.

Pros:
1. Novel and important contribution: The paper addresses a significant gap in evaluating LLMs on real-world engineering tasks, specifically in the domain of finite element analysis.
2. Dataset creation: FEABench includes 13 quantitatively verifiable problems across various physics domains, providing a comprehensive test of LLM capabilities.
3. Multi-faceted evaluation: The authors develop a range of metrics to assess different aspects of LLM performance, including code executability, model tree similarity, and physics-specific metrics.
4. Innovative approach: The authors develop an LLM agent equipped with the ability to interact with the FEA software API and iterate on solutions.
5. Experimentation: The paper evaluates multiple state-of-the-art LLMs and compares different prompting strategies.
Clear presentation: The methodology, metrics, and results are generally well-explained and illustrated with helpful figures and tables.

Cons:
1. Limited scope: The benchmark is currently restricted to problems solvable with COMSOL Multiphysics. Including other FEA software could enhance generalizability.
2. Relatively small dataset: With only 13 problems, the benchmark may not fully capture the diversity of real-world FEA tasks.
3. Lack of human baseline: The paper would benefit from comparing LLM performance to that of human engineers on the same tasks.
4. Potential bias in problem selection: The criteria for selecting problems could be more thoroughly justified to ensure a representative sample of FEA tasks.
5. Limited analysis of failure modes: While the paper presents performance metrics, a deeper analysis of why LLMs fail on certain tasks could provide valuable insights.
6. Reproducibility concerns: The reliance on proprietary software (COMSOL) may limit the reproducibility of the benchmark by other researchers.
7. Questionable real-world relevance: The paper doesn't adequately demonstrate how the ability to generate COMSOL API calls translates to actual problem-solving capabilities in engineering contexts.

General Comments:
- The paper makes a sizable contribution to the field of AI evaluation, specifically in assessing LLMs' capabilities in complex engineering tasks. The FEABench provides a  tool for measuring progress in applying LLMs to real-world physics and engineering problems. 
- However, the benchmark's current limitations in scope and scale somewhat constrain its immediate impact. Expanding the number and diversity of problems, including problems from other FEA software packages, and providing a human performance baseline would significantly strengthen the work.
- The paper does highlight the current limitations of LLMs in solving complex physics problems, but it fails to provide substantive insights or clear paths forward. In its current form, the work serves more as a proof-of-concept for evaluating LLMs on FEA tasks rather than a robust, widely applicable benchmark.
- Despite these limitations, the paper represents an important step forward in evaluating LLMs on practical engineering tasks. It opens up new avenues for research in applying AI to complex scientific and engineering problems, potentially leading to significant advancements in how these tasks are approached in the future.