LLMPhy: Complex Physical Reasoning Using Large Language Models and World Models

## Abstract

Abstract Physical reasoning is an important skill needed for robotic agents when operating in the real world. However, solving such reasoning problems often involves hypothesizing and reflecting over complex multi-body interactions under the effect of a multitude of physical forces and thus learning all such interactions poses a significant hurdle for state-of-the-art machine learning frameworks, including large language models (LLMs). To study this problem, we propose a new physical reasoning task and a dataset, dubbed TraySim . Our task involves predicting the dynamics of several objects on a tray that is given an external impact – the domino effect of the ensued object interactions and their dynamics thus offering a challenging yet controlled setup, with the goal of reasoning being to infer the stability of the objects after the impact. To solve this complex physical reasoning task, we present LLMPhy \operatorname{LLMPhy} , a zero-shot black-box optimization framework that leverages the physics knowledge and program synthesis abilities of LLMs, and synergizes these abilities with the world models built into modern physics engines. Specifically, LLMPhy \operatorname{LLMPhy} uses an LLM to generate code to iteratively estimate the physical hyperparameters of the system (friction, damping, layout, etc.) via an implicit analysis-by-synthesis approach using a (non-differentiable) simulator in the loop and uses the inferred parameters to imagine the dynamics of the scene towards solving the reasoning task. To show the effectiveness of LLMPhy \operatorname{LLMPhy} , we present experiments on our TraySim dataset to predict the steady-state poses of the objects. Our results show that the combination of the LLM and the physics engine leads to state-of-the-art zero-shot physical reasoning performance, while demonstrating superior convergence against standard black-box optimization methods and better estimation of the physical parameters. Further, we show that LLMPhy \operatorname{LLMPhy} is capable of solving both continuous and discrete black-box optimization problems.

## 1 Introduction

Many recent Large Language models (LLMs) appear to demonstrate the capacity to effectively capture knowledge from vast amounts of multimodal training data and their generative capabilities allow humans to naturally interact with them towards extracting this knowledge for solving challenging real-world problems. This powerful paradigm of LLM-powered problem solving has manifested in a dramatic shift in the manner of scientific pursuit towards modeling research problems attuned to a form that can leverage this condensed knowledge of the LLMs. A few notable such efforts include, but not limited to the use of LLMs for robotic planning (Song et al., ; Kim et al., ), complex code generation (Tang et al., ; Jin et al., ), solving optimization problems (Yang et al., ; Hao et al., ), conduct sophisticated mathematical reasoning (Trinh et al., ; Cherian et al., ), or even making scientific discoveries (Romera-Paredes et al., ).

Figure: Figure 1: Frames from an example dynamical sequence in our TraySim dataset. The left-most frame shows the first frame of the scene with many objects on the tray and is going to be impacted by a black pusher (right-bottom). The subsequent frames show the state of the system at the 25-th, 50-th, and the 200-th time step (each step is 0.01s). Our task is for a model to reason through the dynamics of the system and predict the stability of each object on the tray at the end of the episode in a zero-shot manner, when provided as input only the *first frame* of the sequence.

While current LLMs seem to possess the knowledge of the physical world and may be able to provide a plan for solving a physical reasoning task (Singh et al., ; Kim et al., ) when crafted in a suitable multimodal format (prompt), their inability to interact with the real world or measure unobservable attributes of the world model, hinders their capability in solving complex physical reasoning problems (Wang et al., ; Bakhtin et al., ; Riochet et al., ; Harter et al., ; Xue et al., ). Consider for example the scene in Figure 1. Suppose a reasoning model is provided as input only the first image (left-most) and is asked to answer the question: *Which of the objects will remain upright when the tray is impacted by the black pusher with a velocity of 4.8 m/s?*. To answer this question, the model must know the various physical attributes of the system, including the masses of the objects, friction coefficients between the contacts and their respective contact forces, stiffness of the objects, among others. While, a large sophisticated machine learning model (such as an LLM) may be able to provide an educated guess based on the intuitive physics of the system, a useful solution would demand a more intricate reasoning strategy in estimating the real-world physics and dynamics; such complex dynamics may be difficult or even impossible to be correctly learned solely from training data. Conversely, advancements in graphics hardware and software have led to the development of advanced physics engines capable of simulating realistic world models Makoviychuk et al. (); Bear et al. (); Todorov et al. (); Gan et al. (). Thus, rather than having the LLM to learn the world physics, our key idea is to consider using a physics engine in tandem with the LLM, where the LLM may use its world knowledge for generating scene-based reasoning hypotheses while the simulator is used to verify them within the physical world model.

To study this problem, we consider the novel task of predicting the dynamics of objects and their stability under the influence of an impact – an important problem for a variety of robotic applications (Gasparetto et al., ; Ahmed et al., ). In this paper, we consider this problem in a challenging setting using our new dataset, *TraySim*, in which the impact is caused by a pusher colliding to a tray that holds several objects of varied sizes, masses, and centers of gravity, with the goal of predicting the dynamics of each of the object instances. We cast this task as that of answering physical reasoning questions. Specifically, as illustrated in Figure 1, TraySim includes simulated video sequences consisting of a tray with an arbitrary number of objects on it and given the first video frame of a given scene, the task of the reasoning model is to infer which of the objects on the tray will remain upright after the impact when the system has stabilized. As is clear from Figure 1, solving this task will require the model to derive details regarding the physical properties of each of the objects and their contacts, as well as have the ability to imagine the system’s dynamics through multi-body interactions influenced by the various internal and external forces from the impact. Our task presents a challenging reasoning setup for current machine learning models, including LLMs.

To solve this task, we propose $\operatorname{LLMPhy}$, a black-box optimization setup combining an LLM with a physics engine that leverages the program synthesis abilities of the LLM to communicate with the engine for solving our task. $\operatorname{LLMPhy}$ operates in two phases: i) a parameter estimation phase, where $\operatorname{LLMPhy}$ is used as a continuous black-box optimization module towards inferring the physical parameters of the objects, including the friction, stiffness, damping, etc. from a given example video sequence, and ii) a scene understanding phase, where the LLM-simulator combination is used as a discrete black-box optimizer to reconstruct the problem layout for synthesizing the setup within the simulator for execution. Our framework builds a feedback loop between the LLM and the physics engine, where the LLM generates programs using its estimates of physical attributes; the programs are executed in the simulator, and the error from the simulations are fed back to the LLM as prompts to refine its estimates until a suitable convergence criteria is met. Note that we do not assume any differentiablity properties of the simulator, which makes our setup highly general. This allows the approach to function as a black-box optimization framework, enabling its use with a wide range of simulators without the need for gradient-based methods.

While we may generate unlimited data using our simulation program, given the zero-shot nature of our setup, we synthesized 100 sequences in our *TraySim* dataset to demonstrate the effectiveness of $\operatorname{LLMPhy}$. Each sample in TraySim has two video sequences: i) the task sequence of which only the first frame is given to a reasoning agent, and ii) a parameter-estimation video sequence which has a lesser number of instances of each of the object types appearing in the task sequence; the latter sequence has an entirely different layout and dynamics of objects. To objectively evaluate the performance, we cast the task as a physical question answering problem, where the LLM is required to select the correct subset of answers from the given candidate answers. Our results on TraySim show that $\operatorname{LLMPhy}$ leads to promising improvements in performance against alternatives, including using Bayesian optimization, CMA-ES, and solely using an LLM for physical reasoning. Interestingly, we also find that $\operatorname{LLMPhy}$ estimates the physical parameters better and powerful recent LLMs – such as OpenAI o1-preview – show trends of superior optimization convergence.

Before moving forward, we summarize below our main contributions:

- •
We consider the novel task of reasoning over complex physics of a highly dynamical system by combining LLMs with non-differentiable physics engines.
- •
We propose a zero-shot reasoning framework $\operatorname{LLMPhy}$, that combines the reasoning and program synthesis abilities of an LLM with the realistic simulation abilities of a physics engine. This approach is used to estimate the physical parameters of the model, the scene layout, and synthesizing the dynamical scene for inferring the solution.
- •
We introduce a novel synthetic multi-view dataset: TraySim, to study this task. The dataset consists of 100 scenes for zero-shot evaluation.
- •
Our experiments using $\operatorname{LLMPhy}$ on the TraySim dataset demonstrate promising results against related baselines, highlighting its potential for tackling complex physics-based reasoning tasks that involves both discrete and continuous optimization sub-tasks.

## 2 Related Works

Large language models (LLMs) demonstrate remarkable reasoning skills across a variety of domains, highlighting their versatility and adaptability. They have shown proficiency in managing complex conversations (Glaese et al., ; Thoppilan et al., ), engaging in methodical reasoning processes (Wei et al., ; Kojima et al., ), planning (Huang et al., ), tackling mathematical challenges (Lewkowycz et al., ; Polu et al., ), and even generating code to solve problems (Chen et al., ). As we start to incorporate LLMs into physically embodied systems, it’s crucial to thoroughly assess their ability for physical reasoning. However, there has been limited investigation into the physical reasoning capabilities of LLMs.

In the field of language-based physical reasoning, previous research has mainly concentrated on grasping physical concepts and the attributes of different objects. (Zellers et al., ) introduced grounded commonsense inference, merging natural language inference with commonsense reasoning. Meanwhile, (Bisk et al., ) developed the task of physical commonsense reasoning and a corresponding benchmark dataset, discovering that pretrained models often lack an understanding of fundamental physical properties. (Aroca-Ouellette et al., ) introduced a probing dataset that evaluates physical reasoning through multiple-choice questions. This dataset tests both causal and masked language models in a zero-shot context. However, many leading pretrained models struggle with reasoning about physical interactions, particularly when answer choices are reordered or questions are rephrased. (Tian et al., ) explored creative problem-solving capabilities of modern LLMs in constrained setting. They automatically a generate dataset consisting of real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. (Wang et al., ) presented a benchmark designed to assess the physics reasoning skills of large language models (LLMs). It features a range of object-attribute pairs and questions aimed at evaluating the physical reasoning capabilities of various mainstream language models across foundational, explicit, and implicit reasoning tasks. The results indicate that while models like GPT-4 demonstrate strong reasoning abilities in scenario-based tasks, they are less consistent in object-attribute reasoning compared to human performance.

In addition to harnessing LLMs for physical reasoning, recent works have used LLMs for optimization. The main focus has been on targeted optimization for employing LLMs to produce prompts that improves performance of another LLM. (Yang et al., ) shows that LLMs are able to find good-quality solutions simply through prompting on small-scale optimization problems. They demonstrate the ability of LLMs to optimize prompts where the goal is to find a prompt that maximizes the task accuracy. The applicability of various optimization methods depends on whether the directional feedback information is available. In cases when the directional feedback is available, one can choose efficient gradient-based optimization methods (Sun et al., ). However, in scenarios without directional feedback, black-box optimization methods (Terayama et al., ) are useful such as Bayesian optimization (Mockus, ), Multi-Objective BO (Konakovic Lukovic et al., ) and CMA-ES (Hansen & Ostermeier, ). Only a limited number of studies have explored the potential of LLMs for general optimization problems. (Guo et al., ) shows that LLMs gradually produce new solutions for optimizing an objective function, with their pretrained knowledge significantly influencing their optimization abilities. (Nie et al., ) study factors that make an optimization process challenging in navigating a complex loss function. They conclude that LLM-based optimizer’s performance varies with the type of information the feedback carries, and given proper feedback, LLMs can strategically improve over past outputs. In contrast to these prior works, our goal in this work is to combine an LLM with a physics engine for physics based optimization.

Our work is inspired by the early work on neural de-rendering (Wu et al., ) that re-simulates a scene using a physics engine or synthesizing realistic scenes for physical understanding Bear et al. (). A recent prior work that has a similar motivation to ours is (Liu et al., ) that uses a language model to use the given question to produce a program for a simulator, these simulation results are appended to the input prompts of language model for answer generation during inference. Their method can serve as a plug-and-play framework that works with any language model and requires neither handcrafted prompts nor costly fine-tuning. More recently, in SimLM (Memery et al., ), a framework predicting the parameters of projectile motion by an LLM using a physics simulator. However, these works uses relatively simplistic physical settings and does not involve an LLM-simulator loop for optimization, which is one of our key contributions. We also note that there have been physically-plausible video synthesis approaches (Ehrhardt et al., ; Liu et al., ), however our focus is on black-box optimization using an LLM as a program synthesizer in a symbolic (program) space for solving reasoning tasks than pixel-wise generation of video frames.

## 3 Proposed Method

The purpose of this work is to enable LLMs to perform physics-based reasoning in a zero-shot manner. Although LLMs may possess knowledge of physical principles that are learned from their training data, state-of-the-art models struggle to effectively apply this knowledge when solving specific problems. This limitation, we believe, is due to the inability of the model to interact with the scene to estimate its physical parameters, which are essential and needs to be used in the physics models for reasoning, apart from the stochastic attributes implicit in any such system. While, an LLM may be trained to implicitly model the physics given a visual scene – e.g., generative models such as SoRA(^1^11https://openai.com/index/sora/), Emu-video Girdhar et al. (), etc., may be considered as world model simulators – training such models for given scenes may demand exorbitant training data and compute cycles. Instead, in this paper, we seek an alternative approach by leveraging the recent advancements in realistic physics simulation engines, and use such simulators as a tool accessible to the LLM for solving its given physical reasoning task. Specifically, we attempt to solve the reasoning task as that of equipping the LLM to model and solve the problem using the simulator, and for which we leverage on the LLM’s code generation ability as a bridge. In the following sections, we exposit the technical details involved in achieving this LLM-physics engine synergy.

### 3.1 Problem Setup

Suppose $\mathbf{X}^{v}=\langle{\mathbf{x}^{v}_{1},\mathbf{x}^{v}_{2},\cdots,\mathbf{x}^{v}_{T}}\rangle$ denote a video sequence with $T$ frames capturing the dynamics of a system from a camera viewpoint $v$. We will omit the superscript $v$ when referring to all the views jointly. In our setup, we assume the scene consists of a circular disk (let us call it a *tray*) of a given radius, friction, and mass. Further, let $\mathcal{C}$ denote a set of object types, e.g., in Figure 1, there are three types of objects: a *bottle*, a *martini glass*, and a *wine glass*. The tray is assumed to hold a maximum of $K$ object instances, the $k$-th instance is denoted $o_{k}$; $K$ being a perfect square. To simplify our setup, we assume that the instances on the tray are arranged on a $\sqrt{K}\times\sqrt{K}$ regular grid. If the number of instances used in a task example is less than $K$, then some of the positions on this grid will be left empty. We further assume that the masses of the objects in $\mathcal{C}$ are given during inference, while other physical attributes, denoted as $\Phi_{c}$ for all objects $c\in\mathcal{C}$, are *unknown* and identical for objects of the same type. In line with the standard Mass-Spring-Damping (MSD) dynamical system, we consider the following set of contact physics parameters $\Phi_{c}\in\mathbb{R}^{4}$ for each object class: i) coefficient of sliding friction, ii) stiffness, iii) damping, and iv) the rotational inertia (also called armature). To be clear, we do not assume or use any physics model in our optimization pipeline, and our setup is entirely black-box, but the selection of these optimization physics parameters is inspired by the MSD model. We assume the objects do not have any rotational or spinning friction. While the instances $o_{k}$ of the same type are assumed to share the same physics parameters, they differ in their visual attributes such as color or shape. The tray is impacted by a pusher $p$ that starts at a fixed location and is given an initial velocity of $p_{s}$ towards the tray. The pusher is assumed to have a fixed mass and known physical attributes, and the direction of impact is assumed to coincide with the center of the circular tray.

Figure: Figure 2: Illustration of the key components of $\operatorname{LLMPhy}$ and the control flow between LLM, physics simulator, and the varied input modalities and examples.
Refer to caption: x5.png

### 3.2 Problem Formulation

With the notation above, we are now ready to formally state our problem. In our setup, we define an input task instance as: $\mathcal{T}=(\{{\mathbf{x}^{v}_{g}}\}_{v\in{|\mathcal{V}|}},p_{s},Q,\mathcal{O},\mathcal{I},\mathbf{X}_{\mathcal{T}},\mathcal{C}_{\mathcal{T}})$, where $\mathbf{x}_{g}$ is the first frame of a video sequence $\mathbf{X}$ with $\mathcal{V}$ views, $p_{s}$ is the initial velocity of the pusher $p$, $Q$ is a question text describing the task, and $\mathcal{O}$ is a set of answer candidates for the question. The goal of our reasoning agent is to select the correct answer set $\mathcal{A}\subset\mathcal{O}$. The notation $\mathcal{C}_{\mathcal{T}}\subseteq\mathcal{C}$ denotes the subset of object classes that are used in the given task example $\mathcal{T}$. In this paper, we assume the question is the same for all task examples, i.e., *which of the object instances on the tray will remain steady when impacted by the pusher with a velocity of $p_{s}$?* We also assume to have been given a few in-context examples $\mathcal{I}$ that familiarises the LLM on the structure of the programs it should generate. We found that such examples embedded in the prompt are essential for the LLM to restrict its generative skills to the problem at hand, while we emphasize that the knowledge of these in-context examples will not by themselves help the LLM to correctly solve a given test example.

As it is physically unrealistic to solve the above setup using only a single image (or multiple views of the same time-step), especially when different task examples have distinct dynamical physics parameters $\Phi$ for $\mathcal{C}_{\mathcal{T}}$, we also assume to have access to an additional video sequence $\mathbf{X}_{\mathcal{T}}$ associated with the given task example $\mathcal{T}$ containing the same set of objects as in $\mathbf{x}_{g}$ but in a different layout and potentially containing a smaller number of object instances. The purpose of having $\mathbf{X}_{\mathcal{T}}$ is to estimate the physics parameters of the objects in $\mathbf{x}_{g}$, so that these parameters can then be used to conduct physical reasoning for solving $\mathcal{T}$. Note that this setup closely mirrors how humans would solve such a reasoning task. Indeed, humans may pick up and interact with some object instances in the scene to understand their physical properties, before applying sophisticated reasoning on a complex setup. Without any loss of generality, we assume the pusher velocity in $\mathbf{X}_{\mathcal{T}}$ is fixed across all such auxiliary sequences and is different from $p_{s}$, which varies across examples.

Figure: Figure 3: Full architecture of the two phases in $\operatorname{LLMPhy}$. In Phase 1, we use the auxiliary video sequences from which the trajectories of motion of the objects on the tray are extracted and $\operatorname{LLMPhy}$ uses continuous black-box optimization using program synthesis and simulation for implicitly estimating the physical parameters of these objects to reproduce their trajectories. In Phase 2, we use $\operatorname{LLMPhy}$ as a discrete optimizer to reproduce the scene layout where the layout estimation error is used as feedback to improve the synthesized $\operatorname{LLMPhy}$ program. Finally, the layout and the physics parameters are used to synthesize the full scene to find the outcome of the pusher impact using the problem specific parameters, e.g., pusher velocity and location (not shown in the figure).
Refer to caption: x6.png

### 3.3 Combining LLMs and Physics Engines for Physical Reasoning

In this section, our proposed $\operatorname{LLMPhy}$ method for our solving physical reasoning task is outlined. Figure 2 illustrates our setup. Since LLMs on their own may be incapable of performing physical reasoning over a given task example, we propose combining the LLM with a physics engine. The physics engine provides the constraints of the world model and evaluates the feasibility of the reasoning hypothesis generated by the LLMs. This setup provides feedback to the LLM that enables it reflect on and improve its reasoning. Effectively solving our proposed task demands inferring two key entities: i) the physical parameters of the setup, and ii) layout of the task scene for simulation using physics to solve the task. We solve for each of these sub-tasks in two distinct phases as detailed below. Figure 3 illustrates our detailed architecture, depicting the two phases and their interactions.

#### 3.3.1 LLMPhy \operatorname{LLMPhy} Phase 1: Inferring Physical Parameters

As described above, given the task example $\mathcal{T}$, $\operatorname{LLMPhy}$ uses the task auxiliary video $\mathbf{X}_{\mathcal{T}}$ to infer the physical attributes $\Phi$ of the object classes in $\mathcal{C}$. Note that these physical attributes are specific to each task example. Suppose $\tau:\mathcal{X}\to\mathbb{R}^{3\times T\times|\mathcal{C}_{\mathcal{T}}|}$ be a function that extracts the 3D physical trajectories of length $T$ of each of the objects in the given video $\mathbf{X}_{\mathcal{T}}\in\mathcal{X}$, where $\mathcal{X}$ denotes the set of all videos. Recall that $\mathcal{C}_{\mathcal{T}}$ denotes the subset of object types that appear in the given task example.

Suppose $\operatorname{LLM}_{1}$ denotes the LLM used in Phase 1, which takes as input the in-context examples $\mathcal{I}_{1}\subset\mathcal{I}$ and the object trajectories from $\mathbf{X}_{\mathcal{T}}$, and is tasked to produce a program $\pi(\Phi)\in\Pi$, where $\Pi$ denotes the set of all programs. Further, let $\operatorname{SIM}:\Pi\to\mathbb{R}^{3\times T\times|\mathcal{C}_{\mathcal{T}}|}$ be a physics-based simulator that takes as input a program $\pi(\Phi)\in\Pi$, executes it, and produce the trajectories of objects described by the program using the physics attributes. Then, the objective of Phase 1 of $\operatorname{LLMPhy}$ can be described as:

$$ $\Phi^{*}=\arg\min_{\Phi}\|{\operatorname{LLMPhy}_{1}(\pi(\Phi)\ |\ \tau(\mathbf{X}_{\mathcal{T}}),\mathcal{I}_{1})-\tau(\mathbf{X}_{\mathcal{T}})}\|^{2},$ (1) $$

where $\operatorname{LLMPhy}_{1}=\operatorname{SIM}\circ\operatorname{LLM}_{1}$ is the composition of the simulator and the LLM through the generated program, with the goal of estimating the correct physical attributes of the system $\Phi$. Note that the notation $\pi(\Phi)$ means the generated program takes as argument the physics parameters $\Phi$ which is what we desire to optimize using the LLM.

#### 3.3.2 LLMPhy \operatorname{LLMPhy} Phase 2: Simulating Task Example

The second phase of $\operatorname{LLMPhy}$ involves applying the inferred physical parameters $\Phi$ for the object classes in $\mathcal{C}$ to solve the task problem described in $\mathbf{x}_{g}$, i.e., the original multi-view task images (see Figure 3). This involves two steps: i) understanding the scene layout (i.e., where the various object instances are located on the tray, their classes, and attributes (e.g., color); this is important as we assume that different type of objects have distinct physical attributes, and ii) using the physical attributes and the object layout to produce code that can be executed in the physics engine to simulate the full dynamics of the system to infer the outcome; i.e., our idea is to use the simulator to synthesize a dynamical task video from the given input task images, and use the ending frames of this synthesized video to infer the outcome (see Figure 2).

Suppose $\operatorname{LLM}_{2}$ denotes the LLM used in Phase 2, which takes as input the multi-view task images $\mathbf{x}_{g}$, the physical attributes $\Phi^{*}$, and Phase 2 in-context examples $\mathcal{I}_{2}\subset\mathcal{I}$ to produce a program $\pi(\Psi)\in\Pi$ that reproduces the scene layout parameters, i.e., the triplet $\Psi=\{{(\text{class},\text{location},\text{color})}\}_{k}$ for each instance. The objective for estimating the layout parameters $\Psi$ can be written as:

$$ $\Psi^{*}=\arg\min_{\Psi}\|{\operatorname{LLMPhy}_{2}(\pi(\Psi)\ |\ \mathbf{x}_{g},\mathcal{I}_{2})-\mathbf{x}_{g}}\|^{2},$ (2) $$

where $\operatorname{LLMPhy}_{2}=\operatorname{SIM}\circ\operatorname{LLM}_{2}$. Once the correct layout parameters $\Psi^{*}$ are estimated, we can produce a video sequence $\hat{\mathbf{X}}\ |\ \Psi^{*},\Phi^{*}$ using the simulator, and which can then be used for solving the problem by selecting an answer subset $\mathcal{A}$ from the answer options $\mathcal{O}$. We may use an LLM or extract the state of the instances within the simulator to solve the question-answering task.

### 3.4 Optimizing LLM-Simulator Combination

In Alg. 1, we detail the steps for optimizing $\operatorname{LLMPhy}$. Given that we assume the simulator might be non-differentiable, we frame this as a black-box optimization problem. Here, the optimization variables are sampled based on the inductive bias and the knowledge of physics learned by the LLM from its large corpora of training data. The LLM generates samples over multiple trials, which are then validated using the simulator. The resulting error is used to refine the LLM’s hyperparameter search. A key insight of our approach is that, since the hyperparameters in our setup have physical interpretations in the real world, a knowledgeable LLM should be capable of selecting them appropriately by considering the error produced by its previous choices. In order for the LLM to know the history of its previous choices and the corresponding error produced, we augment the LLM prompt with this optimization trace from the simulator at each step.

Figure: Algorithm 1 Pseudo-code describing the key steps in optimizing $\operatorname{LLMPhy}$ for Phases 1 and 2.

## 4 Experiments and Results

In this section, we detail our simulation setup used to build our TraySim dataset, followed by details of other parts of our framework, before presenting our results.

### 4.1 Simulation Setup

As discussed in the previous section, we are determining the physical characteristics of our simulation using a physics engine. MuJoCo  Todorov et al. () was used to setup the simulation and compute the rigid body interactions within the scene. It is important to note that any physics engine capable of computing the forward dynamics of a multi-body system can be integrated within our framework. This is because $\operatorname{LLMPhy}$ implicitly estimates the outcome of a scene based on the specific physical laws the engine is computing. To be clear, $\operatorname{LLMPhy}$ does not assume any physical model of the world and operates entirely as a black-box optimizer. The world model is entirely captured by the physics engine that executes the program $\operatorname{LLMPhy}$ produces.

The simulation environment is build upon a template of the World, $\mathcal{W}$, which contains the initial parametrization of our model of Newtonian physics. This includes the gravity vector g, time step, and contact formulation, but also graphical and rendering parameters later invoked by the $\operatorname{LLM}$ when executing the synthesized program. MuJoCo uses internally a soft contact model to compute for instance complementarity constraints; in our implementation we use a non-linear sigmoid function that allows a very small inter-body penetration and increases the simulation stability during abrupt accelerations. We use elliptic friction cones to replicate natural contacts more closely. We further take advantage of the model architecture of MuJoCo by programatically inserting arbitrary objects $o_{k}$ from the classes in $\mathcal{C}$ into the scene, as described in Section 3.1. For each parametric object class in $\mathcal{C}$, we generate an arbitrary appearance and physical attributes such as static friction, stiffness, damping, and armature. An arbitrary number of object instances are created from each class (up to a provided limit on their total number) and placed at randomly chosen positions on a regular grid (scene layout). The graphical renderer is used to record the frame sequences $\mathbf{X}$ corresponding to five orthogonally placed cameras around the World origin, including a top-down camera. In addition, we support panoptic segmentation of all objects in the scene and store the corresponding masks for arbitrarily chosen keyframes. The simulated data also contains privileged information such the pusher-tray contact information (i.e. force, location, velocity, time stamp), and the stability information for each object, $\mathcal{S}_{k}=\{1|\arccos(\textbf{g},O\textbf{z}_{k})<\alpha,0|otherwise\}$, where $\mathbf{g}$ is the gravity vector, $O\textbf{z}_{k}$ is the upright direction of object $k$ and $\alpha$ is an arbitrarily chosen allowable tilt. In our experiments, we use $\alpha=45^{\circ}$.

Other than the physics parametrization of each object class $\mathcal{C}$ and the scene layout $\cup o_{k}$, the outcome of the simulation for sequence X is given by the initial conditions of the pusher object $p$, namely its initial velocity $\dot{\textbf{p}}_{s}$ and position $\textbf{p}_{s}$. The usual torque representation is used:

$$ $\boldsymbol{\tau}=\textbf{I}_{C}\boldsymbol{\dot{\omega}}+\boldsymbol{\omega}\times\textbf{I}_{C}\boldsymbol{\omega},$ (3) $$

which relates the angular acceleration $\mathbf{\alpha}$ and angular velocity $\mathbf{\dot{\omega}}$ to the objects torque $\mathbf{\tau}$. The simulator computes in the end the motion of each object based on the contact dynamics model given by:

$$ $\mathbf{M}(\mathbf{q})\ddot{\mathbf{q}}+\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})=\mathbf{S}_{a}^{T}\boldsymbol{\tau}+\mathbf{S}_{u}^{T}\boldsymbol{\lambda}_{u}+\mathbf{J}^{T}_{c}(\mathbf{q})\boldsymbol{\lambda}_{c},$ (4) $$

where $\mathbf{M}(\mathbf{q})\in\mathbb{R}^{(n_{a}+n_{u})\times(n_{a}+n_{u})}$ is the mass matrix; $\mathbf{q}\triangleq[\mathbf{q}_{a}^{T},\mathbf{q}_{u}^{T}]^{T}\in\mathbb{R}^{n_{a}+n_{u}}$ are generalized coordinates; and $\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\in\mathbb{R}^{n_{a}+n_{u}}$ represents the gravitational, centrifugal, and the Coriolis term. The selector matrices $\mathbf{S}_{a}=[\mathbb{I}_{n_{a}\times n_{a}}\ \mathbf{0}_{n_{a}\times n_{u}}]$ and $\mathbf{S}_{u}=[\mathbf{0}_{n_{u}\times n_{a}}\ \mathbb{I}_{n_{u}\times n_{u}}]$ select the vector of generalized joint forces $\boldsymbol{\tau}\in\mathbb{R}^{n_{a}}$ for the actuated joints $n_{a}$, or $\boldsymbol{\lambda_{u}}\in\mathbb{R}^{n_{u}}$ which are the generalized contact forces of the unactuated DOF created by the dynamics model, respectively. $\mathbf{J}_{c}(\mathbf{q})\in\mathbb{R}^{6n_{c}\times(n_{a}+n_{u})}$ is the Jacobian matrix and $\boldsymbol{\lambda}_{c}\in\mathbb{R}^{6n_{c}}$ are the generalized contact forces at $n_{c}$ contact points. In our simulated environment, only the pusher object $p$ has actuated joints which sets its initial velocity and heading, while the rest of the joints are either unactuated or created by contacts. The state of the system is represented by $\mathbf{s}\triangleq[\mathbf{q}^{T}\ \mathbf{\dot{q}}^{T}]^{T}$.

### 4.2 TraySim Dataset

Using the above setup, we created 100 task sequences using object classes $\mathcal{C}=\{{\text{wine glass},\text{martini glass},\text{bottle}}\}$ with object instances from these classes arranged roughly in a $3\times 3$ matrix on the tray. The instance classes and the number of instances are randomly chosen with a minimum of 5 and a maximum of 9. Each task sequence is associated with an auxiliary sequence for parameter estimation that contains atleast one object instance from every class of object appearing in the task images. For example, if a task image (that is, the first image in a task sequence) has 3 bottles, then we will have a bottle in the auxiliary sequence. We assume each instance is defined by a triplet: (color, type, location), where the color is unique across all the instances on the tray so that it can be identified across the multi-view images, especially when some views occlude some of the instances. The physical parameters of the objects are assumed to be the same for both the task sequences and the auxiliary sequences, and instances of the same object classes have the same physical parameters.

When generating each problem instance in the TraySim dataset, the physical parameters of the object classes are randomly chosen within the following ranges: sliding friction in (0.1, 1], inertia and stiffness in (0, 1), and damping in (0, 10). We assume a fixed and known mass for each object type across problem instances, namely we assume a mass of 20 units for bottle, 10 units for martini glass, and 4 units for the wine glass. The tray used a mass of 0.5 and the pusher with a mass 20. Further, for both the task and the auxiliary sequences we assume the pusher is located at the same initial location in the scene. However, for all the auxiliary sequences, we assume the pusher moves with an initial (x, y) velocity of (-4.8, -4.8) m/s towards the tray, while for the task sequences, this velocity could be arbitrary (but given in the problem question), with each component of velocity in the range of [-7, -3] m/s. We further assume that the pusher impact direction coincides with the center of the circular tray in all problem instances.

Each sequence was rendered using the simulator for 200 time steps, each step has a duration of 0.01s. We used the last video frame from the task sequence to check the stability of each instance. Specifically, if the major axis of an object instance in the last frame of a task sequence makes an angle of more than 45 degrees with the ground plane, then we deem that instance as *stable*. We randomly select five object instances and create a multiple choice candidate answer set for the question-answering task, where the ground truth answer is the subset of the candidates that are deemed upright in the last frame.

### 4.3 Large Language Model

We use the OpenAI o1-mini (^2^22https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/) text-based LLM for our Phase 1 experiments and GPT-4o(^3^33https://openai.com/index/hello-gpt-4o/) vision-and-language model in Phase 2. Note that our Phase 1 is an entirely text-based LLM as we pre-extract the object trajectories. At the time of this writing, OpenAI o1-mini model does not support visual inputs, and thus we use the GPT-4o model in Phase 2.

Phase 1: In this phase, we provide as input to the LLM four items: i) a prompt describing the problem setup, the qualitative parameters of the objects (such as mass, height, size of tray, etc.) and the task description, ii) in-context examples consisting of sample trajectories of the object instances from its example auxiliary sequence, iii) an in-context program example that, for the given example auxiliary sequence trajectories, shows their physical parameters and the output structure, and iv) auxiliary task sequence trajectories (from the sequence for which the physical parameters have to be estimated) and a prompt describing what the LLM should do. The in-context example is meant to guide the LLM to understand the setup we have, the program structure we expect the LLM to synthesize, and our specific APIs that need to be called from the synthesized program to reconstruct the scene in our simulator. Please see our Appendix for an example of the full prompt that we use. When iterating over the LLM predictions, we augment the above prompt with the history of all the estimations of the physical parameters that the LLM produced in the previous iterations (extracted from the then generated code) and the $\ell_{2}$ norm between the generated and ground truth object trajectories for each object instance in the auxiliary sequence, with an additional prompt to the LLM as follows: “We ran your code in our simulator using the physical parameters you provided below… The error in the prediction of the trajectories using these physical parameters is given below. Can you refine your code to make the trajectories look more similar to the ones in given in …? Your written code should strictly follow the same code structure as provided in …”.

Phase 2: The goal of the LLM in Phase 2 is to predict the object instance triplet from the multi-view task images. Towards this end, the LLM generates code that incorporates these triplets, so that when this code is executed, the simulator will reproduce the scene layout. Similar to Phase 1, we provide to the LLM an in-context example for guiding its code generation, where this in-context example contains multi-view images and the respective program, with the goal that the LLM learns the relation between parts of the code and the respective multi-view images, and use this knowledge to write code to synthesize the layout of the provided task images. When iterating over the optimization steps, we compute an error feedback to the LLM to improve its previously generated code, where the feedback consists of the following items: i) the program that the LLM synthesized in the previous optimization step, ii) the PSNR between the task image and the simulated image (top-down views), and iii) the color of the object instances in error(^4^44This is done by inputting a difference image (between the task and synthesized images) to another vision-and-language LLM that is prompted to identify the triplets that are in error). Using this feedback, the Phase 2 LLM is prompted to fix the code associated with the triplets in error. Our feedback prompt in Phase 2 thus looks like in the following example: “The chat history below shows a previous attempt of GPT-4o in generating Python code to reproduce the task images …. For each attempt, we ran the GPT-4o generated code in our simulator and found mistakes. Below we provide the code GPT produced, as well as the PSNR of the generated image against the given top-down image. Can you refine your code to reproduce the task images correctly? You should not change any part of the code corresponding to correctly inferred objects. $\langle$ code …$\rangle$. Colors of the objects in the code above that are misplaced: colors = {’orange’, ’purple’, ’cyan’}. PSNR for the generated image against given top-down image = 39.2
Please check the locations of these objects in task_image_top_view_1.png and fix the code accordingly.”.

LLMPhy Settings: We compute the trajectory reconstruction error in Phase 1 where the synthesized program from the LLM containing the estimated physics parameters is executed in the simulator to produce the motion trajectory of the center of gravity of the instances. We sample the trajectory for every 10 steps and compute the L2 norm between the input and reconstructed trajectories. We use a maximum of 30 LLM-simulator iterations in Phase 1 and use the best reconstruction error to extract the parameters. For Phase 2, we use the PSNR in the reconstruction of the first frame by the simulator using the instance triplets predicted by the LLM in the generated program. We used a maximum of 5 $\operatorname{LLMPhy}$ iterations for this phase. As the LLM queries are expensive, we stopped the iterations when the trajectory prediction error is below 0.1 on average for Phase 1 and when the PSNR is more than 45 dB for Phase 2.

In Figure 4, we illustrate the experimental setup using an example from the TraySim dataset. Specifically, the inputs for the two phases are depicted, where Phase 2 consists of multiview images from the first time step, while the corresponding Phase 1 input consists of an object instance from each object class used in Phase 2, however have multiview sequences from which the motion trajectories of each instance is extracted using the simulator. In Figure 4, we omit the task question, which is the same text for all problems in our dataset except for the pusher velocity, which differs between problem instances. We also show the answer options, of which more than one option may be correct. The right-most sub-figures show the ground truth outcome of the pusher impact.

How sensitive are the physics parameters to influence the outcome? In Figure 5, we show three Phase 1 sequences consisting of the same objects and their layout, however varying the physics attributes as shown in the histogram plots. The pusher velocity is fixed for all the sequences. As can be seen from the figure, varying the parameters result in entirely different stability for the objects after the impact, substantiating that the correct inference of these parameters is important to reproduce the correct the outcome.

Evaluation Metric and Baselines: We consider various types of evaluations in our setup. Specifically, we use the intersection-over-union as our key performance metric that computes the overlap between the sets of $\operatorname{LLMPhy}$ produced answers in Phase 2 with the ground truth answer set. We also report the performances for correctly localizing the instances on the tray, which is essential for simulating the correct scene. As ours is a new task and there are no previous approaches that use the composition of LLM and physics engine, we compare our method to approaches that are standard benchmarks for continuous black-box optimization, namely using Bayesian optimization Mockus () and Covariance matrix adaptation evolution strategy (CMA-ES) Hansen & Ostermeier (); Hansen ().

Figure: Figure 4: Illustration of our problem setup, specifically the inputs in Phase 1 and Phase 2, the question parameters, options, and the ground truth outcomes.
Refer to caption: x7.png

Figure: Figure 5: Illustration of the changes in the physical parameters (left histogram, sliding friction, rotation inertia, stiffness, and damping, respectively), and the result of the impact on three objects placed at the same location on the tray (Frame 1) and being impacted by the same force from the pusher. The examples are from the Phase 1 of our dataset. As is clear in the last frame (Frame 200) that changes in the the physical parameters results in entirely different outcomes, substantiating that the estimations of these parameters is important in solving our task.
Refer to caption: x8.png

Figure: Figure 6: An example first video frame from a Phase 1 sequence (left), the fitted trajectories along the center-of-gravity of motion of the three objects by $\operatorname{LLMPhy}$ by estimating the physics parameters (second), the trajectory prediction errors for $\operatorname{LLMPhy}$ optimization (third) and that for Bayesian Optimization (fourth); x-axis is the optimization step and y-axis shows the error.

Figure: Figure 7: We show an example sequence from Phase 2 for the Phase 1 scene in Figure 6. The first two images show two views that are used as input to our model (i.e., side-view (left) and top-view (second)). In Phase 2, $\operatorname{LLMPhy}$ generates a program estimating the object layout, class, and color, and executing this code in the simulator produces the third image using these predicted attributes. The right image shows the estimation error which is used to produce the error signal for improving model prediction.

### 4.4 Experimental Results

#### 4.4.1 Comparison with Prior Methods

In Table 1, we compare the performance of Phase 1 and Phase 2 of $\operatorname{LLMPhy}$ to various alternatives and prior black-box optimization methods. Specifically, we see that random choice (Experiment #1) for the two Phases – that is, randomly choosing the physics parameters from the respective range in Phase 1, and random location, type, and objects in the synthesized program in Phase 2 – lead to only 20% accuracy. Next, if we use the Phase 2 multiview images (no Phase 1 or any sequence level information) and directly ask the LLM (GPT-4o in this case) to predict the outcome of the interaction (using the ground truth physics parameters provided), this leads to 32% accuracy as depicted in Phase 2, suggesting the LLM may be able to provide an educated guess based on the provided task images. To clarify, in this experiment, we are measuring the ability of the LLM to directly infer the outcome of the impact by imagining the motion and the impact from the given multiview images (and not sequences). This result is shown in Experiment #2 in the table. In our next Experiment 3, we use $\operatorname{LLMPhy}$ for Phase 2, however use random sampling for the physics parameters. We see that this leads to some improvement in performance, given we are using the simulator to synthesize the dynamical scene. Although the performance is lower than ideal because the physics parameters are random and as noted from Figure 5, we know that the outcome is strongly dependent on the physics parameters.

**Table 1: Performance analysis of $\operatorname{LLMPhy}$ Phase 1 and Phase 2 combinations against various alternatives, including related prior methods. We report the intersection-over-union of the predicted answer options and the ground truth answers in the multiple choice solutions.**
| Expt # | Phase 1 | Phase 2 | Avg. IoU (%) |
| --- | --- | --- | --- |
| 1 | Random | Random | 19.0 |
| 2 | N/A | LLM | 32.1 |
| 3 | Random | $\operatorname{LLMPhy}$ | 50.8 |
| 4 | BO (30 iterations) | $\operatorname{LLMPhy}$ | 59.6 |
| 5 | CMA-ES (30 iterations) | $\operatorname{LLMPhy}$ | 59.7 |
| 6 | $\operatorname{LLMPhy}$ (30 iterations) | $\operatorname{LLMPhy}$ | 62.0 |
| 7 | BO (100 iterations) | $\operatorname{LLMPhy}$ | 61.0 |
| 8 | CMA-ES (100 iterations) | $\operatorname{LLMPhy}$ | 60.7 |
| 9 | Ground truth | $\operatorname{LLMPhy}$ | 65.1 |

**Table 2: Experiments presenting the accuracy of generated code compared to the ground truth in Phase 2 of $\operatorname{LLMPhy}$. We report the accuracy of matching the color of the objects, their locations on the $3\times 3$ grid, and their type. We used a maximum of 5 iterations in $\operatorname{LLMPhy}$ with feedback to the LLM with history of past PSNRs and their respective synthesized programs.**
| Method | Color+Loc. (%) | Loc.+Type (%) | Color+Loc.+Type (%) |  |
| --- | --- | --- | --- | --- |
| $\operatorname{LLMPhy}$ (best PSNR) | 68.7 | 66.3 | 56.0 |  |
| $\operatorname{LLMPhy}$ (1 iteration) | 50.0 | 49.3 | 36.8 |  |

In Experiments 4–6, we compare to the various black-box optimization methods for estimating the physics parameters while keeping the Phase 2 inference from $\operatorname{LLMPhy}$ as in the Experiment 3. To be comparable, we used the same number of iterations for all the methods.(^5^55For $\operatorname{LLMPhy}$, we are limited by the context window of the LLM (GPT-4o) and the cost, and thus going beyond 30 iterations resulted in the synthesized programs having errors, and thus they could not be executed.) As can be noted from the table, $\operatorname{LLMPhy}$ leads to better performances compared to other methods in reasoning on the impact outcomes. In Experiments 7–8, we also executed the prior methods for longer number of steps, which improved their performances, however they appear to be still below that of $\operatorname{LLMPhy}$. In Experiment 9, we used the ground truth physics attributes for the respective objects in the simulation, and found 65.1% accuracy, which forms an upper-bound on the accuracy achievable, also suggesting the remaining performance drops (35%) are from the inability of the LLM to correctly estimate the qualitative attributes of the problem layout and the objects.

In Figure 6 and  7, we present a qualitative result from our dataset. We show an example Phase 1 scene, and the predicted trajectories using our method as well as comparison to the optimization using BO. As is clear, for this specific example, $\operatorname{LLMPhy}$ shows strong trends of convergence, while BO shows an oscillatory behavior with high prediction error. The second subplot in Figure 6 super-imposes the trajectories predicted by our model, demonstrating the agreement, and efficacy of predicting the physics parameters.

In Table 2, we present the accuracy of $\operatorname{LLMPhy}$ in localizing the triplets correctly in Phase 2. We found that with nearly 68% accuracy, our model correctly infers the color and location of the objects, and with an accuracy of 56% in estimating all the triplets (color of the objects, their type, i.e., wine glass, etc., and their location in the $3\times 3$ grid). Note that, interestingly we found that GPT-4o cannot easily imagine the grid layout of our object arrangement on the tray(^6^66A curious reader might ask why not use computer vision methods to infer the layout triplets? While that may be possible, we believe that is perhaps outside the focus of this study, where our objective is to use an LLM-simulator combination for discrete black-box optimization to show that the error feedback to the LLM can be used to improve its inference.) and needs several iterations through our optimization prompting to improve its results. We illustrate this interesting aspect of GPT-4o in Figure 7, where we show the input image, and its top-down view (second column), and the layout predicted by the LLM (third column). In the last column, we show the difference image between the second and third columns that clearly show the red wine glass being misplaced by the generated code. This result is after 5 iterations of Phase 2 LLM. We see that while the $\operatorname{LLMPhy}$ iterations do improve the layout estimation (Table 2), the difference image still shows layout errors, which impacts performance, as the answer options are conditioned the ground truth object instance triplets. For example, inferring an object at location (2,2) of the $3\times$3 layout grid as a *brown martini glass* instead of a *green wine glass* may result in an entirely different selection from the answer options after the impact simulation.

Figure: Figure 8: Left: comparison of convergence when using various state-of-the-art LLMs in $\operatorname{LLMPhy}$ against Bayesian optimization and CMA-ES. We plot the minimum loss computed thus far in the optimization process against the number of optimization steps. Right two plots show the convergence of $\operatorname{LLMPhy}$ and the error variance for Phase 1 and Phase 2. We also compare the convergence using OpenAI o1-preview model as the LLM used in $\operatorname{LLMPhy}$.
Refer to caption: x17.png

Figure: Figure 9: Left: Histogram of the best optimization iteration when using $\operatorname{LLMPhy}$ against other methods. Right: Comparison of errors in the estimation of physical parameters using $\operatorname{LLMPhy}$ against alternatives using the ground truth physical parameters. Note that the black-box optimization methods are optimized to minimize the trajectory error by implicitly learning the physics parameters.
Refer to caption: figs/best_iteration_histogram_cm_bo_gpt.png

Figure: Figure 10: We show an example Phase 1 sequence (top). Below, we plot the motion trajectories for each of the objects in the frames and the predicted trajectories by $\operatorname{LLMPhy}$ from the optimization steps. The trajectory plots (below) show the ground truth trajectory (gt) and the predicted trajectory (llm_pred), and as the iterations continue, we can see improvements in the alignment of the predicted and the ground truth object trajectories (as pointed out by the arrows).
Refer to caption: x20.png

Figure: Figure 11: We show qualitative results from $\operatorname{LLMPhy}$ Phase 2 iterations. The input Phase 2 image is shown on the left. The top row shows the images produced by the simulator using the layout prediction code generated by $\operatorname{LLMPhy}$ for each Phase 2 optimization step. Below, we show the difference image between the predicted and the input Phase 2 images, clearly showing the errors. In Phase 2, the feedback to $\operatorname{LLMPhy}$ is produced using PSNR computed on the predicted and the ground truth images, as well as asking LLM (using the difference image) which of the objects are in error, and asking the LLM to fix the layout of these objects in the next iteration. As can be seen, the errors in the LLM layout prediction improves over iterations.
Refer to caption: x21.png

### 4.5 Detailed Analysis

In this section, we make a deeper look into the performances reported in Table 1.

Convergence:
In Figure 8, we plot the mean convergence (over a subset of the dataset) when using GPT-4o, o1-mini, Bayesian Optimization, and CMA-ES. We also compare the results to the more recent, powerful, expensive, and text-only OpenAI o1-preview LLM model, on a subset of our dataset; these experiments used a maximum of 20 optimization iterations. The convergence trajectories in this figure show that o1-mini and o1-preview perform significantly better than GPT-4o in Phase 1 optimization. Specifically, the o1 model being explicitly trained for solving scientific reasoning appears to be beneficial in our task. Interestingly, we see that o1’s initial convergence is fast, however with longer iterations CMA-ES appears to outperform in minimizing the trajectory error. That being said, the plots in Figure 9 and Table 1 points out that having lower trajectory error does not necessarily imply the physical parameters are estimated correctly (as they are implicitly found and are non-linear with regards to the trajectories), and having knowledge of physics in optimization leads to superior results. Further to this, in Figure 9, we plot the histogram of best Phase 1 iterations between the various algorithms. Recall that the optimization methods we use are not based on gradients, instead are sampled discrete points, and the optimization approach is to select the next best sample towards minimizing the error. The plot shows that $\operatorname{LLMPhy}$ results in its best sample selections happen early on in its iterations than other methods.

In Figure 8, we plot the convergence of $\operatorname{LLMPhy}$-Phase 1, alongside plotting the variance in the trajectory error from the estimated physical parameters when used in the simulations. We found that a powerful LLM such as OpenAI o1-mini LLM or o1-preview demonstrates compelling convergence, with the lower bound of variance below that of other models. Our experiments suggest that better LLMs may lead to even stronger results.

In Figure 8, we plot the convergence of $\operatorname{LLMPhy}$ Phase 2 iterations improving the PSNR between the synthesized (using the program) and the provide task images. As is clear, their correctness of the program improves over iterations. We would like to emphasize that BO and CMA-ES are continuous optimization methods and thus cannot optimize over the discrete space of Phase 2 layout. This is an important benefit of using $\operatorname{LLMPhy}$ for optimization that can operate on both continuous and discrete state spaces.

Correctness of Inferred Physical Parameters: While the trajectory error is being minimized in Phase 1 via implicitly learning the physical parameters, how well do the LLMs optimize for the actual physical parameters? In Figure 9, we plot the mean absolute error between the predicted physics parameters and their ground truth. Interestingly, we see that the $\operatorname{LLMPhy}$ estimations are better than those by other methods; note that the prior methods do the optimization without any knowledge about the semantic meaning of these optimization variables or their physical interpretations, which we believe the LLM is able to make use of. Specifically, we see the errors in sliding-friction, rotation-inertia, and stiffness are lower than those estimated using Bayesian optimization and CMA-ES, that is perhaps leading to the better performances of $\operatorname{LLMPhy}$ in the results in Table 1.

### 4.6 Qualitative Results

In Figure 12, we show several qualitative results from our TraySim dataset and comparisons of $\operatorname{LLMPhy}$ predictions to those of BO and CMA-ES. In general, we find that when the velocity of the pusher is lower, and the sliding friction is high, objects tend to stay stable if they are heavier (e.g., a bottle), albeit other physics parameters also playing into the outcome. In Figure 10, we show example iterations from Phase 1 that explicitly shows how the adjustment of the physical parameters by $\operatorname{LLMPhy}$ is causing the predicted object trajectories to align with the ground truth. In Figure 11, we show qualitative outputs from the optimization steps in Phase 2, demonstrating how the error feedback to the LLM corrects its previous mistakes to improve the layout estimation.

Figure: Figure 12: Qualitative comparisons between $\operatorname{LLMPhy}$, Bayesian optimization, and CMA-ES.
Refer to caption: x22.png

## 5 Conclusions

In this paper, we introduced a novel task of predicting the outcome of complex physical interactions using an LLM with the help of a physics engine; a combination we refer to as $\operatorname{LLMPhy}$. Our model systematically synergizes the capabilities of each underlying components, towards estimating the underlying physics of the scene, and experiments on our proposed dataset TraySim, demonstrate its superior performances. Notably, as we make no assumptions on the differnetiability of the simulator, our framework could be considered as an LLM-based black-box optimization framework, leveraging LLMs’ knowledge for hyperparameter sampling.

Our study shows that the recent powerful LLMs have enough world “knowledge” that combining this knowledge with a world model captured using a physics engine allows interactive and interative problem solving towards better reasoning. We also find that using more powerful LLMs that are trained for scientific reasoning tasks, seems to demonstrate significantly superior physical reasoning abilities in an optimization pipeline as in LLMPhy. Our study suggests that perhaps training or fine-tuning large scale LLMs may not be necessary, instead we may start finding novel ways to extract and leverage the implicit LLM world knowledge for solving complex problems.

## 6 Limitations

While, our simulation setup has been designed to be as general as possible, we note that we are only experimenting with four physical attributes of the physics engine to study the plausibility of implicitly estimating a world model. In real-world settings, there are many more important physics attributes that need to be accounted for. Further, we considered the use of GPT-4o in our evaluations, as it is popular for its excellent performance. We note that as it is a closed-source model and its responses may be stochastic, making it difficult to provide any guarantees on the performance. However, we would like to note that our key intention is to show the usefulness of an LLM for solving this task, and we hope future open-source LLMs would be applicable as well to our setting.

## Appendix

Below, we present the prompts we used for the LLM in our experiments for Phases 1 and 2, as well as depicting the programs LLM generate.

Phase 1 Prompt:

The given scene has a tray with three objects (a bottle, a wine_glass, and a martini_glass) on it. The radius of the tray is 1.8 and its center of gravity is 0.05 above the ground with a sliding friction of 0.1 and no spin or roll friction. The radius of bottle is 0.4 and its center of gravity is 1.1 above the ground. The center of gravity of the martini_glass is at a height of 0.5.The center of gravity of the wine_glass is 0.9 above the ground. The tray is impacted by a pusher and the tray with the objects on it moves. Python code in example_code_1.py creates the scene and runs the simulation. The trajectories in object_traj_example_1.txt show the motion of the center of gravity of the objects when running the simulation. Your task is to analyze the given example and then write similar code to produce the trajectories given in ’problem_trajectories.txt’.

You must assume the scene is similar to the one given, however the physics between the tray and the objects are different, that is, the sliding-friction, damping, stiffness, and armature need to be adjusted for all the physical_parameters_for_object_id_* dictionaries in the example_code_1.py so as to reproduce the trajectories in ’problem_trajectories.txt’. You must assume that the physics of the tray with the ground remains the same and so is the external force applied on the tray by the pusher. The trajectories use a time step of 0.2s. Do not attempt to change the physics parameters beyond their first significant digit. Your written code should strictly follow the same code structure as provided in example_code_1.py. You may further assume that multiple instances of the same object will have the same physical parameters.

You must not change the ’mass’ of the objects in your generated code. Do not include the object trajectories in your generated code as that will fail our simulator.

Note that the simulation trajectory in problem_trajectories.txt may use instances of bottle, martini_glass, and wine_glass. The name of the objects is provided in the problem_trajectories.txt file. The mass for the objects are as follows: wine_glass is 4.0, martini_glass is 10.0 and bottle is 20.0.”

Phase 2 Prompt:

## Appendix A An Example of Synthesized Programs:

Below, we show an example of Python programs synthesized by $\operatorname{LLMPhy}$ models, which are then run to invoke the Mujoco Simulator to execute and render the dynamical sequence, to extract the outcome of the simulation.

Phase 1 program:

Phase 2 program:

## Appendix B Example LLM Response (Phase 1)

Below, we provide the full log of the optimization iterations produced by a phase 1. Specifically, the text shows how the LLM uses the provided feedback to adjust the physical attributes, with explanation of the justification for the adjustments. Note that each response from the LLM are separated by strings ”=======”.

## Appendix C Example LLM Response (Phase 2)

Similar to above, we present an example optimization steps from a phase 2 $\operatorname{LLMPhy}$ optimization.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: I am recommending a weak rejection (marginally below acceptance) at this point primarily due to the limitations of the experiments. I think the paper is very well written with a robust technical description, but it is hard to draw conclusions from the experiments, preventing the paper from being a sufficiently significant scientific contribution. While I also think that the specific physical reasoning being evaluated is simple (which objects remain upright), more comprehensive experimentation would make this paper acceptable. To be clear, its potential impact is bounded by this simplicity, but I think the problem is still sufficiently interesting and challenging.
	
Here are some actionable additional experiments that would certainly help improve my recommendation:
- Introduce more object categories that exhibit sufficiently distinct physical behaviors after interaction
- Allow multiple sizes of the tray during testing to investigate having more objects and different configurations of objects.
- Test multiple initial positions and velocities for the pusher; as I understood, only one initial position and velocity was used in the tests (line 403-405).
- It would be better to include more LLMs for evaluations, ideally at least another SOTA black-box model and a SOTA open model. However, this can be prohibitive due to cost, so this is a lower priority than other experimental improvements as long as it is clearly acknowledged that conclusions about LLMs more generally are not possible.
	
I would also be interested in an ablation that provided ground-truth scene layouts during the optimization process in order to better understand the optimization performance of the LLM with respect to just the physical properties.

Here are some questions that I would appreciate having answered:
- What is the format of the multiple choice answers?
- Does the model exhibit consistency when presented with the inverse question, i.e., which objects will be knocked over?
- Why does each object category have the same allowable tilt \alpha?
- While Figure 3 shows the trajectory error, I wonder about the error for each individual parameter value; it seems like this should be available and might be interesting. Do the individual physical properties pose different difficulty for the LLM to infer?

Here are some minor comments (these had no impact on my recommendation):
- Since images are provided as input in some phases, it would be good to clarify that the LLM in question is multi-modal. Some readers may expect the acronym VLM for models that can also accept images as input. This is stated, but I recommend being proactive about avoiding any confusion about what the input is and how it is encoded.
- Figure 3 x-axis intervals should be integers
- missing space: "atleast" line 396
- double word: "aproaches that that use" line 460
- mispelling: "differnetiability" line 487
- Recommend a general grammatical pass, several other minor errors
- It would be good to write out PSNR (line 427)
soundness: 2
strengths: - The paper is well written and organized, and the method is clearly explained and reasonably illustrated for further clarity.

- Both the problem and the proposed method are thoroughly mathematically formulated, which further helps with making the method understandable.

- The problem being addressed, namely reasoning about the result of complex multi-body physical interactions, is challenging and is relevant for enabling important future embodied capabilities.

- I like the ablation over the source of the physical parameter values (Random and Ground Truth), and I agree with the conclusions drawn from that ablation.
summary: This paper addresses the challenge of reasoning about the result of a complex multi-body physical interaction; specifically, the goal is to predict which objects will remain upright on a tray after an external force is applied by a pusher mass. This prediction is done by prompting an LLM to answer a multiple choice question about upright objects given the context of a synthesized video sequence of the multi-body physical interaction produced by a physics simulator. The key contribution of LLMPhy is an iterative framework between an LLM and the physics simulator to infer unknown physical properties of the objects in the scene, thereby allowing the physics simulator to synthesize the aforementioned video sequence.
	
The physical properties of the object categories are inferred by having the LLM generate programs for the simulator that attempt to model a reference dataset of videos of objects experiencing a different external force than the test set. The generated programs essentially consist of LLM-generated values for the unknown physical parameters, LLM-generated scene layouts that attempt to match those in a given reference video (object class, object location, and object color for each object in the video), and executable code for the physics simulator. This inference process is iterative and uses the error between the reference video's object trajectories and the simulated object trajectories as feedback to the LLM in order to generate updated (and ideally better) values of the unknown physical properties and a better scene layout. Once the values of the physical properties of the objects are estimated through this optimization process, they are applied to new scenarios involving more objects and a different external force; given only the first frame of a video of an interaction and the inferred object properties, an LLM generates a program for the simulator which then synthesizes a video of the interaction. This video is the video mentioned above that is provided as context to an LLM to answer a multiple choice questions about which objects will remain upright.
	
In addition to the LLMPhy framework, this paper also introduces a novel task and dataset.
weaknesses: - The experiments are limited. Only 3 objects categories are included in the experimentation, and the scenarios consist exclusively of a single external force being applied to a tray containing objects; further, the reasoning question that is being answered is relatively simple (which objects remain upright) compared to the kind of information one might want for practical uses, such as the pose of the objects.

- Only GPT4o is used for the LLM, which makes any extrapolation or empirical trend to other models imposible.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. A qualitative example should be provided in the main paper instead of just in supplements to ground the method for the reader.
2. How will this method generalize to real world if ground truth values from the simulator are used?
3. What are plots submitted in supplementary supposed to mean, they are neither referenced in the main of supplementary pdfs.
soundness: 2
strengths: The authors attempt a novel task for improving low-level physical reasoning using an LLM, such as predicting physics parameter or object trajectories, instead of most prior works (to the best of my knowledge) do very high level semantic physical reasoning with LLMs. I believe this line of work is quite interesting, but the paper is needs more work both in terms of experimental study and presentation. See comments / suggestions below.
summary: The paper proposes a task, dataset, and solution for an LLM-prompting based physics parameter estimation and trajectory estimation using privileged simulator feedback in the loop.
weaknesses: 1. The abstract doesn’t differentiate proposed work with existing literature and justify the need of a new dataset/method.
2. Writing is a slightly unstructured and somewhat informal. For instance, L110: ‘..leads to significant improvements in performance against alternatives, including using Bayesian optimization and solely using only LLM for..’ would read more formal for a research paper as ‘leads to x% improvements over baseline methods that use only LLM or Bayesian optimization for..’. The current statement is unclear as to weather these are two different baselines or not. The use of word ‘significant’ without any concrete number is not standard for a research paper. L123: ‘Our experiments using LLMPhy on the TraySim dataset demonstrate promising results’, same issue with the word ‘promising’ - it is vague and not very concrete. L192: ‘Our attempt to solve the…’ - is grammatically incorrect and not sure what’s the meaning of the statement. These are just a few examples but for me most the paper reads like an initial draft and some writing iterations will significantly improve it’s presentation and understanding. Also, organizing the text in subsections (in Section 2, for instance) will improve the presentation. L325: ‘X is the input data, and Λ is the desired result.’ - very ambiguous notation definition. Major writing revision is needed.
3. There’s a claim that the method is generalizable to any simulation however results for only one simulation is presented. If the claim to true, the method application on other existing simulations should be shown as well. Or the authors should discuss what are the challenges for doing so.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) how would LLMPhy perform when scaled to more complex environments with additional physical parameters? Besides, since different physics engines can have unique methods for simulating physical interactions, how might the choice of simulator (e.g., using MuJoCo vs. others) affect the model’s predictions and the general applicability of the method?

(2)  the simulator access setup is interesting, if applicable to real-world tasks, we can use LLM augmented with simulators to perform physical reasoning in real-world. However, the paper is assuming a perfect simulator  that can simulate real-world environment, which makes the results less interesting. It would be nice to add such experiments (studying how sim2real gaps will impact the evaluation)
Since the paper only use synthetic dataset to evaluate, this raise the question if the method applies to real-world robotics settings

(3) Reproducibility with Open-Source LLMs: Given that GPT-4o is closed-source, how feasible is it to adapt LLMPhy to open-source alternatives, and what changes would be necessary to maintain accuracy?

(4) Can the authors provide insights into how the framework might handle situations where parameter estimation fails due to high variability in initial conditions or other environmental factors?
soundness: 3
strengths: The paper proposes a black-box optimization framework that uses In-context learning to enhance the physical reasoning skills of SOTA LLMs.To augment the reasoning, the framework leverages the coding capability of LLMs and give it access to a simulator. This paper is a valuable contribution to physical reasoning using LLMs, presenting an original approach that effectively combines the strengths of language models and physics-based simulations. The results are promising, though some improvements in scalability, dataset credibility, and validation across physical engines would strengthen the framework’s applicability and robustness.
summary: The paper introduces a framework called LLMPhy, which combines large language models (LLMs) with physics engines to tackle complex physical reasoning tasks. The proposed method is tested on a new dataset, TraySim, where the model predicts object stability and interactions on a tray after an impact. LLMPhy operates in two phases: parameter estimation and simulation, with the LLM synthesizing hypotheses and the physics engine verifying them in a feedback loop. 

Key Strengths: 

TraySim Dataset: A tailored benchmark for multi-object physical interactions.

Zero-Shot Reasoning: Achieves notable improvements over Bayesian optimization on complex tasks. 

Limitations: 

Limited Parameter Scope: Currently models only four physical attributes, reducing generalizability. 

No Real-World Testing: Results are based solely on simulations, lacking validation in physical settings.
weaknesses: (1) my first concern is the limited physical parameters, the model currently explores only four physical parameters, which restricts its application to broader and more complex real-world scenarios. This constraint is acknowledged by the authors but remains a significant limitation for potential applications in varied physical environments.

(2) Dataset Scale and Generalization: With only 100 sequences, TraySim may be limited in capturing the full complexity of physical interactions. Expanding this dataset or testing LLMPhy on existing physics-based benchmarks could provide a more comprehensive understanding of the model’s generalization ability.

(3) Absence of Real-World Validation: While the simulation results are promising, real-world validations or experiments in a physical setup would enhance the credibility of the proposed method. Without real-world tests, it remains uncertain how well the model's inferred physical parameters would translate to actual physics.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The authors may provide answers for my clarification questions about the method.
soundness: 3
strengths: This paper addresses an important question in AI by enabling large language models to perform complex physical reasoning, which is essential for real-world applications. The authors introduce a new framework, LLMPhy, that combines large language models with optimization algorithms and a physics simulator to estimate physical parameters and predict interactions. They also present a new dataset, TraySim, designed to benchmark models on multi-object physical reasoning tasks, making it a valuable resource for advancing AI’s capabilities in dynamic environments.
summary: This paper proposed a new framework combining Large Language Models (LLMs) with physics engines and optimization algorithms for complex physical reasoning. This approach cab infer and simulate multi-object dynamics under physical interactions. The authors propose a new dataset, TraySim, where objects on a tray are impacted by an external force, simulating dynamic, inter-object interactions.
weaknesses: First, the paper lacks a discussion on its relationship with existing datasets like CoPhy https://arxiv.org/abs/1909.12000 and ComPhy https://comphyreasoning.github.io/, which also aim to infer physical properties of objects. These datasets offer benchmarks widely used in physical reasoning research, and a comparison of the proposed method on them would strengthen the validation of LLMPhy and clarify its distinct contributions.

Second, several established methods already tackle similar tasks, such as the approach presented in the ComPhy paper and methods like https://arxiv.org/abs/2012.08508 The absence of comparisons with these methods and relevant literature weakens the contextual positioning of LLMPhy within existing approaches.

Third, the methodology lacks clarity, especially regarding critical details like how multiview images are utilized to reconstruct object shapes and the specific inputs provided to the LLM. For example, if the LLM samples optimization variables based on physics knowledge from its training data, is object category information also supplied to guide this process?

Lastly, the unclear explanation of how raw RGB video data is converted into simulation inputs makes it difficult to assess how this approach could apply in realistic scenarios beyond controlled simulations. This raises concerns about the generalizability of LLMPhy to real-world applications.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: 1. How is $\tau$ implemented?
2. Why choose to generate programs rather than directly generating relevant values for each attribute, especially when no real program logic, such as if conditions, for loops, or external libraries, is used?
soundness: 2
strengths: This work presents a new approach that combines the strengths of large language models with advanced physics simulators. Additionally, it establishes a benchmark dataset, TraySim, which could be valuable to support future research in physical dynamics prediction.
summary: This work introduces LLMPhy, a zero-shot black-box optimization framework that integrates LLMs with physics engines to predict the dynamics of objects on a tray under external impact. The framework comprises two main phases:
1. The LLM infers the physical parameters of objects, including the coefficient of sliding friction, stiffness, damping, and rotational inertia, based on in-context learning examples and the objects' physical trajectories extracted from video data. The LLM then generates a program to simulate these trajectories, aiming to minimize the error between simulated and actual observed trajectories.
2. The LLM determines the scene layout by inferring object characteristics such as class, location, and color, based on the previously inferred physical parameters, the initial frame of the video sequence, and additional in-context examples.

Finally, LLMPhy refines its predictions using feedback from the simulator to minimize error. At each step, it keeps a record of previous attempts and errors, which helps the model improve its accuracy over multiple trials.


This work also introduces TraySim, a new dataset of simulated video sequences featuring a tray with varying numbers of objects. Evaluation of TraySim reveals that the proposed method outperforms baseline approaches, including random selection of physical parameters and direct use of an LLM.
weaknesses: 1. The optimization step in Algorithm 1 appears to depend largely on the LLM’s ability to learn from previous errors through repeated attempts, making the approach heavily reliant on trial-and-error. Other optimization algorithms, such as CMA-ES, might also achieve comparable or potentially even better results. To strengthen the paper’s claims, consider including comparisons with additional optimization algorithms.
2. The proposed method is similar to the approach in the paper "Eureka: Human-Level Reward Design via Coding Large Language Models." In that work, an LLM is used to design reward functions that help train a reinforcement learning agent to teach a five-fingered robotic hand how to spin a pen.
3. The choice of intersection over union (IoU) as the evaluation metric is unclear. What does the answer set consist of? Why not simply use an exact match with the answer set instead?
4. In the second phase of LLMPhy, it’s unclear why predicting the color of the object is relevant for understanding its dynamics. Additionally, why is an LLM necessary to infer the object’s starting location? Wouldn’t it be simpler to use the tau function to extract the object’s location directly from the first frame of the image?
5. In Figure 2, an LLM processes the question along with the synthesized dynamics video to produce an answer, but it is unclear where the paper explains this process in detail. How exactly is this achieved?
6. The generalizability of the proposed Algorithm 1 is unclear, as it appears that the algorithm needs to be reinvoked for each new scenario to infer the physical parameters of the objects. Is there a way to enable the LLM to remember what it has learned and apply it to future scenarios?

Meta-review: This paper introduces LLMPhy, which integrates LLMs with a physics simulator to address physics reasoning tasks. The main claim is that LLMPhy can iteratively estimate the physical hyperparameters of a system using an analysis-by-synthesis approach and leverage the inferred properties to solve reasoning tasks.

Strengths:

+ Dataset Contribution: TraySim provides a new benchmark for evaluating multi-object physical interactions.
+ Potential in Advancing Physical Reasoning: The paper demonstrates the promise of using LLMs for physical reasoning tasks.

Weaknesses
- Limited Experimental Scope: Multiple reviewers note that the experiments are limited in scope. Tests are restricted to a few object categories (initially three, later expanded to five). Additionally, the scenarios are simplistic, as they involve only single external forces applied to objects on a tray.
- Real-World Validation: A significant limitation of the paper is the lack of real-world testing. Reviewers also question the real-world applicability of the proposed method and whether it can translate effectively outside the simulation environment.

Overall:
The novelty of LLMPhy and its potential for advancing physical reasoning are notable strengths. However, several critical issues prevent its acceptance in its current form. The experimental scope is too narrow, and additional testing with diverse setups, simulators, and real-world validations is crucial to substantiate its claims.

Decision: Reject