I-PHYRE: Interactive Physical Reasoning
Authors: Shiqian Li
Kewen Wu
Chi Zhang
Yixin Zhu
Venue: ICLR 2024 poster (2024)
Keywords: Intuitive physics
physical reasoning
Primary area: datasets and benchmarks
TL;DR: Build human-level interactive agents that can control the physical dynamics by reasoning, planning, and intervention.

Abstract: Current evaluation protocols predominantly assess physical reasoning in stationary scenes, creating a gap in evaluating agents' abilities to interact with dynamic events. While contemporary methods allow agents to modify initial scene configurations and observe consequences, they lack the capability to interact with events in real time. To address this, we introduce I-PHYRE, a framework that challenges agents to simultaneously exhibit intuitive physical reasoning, multi-step planning, and in-situ intervention. Here, intuitive physical reasoning refers to a quick, approximate understanding of physics to address complex problems; multi-step denotes the need for extensive sequence planning in I-PHYRE, considering each intervention can significantly alter subsequent choices; and in-situ implies the necessity for timely object manipulation within a scene, where minor timing deviations can result in task failure. We formulate four game splits to scrutinize agents' learning and generalization of essential principles of interactive physical reasoning, fostering learning through interaction with representative scenarios. Our exploration involves three planning strategies and examines several supervised and reinforcement agents' zero-shot generalization proficiency on I-PHYRE. The outcomes highlight a notable gap between existing learning algorithms and human performance, emphasizing the imperative for more research in enhancing agents with interactive physical reasoning capabilities. The environment and baselines will be made publicly available.

Published as a conference paper at ICLR 2024
I-PHYRE: INTERACTIVE PHYSICAL REASONING
Shiqian Li Kewen Wu?.3, Chi Zhang 2
Yixin Zhu [
1 Institute for Artificial Intelligence, Peking University
2 National Key Laboratory of General Artificial Intelligence, BIGAI
3 Department of Automation, Tsinghua University
https: https:://1sshigiannggg www.aanhggggg.gh....b...o github. i TPHYRE/
ABSTRACT
Current evaluation protocols predominantly assess physical reasoning in stationary scenes, creating gap in evaluating agents' abilities to interact with dynamic events. While contemporary methods allow agents to modify initial scene configurations and observe consequences, they lack the capability to interact with events in real time. To address this, we introduce Interactive PHysical Reasoning (I-PHYRE), framework that challenges agents to simultancously exhibit intuitive physical reasoning, multi-step planning, and in-situ intervention. Here, intuitive physical reasoning refers to quick, approximate understanding of physics address complex problems; multi-step denotes the need for extensive planning sequences in I-PHYRE, considering each intervention can significantly alter subsequent choices; and in-situ implies the necessity for timely object manipulation within a scene, where minor timing deviations can result in task failure. We formulate four game splits to scrutinize agents' learning and generalization of essential principles of interactive physical reasoning, fostering learning through interaction with representative scenarios. Our exploration involves three planning strategies and examines several supervised and reinforcement agents' zero-shot generalization proficiency I-PHYRE. The outcomes highlight a notable gap between existing learning algorithms and human performance, emphasizing the imperative for more research in equipping agents with interactive physical reasoning capabilities.
INTRODUCTION
Consider the dynamics of playing game of 3D pinball, where achieving a high score is contingent upon making swift and accurate physical judgments. Players must meticulously control the launcher, anticipate the ball's trajectory, understand the paths it will take upon interacting with targets and bumpers, and determine the optimal moments to activate the flippers. Given the intricate nature of the ball's trajectories, players are compelled to adapt in real time to the game's 's sudden shifts; professional players demonstrate mastery in this game, with records of continuous play near three hours.' ! Indeed, such proficient and interactive physical reasoning is not exclusive to humans but common trait observed across various species. Even toddlers exhibit a remarkable ability to interact seamlessly with the physical world, demonstrating capabilities in seeing, interacting. and in-situ planning (Spelke & Kinzler, 2007; Fazeli et al., 2019). Studies on animals such as crows (Gruber al., 2019), pigeons (Epstein et al., 1984), and apes (Tecwyn al., 2013) also reveal analogous multi-step planning abilities employed for solving complex problems.
The profound physical reasoning aptitude observed in humans and animals has spurred our interest in equipping intelligent agents with analogous capabilities. Contemporary benchmarks have emerged to assess the prowess of physical reasoning agents, ranging from determining the stability of block stack (Lerer et al., 2016) to navigating intricate physical games (Bakhtin et al., 2019; Allen et al., 2020). However, these prior works often overlook a pivotal facet of physical reasoning-interr interaactivtyy This dimension demands that an agent concurrently: (i) understand the physical events intuitively. enabling swift yet approximate predictions of future outcomes, (ii) strategize multi-step actions, considering the anticipated environmental shifts resulting from interventions, and (iii) execute in-situ interventions, ensuring precise timing with the environment.
'htpps: :ttp:///wwwwwww youtube. com hoom/wat.oe 24 04_84844888888 1PFV

---

Published as a conference paper at ICLR 2024
Task: Drop the red balls into the abyss by eliminating gray blocks
Intermediate State
Suboptimal solution: steps
Porfect soluton: Use stage
Soccess
Falluze
Basic Game
Fullure Miss the perfect timingg
Fuilure Wrong order leads missing elimination timingg for two balls
Noiey Game
Compositional Game
Multi-bell Game
Perfect solution: Use 2 steps
Suboptimal solution: Disturbod by the noley block
In sataa adaptaztion
Optimal
Failurm Wining order
Compositional solution: Long sequence planning
Figure I: Illustrations the four splits I-PHYRE. The objective guide all red balls into the hole by strategically climinating gray blocks stepwise manner. We showcase one game from each split (basid, noisy, compositional, and multi-ball) and explore the potential dynamics that different interventions could unfold. Images encased in black borded, cyan border, green border, and red borded represent intermediate states, time steps at which an elimination action undertaken. frames depicting success, and frames indicating failure, respectively. The gray and black blocks remain stationary. while the light blue blocks are mobile, influenced by gravity and collisions; only the gray blocks are subject to climination. Successfully solving the games necessitates reasoning about the sequence of interventions and meticulously managing the exact timing of interventions.
Indeed, prevailing studies exhibit notable limitations in exploring physical reasoning due to the following constraints:
Existing works predominantly permit either passive observation or a single-round intervention. These one-round settings encompass (i) rendering responses post-observation of dynamic scenarios (ce Baillargeon, 1987; Hespos & Baillargeon, 2001; Spelke et al., 1992), and (ii) modifying the initial setup through singular intervention in a given stationary scene (Bakhtin al., 2019; Allen et al., 2020). The restriction to single action precludes the exploration of action compositionality in extended sequences and the sequential impact of physical interventions.
Existing frameworks are inadequate for assessing the repercussions of action timing. Given that all interventions are executed while the scenes are static static-albeii subsequent unfolding of dynamic events is possible (Bakhtin al., 2019; Allen et al., 2020) the evaluation of the temporal impact actions remains unattainable.
To rectify the aforementioned shortcomings, present Interactive PHysical Reasoning (I (I-PHYRE), the first benchmark explicitly crafted to bridge the existing gaps in physical reasoning. I-PHYRE necessitates agent to contemplate the physical repercussions of its interventions and to interact sequentially with the environment with impeccable timing to attain the objective. I-PHYRE is conceptualized based the following principles:
Intuitive physical reasoning: Our focus is on fostering a rapid yet approximate capability for predicting physical outcomes, rather than exact simulation or modeling of the physical events. Multi-step interventions: The resolution of a problem demands the execution of actions in multiple steps. contrast to single-action setups, multi-step actions require agents engage in both long-range prediction and interventional reasoning within the specified scene. In-situ interventions: The essence of interaction in physics dynamics lies in timing. Agents that deviate even slightly in timing at any step may potentially fail to complete the task.
I-PHYRE is block elimination task; see Fig. for illustrations. Agents must ensure all red balls fall into the hole by removing the minimum number of blocks possible. The benchmark incorporates varying levels difficulty and segmented into different generalization splits. We have formulated 40 distinct games and segregated them into four splits, including basic split for training and three additional generalization splits. These are intended to assess the authentic interactive physical understanding of agents beyond mere data fitting. Specifically, the generalization splits designed to test the agents' ability to (i) discern key physical elements amidst noise, (ii) strategize for long stringent timing constraints.
sequences through compositionality, and (iii) conform
2

---

Published as a conference paper at ICLR 2024
Importantly, agents must discern when to remove which block as the game progresses. In certain intricate scenarios, the sequence and timing of interventions become pivotal due to the perpetually evolving dynamics. To illustrate, refer to Fig. to analyze how interventions, varying in order and timing, can precipitate entirely disparate physical occurrences.
Basic game Initially removing the block either to the right of or beneath the red ball will set it free; however, the latter approach will merely cause the ball to descend vertically, bypassing the hole, as depicted in Failure: Wrong order. The former strategy propels the ball downward the right, necessitating agents to act promptly to eliminate the block below, directing the ball straight into the hole; this is illustrated in Perfect solution: Use 2 steps. An alternate tactic involves removing the two blocks on the right platform post the elimination of the top block, resulting in three steps; this scenario is portrayed in Suboptimal solution: Use 3 steps.
Noisy game The introduction of an extraneous block should not alter the optimal strategy. The newly added gray block beneath the inclined block can be conveniently disregarded the timing is accurately computed; this is illustrated in Suboptimal solution: Disturbed by the noisy block.
Compositional game Compositional games amalgamate concepts found in basic games (e.g.. events in Compositional solution: Long-sequence planning), leading to scenarios with an increased number of blocks and necessitating longer-range planning.
Multi-ball game Introducing an additional ball to the scene elevates the level of challenge. The optimal solution requires an agent to simultaneously drop two balls on the long block and remove the block at the right moment, directing both balls into the hole; refer to Perfect solution: Use 3 steps. The crucial aspect here is the sequence in which the blocks on either side are removed; if the order of actions is inverted, the right ball will arrive at the block too late and will descend directly to the left platform, as seen in Failure: Wrong order leads to missing elimination timing for two balls. Nonetheless, one can still succeed in the game by swiftly adapting to the alteration and removing the top right inclined block at the correct moment; this is depicted in In-situ adaptation.
We conduct experiments utilizing three distinct planning strategies within I- PHYRE. (i) Planning in advance: the agent predefines sequence of interventions at specific time steps, relying solely on the initial scene. (ii) Planning on-the-fly: the agent decides its current action based on the present scenario. (iii) Combined strategy: the agent recalibrates its planned timing after executing the carliest action. Both supervised and reinforcement agents undergo evaluation on I-PHYRE using zero-shot generalization approach, being trained on the basic split and tested on the remaining three splits. To gain deeper insights into the capabilities of existing algorithms in interactive physical reasoning, we also establish human bascline. The comparative analysis reveals that current learning algorithms are yet to match human proficiency in interactive physical reasoning, particularly regarding generalization (Grenander, 1993; Fodor et al., 1988; Lake al., 2015; Zhu et al., 2007; George et al., 2017; Lake & Baroni, 2018).
In conclusion, work renders three significant contributions:
We delineate the challenge of interactive physical reasoning and unveil I-PHYRE, the first bench- mark developed to scrutinize interactivity in physical reasoning.
We devise three planning strategies for interactive physical reasoning (i.e., planning in advance, planning on- and combined) and implement with both supervised and reinforcement agents. We facilitate a comparative analysis between human performance and various learning algorithms, elucidating the challenges and charting the prospective avenues in interactive physical reasoning.
2 RELATED WORK
Intuitive physics Research in intuitive physics is broadly bifurcated into two methodological streams. Traditional methods are anchored in the violation-of-expecattio (VoE) paradigm (Spelke et al., 1992), where developmental studies gauge infants' intuitive physics acquisition by presenting them with events that either adhere to or violate physical laws, quantifying surprise through looking time at both event types (ce Baillargeon, 1987; Hespos & Baillargeon, 2001); a longer gaze typically indicates perceived violation of physical laws. To instill comparable human- physical intuition in learning algorithms, contemporary computational methods utilize simulators to replicate similar stimuli, evaluating whether modern learning models exhibit "surprise" at events that contravene
3

---

Published as a conference paper at ICLR 2024
Table 1: Comparison between I-PHYRE and related benchmarks. I-PHYRE employs intuitive physics
(instead of precise computation), showcases intricate dynamics, necessitates multi-step planning, demands accurate sequencing of interventions (i.e. action order), and enforces impeccable timing of interventions i.e. action timing) to resolve intricate problems.
Benchmarks
mechanism
rich dynamics
multi-step
action order
action timing
Block Towers (Lerer el al., 2016)
intuition
X
x
ComPhy (Chen ot al., 2022)
intuition
x
x
X
x
PHYRE (Bakhtin et al,, 2019)
intuition
/
X
X
X
Virtual Tools (Allen et al., 2020)
intuition
/
X
X
X
SMP (Toussaint et al., 2018)
computation
X
/
/
/
I I-PHYRE (ours)
intuition
/
physical laws (Piloto et al., 2022). Importantly these prediction-centre methods only emphasize passive observation, yielding only a rudimentary understanding of physical reasoning capabilities due to the absence of scene interactivity. The exploration of non-interactive physical reasoning also extends to trajectory prediction and question-answeriig muestion-ansseriing (Yi et al., 2020; Chen et al., 2022).
Contemporary approaches address intuitive physics primarily through the perspective of tool use (Zhu ct al., 2015; Toussaint et al., 2018). exemplified by frameworks like PHYRE (Bakhtin et al., 2019) and Virtual Tool Game (Allen et al., 2020). Even though tool use inherently necessitates multi-step planning (Toussaint et al., 2018; Zhang et al., 2022), prevailing benchmarks (Bakhtin et al., 2019; Allen al., 2020) allow only single interaction step with the scene. Consequently, computational models addressing these benchmarks predominantly develop supervised models, utilizing classifiers distinguish between the success and failure of an action (Li et al., 2022) incorporating dynamic prediction modules to inform judgments (Girdhar et al., 2020; Qi et al., 2021). However, none of the existing works explicitly incorporate any degree of interaction.
In summmary, existing studies fail to thoroughly investigate the interactive aspect of physical reasoning in both humans and machines. This absence of interactivity hinders exploring multi-step physical reasoning. To bridge this gap, we introduce I-PHYRE, new benchmark for interactive physical reasoning. Tab. provides a comprehensive comparison between I-PHYRE and preceding works.
Multi-step planning Planning in intricate enviromments is a distinctive feature of intelligence. We direct readers seeking broader understanding of this extensive field monograph on planning (LaVal 2006) and a contemporary review (Garrett et al., 2021), while we narrow our focus to recent developments in multi-step planning that incorporate explicit modeling of physical constraints.
Sequential Manipulation Planning (SMP) stands out pivotal domain within multi-step planning. Contemporary research in this area addresses multi-step physical reasoning through (i) optimization in robotic simulators (Toussaint et al., 2018; 2020), (ii) adaptive sampling with tangible robots (Wang et al., 2021; Konidaris et al., 2018), or (iii) graph- planning with visually grounded symbols (Jiao et al., 2022; Han et al., 2022). Although advancements have been made in multi-step planning, interaction with dynamic environments remains a formidable challenge; precise timing of interventions necessitates swift responsiveness opposed prolonged computation.
Contrastingly, I I-PHYRE emphasizes swift, intuitive multi-step planning. The intricate dynamics (e.g., falling, rotation, collision, friction, pendulum, elastic motion) and varied interactivity within I- I-PHYRE necessitate meticulous sequencing and impeccable timing of interventions- areas that have remained largely unexplored in the realm of physical reasoning.
RL agents Reinforcement Learning (RL) agents have achieved significant triumphs in the realm of gaming, surpassing human performance in strategic games post extensive training: (i) mastering Atari games (Mnih et al., 2015), (ii) conquering Go (Silver et al., 2016), and (iii) excelling in StarCraft (Vinyals et al., 2019). Despite the increasing complexity in action strategies (i.e., evolving from Go and Atari to StarCraft), few environments explore fundamental physical reasoning beyond basic rules in games like Breakout. While some benchmarks incorporate physics, the emphasis predominantly lies on the generalization of actions (Jain et al., 2020). To counter this limitation, we introduce I-PHYRE, a novel challenge centered around physical reasoning, with a focus on the generalization of interactivity. During our experiments, their limitations. A comparative study between humans and RL agents fosters discussions regarding prospective developmental trajectories in this domain.
assess classic RL algorithms against I -PHYRE, revealing
4

---

Published as a conference paper at ICLR 2024
3 CREATING I-PHYRE
3.1 GAME DESIGN
I-PHYRE encompasses 40 distinctive interactive physics games, each primarily featuring gray, black, and blue blocks, red balls, rigid sticks, and springs. All games share a unified objective: to guide all red balls into the hole by strategically eliminating gray blocks at different time steps, the only blocks that can be removed. Beyond the initial stationary blocks (gray and black), the games also incorporate movable blocks (blue) that react gravity and impact, springs that can expand and compress, and rigid sticks that can roll within the environment. These elements collectively represent fundamental Newtonian mechanics, thereby enriching the physical dynamies of the games.
Splits Games are categorized into four splits-basic, noisy, compositional, and multi-ball-baase on algorithmic stability and varying generalization levels (Xie et al., 2021). These splits are designed to assess abilities to (i) learn fundamental physical concepts: (ii) understand intuitive physics amidst noise; (iii) compose learned knowledge for long-sequence planning; and (iv) generalize across varying object quantities. For further details game division and scene designs, refer to Appx. A. Agents are trained on the basic split and evaluated on the remaining generalization splits. Unlike PHYRE, which emphasizes data-driven approaches via extensive stochastic game data, our approach fosters robust generalization physical reasoning through interaction with a select set scenarios.
Basic split. This split focuses on foundational physical concepts like angle, direction, fill, hinder, hole, impulse, pendulum, seesaw, spring, and support, essential for intuitive physics understanding and foundational for more intricate physical dynamics.
Noisy split. Created to assess agents' resilience in physical reasoning against minor perturbations, this split tests the ability to maintain invariance noisy blocks, indicative of genuine under- standing of physical dynamics. It includes games from the basic split, augmented with additional distractive gray blocks.
Compositional split. Evaluating compositional generalization (Fodor et al., 1988; Lake & Baroni, 2018), this split combines known knowledge to solve novel tasks necessitating long-range reasoning. It contains games formed by chaining two basic split games or by merging two high-level concepts from the basic split to craft new one.
Multi-ball split. This split examines the agent's 's capability to manage multiple dynamic events concurrently, involving least two balls. Success achieved only when all balls are dropped into the hole, with dynamics being notably more intricate due to additional interactions among the balls, and action timings in successful solutions being more critical.
To encourage learning efficient policies, we assign a negative reward of -1 per second and 10 for each gray block climinated. Achieving the goal yields a substantial reward of 1000. The final score is the sum of accumulated rewards, promoting strategies that are both swift and action-efficient.
Reward
Implementation I-PHYRE games are implemented using pymunk, rendered in pygame at a x00 600 resolution, and integrated into Gym to streamline testing with advanced RL frameworks. The simulator conveys essential game information, including object positions, sizes, and pertinent indicators. For additional details regarding observation and action space, refer Appx. C.
3.2 PLANNING STRATEGIES
Drawing inspiration from human proficiency in navigating games requiring intricate physical reason- ing, we explore two planning strategies and their combination for formulating interactive physical reasoning problems: planning in advance, planning an-the-fly. and combined strategy. These realized within the frameworks of supervised learning and RL.
Planning in advance While I- PHYRE is fundamentally multi-step, it can be simplified through one-step planning strategy. Here, agents observe the initial stationary scene and predict the timing to eliminate each block. Once initiated, agents adhere to this plan, executing block removals at predetermined moments. Practically, we employ deterministic agents, with their action space aligning with the timings at which blocks are eliminated.
5

---

Published as a conference paper at ICLR 2024
Planning on-the-fly Agents employing on-the-fly planning continuously interact with the envi- romment, deciding actions at each step. This approach aligns with the classic RL framework, where agents, at each time step, propose distribution of actions and select the maximizing total rewards. By considering each observation as the state space, the blocks to be eliminated as the action space, and the physics simulator the state transition function, the problem can be modeled as a standard Markov decision process.
Combined strategy Agents can leverage hybrid of the aforementioned strategies, benefiting from the relative reward density in planning in advance and the adaptability inherent in on-the-fly planning. In this strategy, agents, upon receiving the initial scene, determine the execution time for all actions. wait and execute the earliest action, and update the execution time based on the altered scene until the game's conclusion. The action space is also the timings at which blocks are eliminated.
4 EXPERIMENTS
Through a series of experiments, we assess the capability of current learning agents to acquire interac- tive behaviors and adapt novel situations. We also present human performance benchmark for comparison. Additionally, we delve into the distinctions between various planning strategies within the context of interactive physical reasoning.
4.1 HUMAN BASELINE
Procedure With approved IRB, we recruited total of 46 participants from designated participant pool. Each participant played all interactive physics games, aiming to achieve the highest scores. Prior to the experiments, participants were briefed that:
The objective is to guide the red balls into the hole by removing gray blocks;
Gray and black blocks are stationary, while blue blocks are movable due to gravity impact;
Scores are determined by the number of interventions and total time tene-fwerr actions and quicker completion yield higher scores;
Each game can be attempted maximum of 5 times;
Each game, once started, has 15-second time limit, but pre-start contemplation time is unlimited; Failure is declared if the objective is not met within the time constraint.
The highest achieved by each participant was automatically logged. The average score for each game was computed from the mean of participants' highest scores. To set an upper bound on these intricate tasks, scores achieved by the experimenters were considered the maximum attainable.
Results Tab. 2 outlines the par-
Table 2: Rewards and success rates (SR) of participants and oracle. Each player have five attempts per game.
ticipants' performance relative the oracle. The participants ex- hibit success rate above 80%, demonstrating robust ability to
Player
Metric
Basic
Noisy
Compositional
Multiball
Purticipants
Reward
898.01 92.39
879.23
798.66
800.97 82.83
SR (%)
90.00
82.83
solve the majority of the games by effectively reasoning about in- teractive physical dynamics to attain specific goals. However, is noteworthy that participants often do not achieve solutions that optimal compared to the oracle, quantitatively depicted in Tab. 2. A more detailed analysis the results, available in Appx. J, indicates that participants find it challenging solve certain compositional and noisy games with intricate dynamics, such as activated-pendulun and noisy-angle Additionally, participants encounter difficulties in accurately timing interventions under stringent temporal constraints, as seen in games like impulse-spring and ball-hide....
Oracle
Reward
976.06
974.71
971.20
968.61
SR (%)
100
100
100
4.2 LEARNING AGENTS
We employ the contemporary RL framework to model the dynamic decision-making. Specifically, we explore five model-free deep RL methods: PPO (Schulman et al., 2017), A2C (Mnih et al., 2016), SAC (Haarojj et al., 2018), DDPG (Lillicrap et al., 2016), and DQN (Mnih al., 2015). Additionally, we experiment with model-based RL, offline RL, and supervised learners to establish algorithm baselines. Refer Appxs. D and E for their results.

---

Published as a conference paper at ICLR 2024
Noisy
Compasitional
Multi ball
800
600
Reward
400
200
Random PPO-1
A2C-1
SAC-1
DDPG-I
PPO-O
A2C-O SAC-O DQN-O PPO-C A2C-C
SAC-C
Agent
Figure 2: Performance of various RL agents I-PHYRE. Agents, trained on the hasic split, evaluated on
the remaining three splits in zero-shot fashion. The suffices '-I', '-O', and "-C" denote planning advance, on-the-fly planning, and the combined strategy, respectively. The dashed lines are human results.
Architectures We configure PPO-I, A2C-I, SAC-I, and DDPG-I to follow the planning in advance paradigm (indicated by the suffix -I), as they are all apt for continuous action spaces, representing the timing of each action. Here, they predict the entire sequence of action timings once. During testing, the game engine takes in the action timing sequence, executing each action at the designated time.
Conversely, PPO-O, A2C-O, SAC-O, and DQN-O are structured planning on-the-fly agents (indicated by the suffix -O), interacting with the environment in discrete time steps. Given the current state and action space, DQN-O selects action from the action space, based on the action value determined by a neural network. The other three, being actor-critic bbased agents, follow the same training pipeline as in planning advance but generate single action at each time step. This cycle continues until the goal is achieved or terminal state is reached.
We also devise combined strategy to mimic human planning. Specifically, agents, initially create sequence of action timings based on the initial scene, similar to planning in advance, but adjust this action policy post-execution of the earliest action. For this approach, we utilize PPO-C, A2C-C, and SAC-C algorithms (indicated by the suffix -C).
4.2.1
How DO AGENTS PERFORM ZERO- SHOT GENERALIZATION ON DIFFERENT SPLITS?
Our assessment is centered on generalization to novel scenarios, scrutinizing agents' capabilities on noisy, compositional, and multi-ball splits after their training the basic split, with no further fine-tuning. The generalization to similar tasks is trivial for agents as shown in Appx. F. The results, illustrated in Fig. 2, indicate that compositional and multi-ball splits notably more demanding. They necessitate the ability to chain existing solutions effectively and to manage multiple physical objects with precise action timing for successful task completion.
Current RL agents manifest substantial gaps in generalization compared to humans. Humans display consistent and harmonious performance across all splits, highlighting balanced development of interactivity-relatn
capabilities, as detailed in Tab. 2. Conversely, some agents, such as DDPG-I, struggle to outperform even random agents. The data reveals pronounced proficiency of RL agents in the noisy split, where their performance shows a correlation with their achievements in the basic split. However, this correlation diminishes in the compositional and multi-ball splits, where the inherent complexities of these tasks impact performance negatively. See Appx. B for analysis.
Distinct performance patterns are observed between humans and artificial agents across general- ization splits. Humans maintain uniform performance, reflecting well-coordinated development of interactivityyrelatee abilities. contrast, artificial agents excel mainly the noisy split, with no clear correlation with human performance, possibly due to task-specific variations and inherent complexities. The variability in the compositional and multi-ball ball splits underscores RL agents' unmet capabilities in interactive physical reasoning, detailed in Appx.
4.2.2 How DO PLANNING STRATEGIES DIFFER?
Distinct characteristics of the three planning strategies significantly impact agent performance and generalization. As depicted in Fig. agents using planning in advance converge swiftly,
7

---

Published as a conference paper at ICLR 2024
Inadvance
Combine
Combine
PPO Inadvance
PPO Onthefly
PPO Combine
A2C Onthefly
800
M
800
Reward
400
200
0
400k
700k
Step
Figure 3: The training curves of different agents on basic split. Training steps vary among agents.
stably, and effectively compared to on-the-fly fly planners, regardless of the specific agents used. This is likely due to the concise representation of action space. although it limits real-time time decision- capabilities.
In contrast, the oscillations in the training curves of on-the-fly planners may stem from the difficulties in learning sparse action distribution over prolonged periods, marked by few eliminations amidst nu- merous no-ops. The combined strategy, while converging slower than planning advance, eventually reaches similar reward levels.
closer look Fig. reveals that all strategies generalize similarly across different splits, with variations mainly in the basic split. The combined strategy, blending the temporal dimensions of planning in advance with updating mechanisms, offers enhanced adaptability and comparable generalization, even for planning on-the-fly agents, which, despite their learning challenges, emerge as viable contenders for interactive physical reasoning tasks.
DISCUSSION
I-PHYRE is designed to assess and advance the development of interactive agents capable of engaging in precise physical interactions through reasoning, planning, and intervention. We introduce three planning strategies and assess various classic learning agents, discussing challenges, limitations, and potential future directions below.
5.1 WHY CURRENT RL AGENTS FAIL ON I-PHYRE?
The findings in Sec. 4.2 reveal substantial disparity between RL agents and humans in performance on I-PHYRE. We explore the potential reasons for this gap, considering both environmental and model design perspectives.
Physics modeling: I-PHYRE incorporates extensive physical rules and concepts. However, ex- isting RL agents, primarily focusing mapping states to actions, lack understanding of the objects' physical properties and the inherent physical commonsense within the scenes. Without comprehensively exploring all possible physical objects and state transitions, achieving profound understanding of the governing physical rules remains clusive. The field has seen limited exploration into abstracting physical knowledge RL agents, and even state-of-the- methods employing GNN struggle with significant enrors in unseen scenarios when predicting future trajectories under physical rules. We posit that physics modeling is pivotal for enhancing agent interaction with physical environments. More details are provided in Appx. I.
Contrasting the one-step nature of benchmarks like PHYRE (Bakhtin et al., 2019), I-PHYRE necessitates multi-step interventions, introducing longer delays in receiving feedback. This demands agents to leam patient interventions in the physical dynamics progressively. The delayed feedback may cause agents to struggle in discerning the efficacy intermediate interventions in solving the I-PHYRE problems.
Multi-step interventions:
8

---

Published as a conference paper at ICLR 2024
Action timing:
The exact timing requirements of I-PHYRE are challenging for RL agents, cessitating the identification and timely elimination of crucial blocks to optimize scores. The predominance of no-ops in action distribution makes current on-the-fly planning methods for RI. agents overly cautious, and the environment's sensitivity to precise action timing introduces additional hurdles that are challenging for current agents surmount. Further analysis of failure sources from action order and timing is discussed in Appx. H.
5.2 LEARNING TO JUDGE, PREDICT, AND ACT WITH OFFLINE DATA
The intuitive interaction with the physical environment can manifest through various learning forms. Beyond the realm of online RL, we delve into several offline learning tasks, allowing agents to leverage a repository of pre-collected data to make informed judgments, predictions, and actions.
Specifically, we scrutinize three learning paradigms: supervised learning, model-based RL learning, and offline RL learning, all operating as offline learners.
Supervised learners: Functioning offline learned discriminators, these learners assess whether a sequence of actions can resolve a task. However, the outcomes, detailed in Appx. D, depict their struggle to navigate new game splits without supplementary data.
Model-based learners: These are offline learned predictors formulating projections of future states. Their efficacy is gauged by the rewards secured through the amalgamation of current and anticipated future states. Yet, as revealed in Appx. E, they don't exhibit notable advancements over their model-free RL counterparts.
Acting as offline learned actors, these learners dictate actions based on the present states. However, the findings in Appx. E illustrate their inability to assimilate interactive principles from the available data.
Offline RL learners:
In conclusion, our empirical investigations underscore the suboptimal performance of agents in offline learning scenarios, highlighting the pivotal role of real-time environments and adaptive exploration in learning interactive physical reasoning with current algorithms. Nonetheless, we conjecture that agents, when endowed with extensive physics modeling, can potentially excel in offline learning scenarios, enabling them to adeptly judge, predict, and act.
5.3 LIMITATIONS AND FUTURE WORK
We now discuss the limitations of our current approach and explore potential avenues for future research aimed at developing enhanced agents capable of interactive physical reasoning.
Optimal planning strategy: While have explored combined strategy incorporating planning in advance and planning on-the-fly, it may not represent the pinnacle of human planning strategies. Future research could delve into optimizing the amalgamation of these strategies model and augment advanced reasoning capabilities effectively.
Integration with large pre-trained models: The rising interest in large pre-trained models within the community is noteworthy, Preliminary studies are conducted in Appx. G, and the fusion of these extensive pre-trained language models with quantitative methodologies could potentially enhance performance I-PHYRE.
3D interactive environments: Our study is centered around interactivity in a 2D physical environ- ment emphasize reasoning. However, developing a more realistic and interactive benchmark in 3D space could further propel the advancement of agents exhibiting human-like physical reasoning.
CONCLUSION
We present Interactive PHysical Reasoning (I-PHYRE) to assess learning methods in interacting with the physical world, focusing intricate dynamics and precise action timing. We introduce three planning strategies and reveal a significant disparity between human capabilities and learning agents through extensive studies and experiments. The limitations and potential future directions highlighted, with the hope that I-PHYRE will inspire advancements understanding the interactive aspects of physical environments.

---

Published as a conference paper at ICLR 2024
The authors would like to thank NVIDIA for their generous support of GPUs and hardware. This work is supported in part by the National Science and Technology Major Project 00222D0114900) the NSFC (62376009), and the Beijing Nova Program.
Acknowledgment
REFERENCES
Kelsey Allen, Kevin A Smith, and Joshua B Tenenbaum. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences (PNAS), 117(47):23302222) 29310, 2020. 1,
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick, Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 1, 2, 4, 8, A4
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation engine of physical scene understanding. Proceedings of the National Academy of Sciences (PNAS), 2013. A7
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch, Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. A4
Zhenfang Chen, Kexin Yi. Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In International Conference on Learning Representations (ICLR), 2022. 4
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through dtysics-informet neural networks: Where we are and what's 's next. Journal of Scientific Computing, 2022. A7
Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, Yi Lin, Su-hua Wang, Renée Baillargeon, and Cheston Tan. benchmark for modeling violation-of-expeee offatioonofeeepeeaation in physical reasoning across event categories. arXiv preprint 1xXi:22/1.08885 2021. A7
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929,, 2020. A4
Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. survey on machine learning approaches for modelling intuitive physics. arXiv preprint arXiv:2202.0648)) 2022. A6
Ren ee Baillargeon. Object permanence in 31 2-and 41 2-month-old infants. Developmental Psychology, 23(5):655-666 664, 1987. 3,
R Epstein, CE Kirshnit, RP Lanza, and LC6700713 Rubin. 'insight' in the pigeon: Antecedents and determinants an intelligent performance. Nature, 308(555))6)1-2,, 1984.
Nima Fazeli, Miquel Oller, Jiajun Wu, Zheng Wu, Joshua B Tenenbaum, and Alberto Rodriguez. See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion. Science Robotics, 3(26):eaav3123, 2019.
Jerry Fodor, Zenon W Pylyshyn, et al. Connectionism and cognitive architecture: critical analysis. Cognition, 28(1-2):3-71, 1988. 5,5
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Lázaro- Gredilla, Christopher Laan, Bhaskara Marthi, Xinghua Lou, Zhaoshi Meng, Yi Liu, Huayan Wang, al. A generative vision model that trains with high data efficiency and breaks text- bext-based captchas. Science, (336)))..,,,2261 2017. 3
10

---

Published as a conference paper at ICLR 2024
Rohit Girdhar, Laura Gustafson, Aaron Adcock. and Laurens der Maaten. Forward prediction for physical reasoning. arXiv preprint arXiv:2006.10734, 2020. 4, A4
Ulf Grenander. General pattern theory: A mathematical study of regular structures Oxford mathemat- ical monographs. Oxford University Press: Clarendon, 1993.
Romana Gruber, Martina Schiestl, Markus Boeckle, Anna Frohnwieser, Rachael Miller, Russell D Gray, Nicola S Clayton, and Alex H Taylor. New caledonian crows use mental representations to solve metatool problems. Current Biology, 29(4):686-69) 692, 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018. A4
Tuomas Haamoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning (ICMI.), 2018. 6
Muzhi Han, Zeyu Zhang, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Scene reconstruction with functional objects for robot autonomy. International Journal of Computer Vision (IJCV), 2022. 4
Susan Hespos and Renée Baillargeon. Infants' knowledge about occlusion and containment events: A surprising discrepancy. Psychological Science, 12(2):141-147, 2001. 2,3
Ayush Jain, Andrew Szot, and Joseph Lim. Generalization to new actions in reinforcement learning. In International Conference on Machine Learning (ICML), 2020. 4
Ziyuan Jiao, Yida Niu, Zeyu Zhang, Song-Chun Zhu, Yixin Zhu, and Hangxin Liu. Planning sequential tasks on contact graph. In International Conference on Intelligent Robots and Systems (IROS), 2022. 4
George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215 289, 2018. 4
James Kubricht, Chenfanfu Jiang. Yixin Zhu, Song-Chun Zhu, Demetri Terzopoulos, and Hongjing Lu. Probabilistic simulation predicts human performance on viscous fluid-pouring problem. In Annual Meeting of the Cognitive Science Society (CogSci), 2016. A7
James R Kubricht, Keith Holyoak, and Hongjing Lu. Intuitive physics: Current research and controversies. Trends in cognitive sciences, 2017. A7
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of ceruuence-tt-sequeeee recurrent networks. In International Conference on Machine Learning (ICML), 2018. 5,5
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332-266 2015. 3
Steven M Valll. Planning algorithmss. Cambridge university press, 2006. 4
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example.
In International Conference on Machine Learning (ICML), 2016. 1,4
Shiqian Li, Kewen Wu, Chi Zhang, and Yixin Zhu. On the learning mechanisms in physical reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2022. 4, A4
Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference Learning Representations (ICLR), 2016. 6
Ethan Ludwin-Pesn Peery, Neil R Bramley, Ernest Davis, and Todd Gureckis. Limits on simulation approaches in intuitive physics. Cognitive Psychology, 2021. A7
11

---

Published as a conference paper at ICLR 2024
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529 -3353,,, 2015. 4,
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016.
Luis S Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physies learning in a deep-learning model inspired by developmental psychology. Nature Human Behaviour, 6(9):
1257-1267, 2022. 4, A7
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik, Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations (ICLR), 2021. 4, A4
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 2019. A7
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint a7Xiv:1707.06347, 2017. 6
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):844 489, 2016.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental Science, 10(1):89-96. 2007. 1
Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge.
Psychological Review, (9(4))605, 1992. 2, 3
Emma C Tecwyn, Susannah KS Thorpe, and Jackie Chappell. A novel test of planning ability: Great apes can plan step-by-step but not in advance of action. Behavioural Processes, 100:174-184, 2013.
Marc Toussaint, Jung-Su Ha, and Danny Driess. Describing physics for physical reasoning: Force- based sequential manipulation planning. IEEE Robotics and Automation Letters (RA-L), 5(4): 6209-6216 6216, 2020. 4
Mare A Toussaint, Kelsey Rebecca Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems (RSS), 2018.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019. 4
Zi Wang, Caclan Reed Garrett, Leslie Pack Kaelbling, and Tomás Lozano- Learning compo- sitional models of robot skills for task and motion planning. International Journal of Robotics Research (IJRR), 40(6-7):866-888 866-99444 2021. 4
Luca Weihs, Amanda Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in ai. Transactions on Machine Learning Research, 2022. A7
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Halma: Humanlike abstraction learning meets affordance in rapid problem solving. arXiv preprint arXiw:2102.11344,
2021. 5
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference Learning Representations (ICLR), 2020. 4
12

---

Published as a conference paper at ICLR 2024
Zeyu Zhang, Ziyuan Jiao, Weiqi Wang, Yixin Zhu, Song- Chgg- Zhu, and Hangxin Liu. Understanding physical effects for effective tool-use. IEEE Robotics and Automation Letters (RA-L), 7(4): 9469-9476, 2022. 4
Song-Chun Zhu, David Mumford, al. A stochastic grammar of images. Foundations and Trends in Computer Graphics and Vision, 2(4):259-362, 2007.
Yixin Zhu, Yibiao Zhao, and Song-Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 4
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 2020. A6
13

---

Published as a conference paper at ICLR 2024
A
GAMES
We present the initial scenes of all the 40 games in Figs. A2 A5. The games vary in different positional setups. We keep invariant the dynamic properties like density and friction to emphasize the interactive aspects of the task.
B GAME DIFFICULTY TERMS OF SAMPLING
To evaluate the game difficulty I-PHYRE, we measure the number of iterations required to gather 50 successful action sequences that do not repeat for each game. As approximate estimation, the more iterations random sampling needs, the more difficult it is to solve; see Fig. Al for results. The compositional games show significant difficulty compared with those in the other three splits, especially the seesaw angle and activated pendulum. These games have stringent requirement on execution timing: Players and agents may miss the perfect timing easily. Another interesting discovery is that the games related to the seesaw show higher difficulty regardless of the split. The analysis of game difficulty demonstrates a clear alignment with human performance, as humans tend to achieve lower scores in games that are more challenging sample, See Tab. A4 for human scores in detail.
1400
1200
Ileralions
400
200
0
support
hinder
direcion
hole
setsaw
angle
Imiulee
perdulum
spitng
support hinder
suppor dircctinn
supportin
hinder fil
impulta spring
impulte pendiluum
sapu
aopurt copu
sopu
pou (opu
liz Aopu
Resesae Asppu
ofaa Aspua
mulli aur auru
multca hinder
multi out redinect
mulli oot hole
multi bal fil
muiti ball lnver
munti-oo pendutum mad muiti bal angle
hole
flck
Game
Figure A1: The iteration numbers required to generate 50 different successful action sequences.
(a) angle
(b) direction
(c) fill
(d) hinder
(e) hole
(f) impulse
(g) pendulum
(h) seesaw
(i) spring
(i) support
Figure A2: The initial scenes of basic games.
A1

---

Published as a conference paper at ICLR 2024
(a) noisy fill
(b) noisy hole
(c) noisy angle
(d) noisy hinder
(c) noisy seesaw
(f) noisy spring
(g) noisy impulse
(h) noisy support
(i) noisy pendulum
(j) noisy direction
Figure A3: The initial scenes of noisy games.
(a) hinder fill
(b) seesaw angle
(c) spring flick
(d) support hole
(e) impulse spring
(f) step hole
(g) support hinder
(h) impulse pendulum (i) support direction (j) activated pendulum
Figure A4: The initial scenes of compositional games.
(a) multi-ball spring (b) multi-ball lever flick
(c) multi-ball stack
(d) multi-ball redirect
(e) multi-ball
( multi-ball hole
(g) multi-ball angle
(h) multi-ball hinder (i) multi-ball spring (i) multi-ball pendu- lum
Figure A5: The initial scenes of multi-ball games.
A2

---

Published as a conference paper at ICLR 2024
C TRAINING DETAILS OF MODEL-FREE LEARNERS
We run all our experiments on RTX 3090 GPUs. The simulator produces the next state, reward, and termination indicator from the action per time step.
Architectures We adopt three distinct strategies in training: planning in advance, planning on-the- fly, and the combined strategy.
In the planning in advance strategy, the combination of initial observation of the game and the entire action space serves as the model's input, while the output consists of continuous-valued vector whose dimension equals to the maximum number of actions (6 in our case). Each vector element represents the normalized action timing. We implement model-free reinforcement learning algorithms, namely Proximal Policy Optimization (PPO-I), Advantage Actor-Crite (A2C-I), Soft Actor Critic (SAC-I), and Deep Deterministic Policy Gradient (DDPG-I), to generate continuous action values in accordance with this setup.
In the planning on-the-fly strategy, at each time step, the model's input comprises the current observation combined with the entire action space, while the output consists of the probability for each possible action, including no action. The action with the highest probability is executed during inference time at each step. Following this approach, we implemented model-free reinforcement learning algorithms Proximal Policy Optimization (PPO-O), Advantage Actor-Criit (A2C-O), Soft Actor-Crite (SAC-O), and Deep Q-Network (DQN-O).
For the combined strategy, the model's input is the fusion of the current game observation and the entire action space, while the output is the same that of the planning in advance strategy. This single-step procedure is analogous to the planning in advance strategy; however, after executing the first action, the entire action distribution is updated based on the current observation. Employing this framework, we implemented model-free reinforcement learning algorithms Proximal Policy Optimization (PPO-C), Advantage Actor-Critic (A2C-C), and Soft Actor-Critic (SAC-C).
The policy architecture of model-free learners is MLP with two hidden layers of size 256 and the activation function is tanh.
Learning The observation for a scene is processed in the symbolic space, represented as a 12 x 9 matrix. Each row denotes one object's 's features in the scene. We use the symbolic representation of objects instead of visual input since symbolic representation consists of all the necessary components do reasoning and visual information does not contribute any additional useful information for this particular task but introduces uncertainty and noise into the planning process.
The object feature vector consists of the essential components for interactive physical reasoning:
Object Position: The spatial coordinates of the objects with four scalars representing the two endpoints for bars or two duplicate centers for balls.
Object Size: The radius of the object.
Eliminable Indicator: An indication of whether the object can be eliminated in the given context.
Fixed Object Indicator: The identification of whether the object is stationary and not be moved due to gravity.
Joint Indicator: An indicator of whether the object connected to joint.
Spring Indicator: An indicator of whether the object is connected to a spring.
The action space is the concatenation of the positions of the objects that can be removed from the scene, padded to the maximum number of 6. Since different games have different action spaces, we concatenate the action space with the observation space as input to enable generalizable reasoning with single agent.
The object features and action space used in the other learning paradigms as well.
A2C-I, A2C-C, and SAC-C are trained with learning rate of 1 x 10-5, All other models are trained with learning rate of 10-6, 6, SAC-I is trained for 57k steps. SAC-O and SAC-C are trained for 80k steps. A2C-I are trained for 426k steps. Other models are trained for 800k steps.
Results Please refer to Sec. 4.2 for analysis. The detailed rewards are listed in Appx.
A3

---

Published as a conference paper at ICLR 2024
D SUPERVISED LEARNERS
commonly used metric in evaluating physical reasoning (Qi et al., 2021; Bakhtin et al., 2019; Girdhar et al., 2020; Li et al., 2022) is the accuracy of classifier model to correctly predict the outcome of an action. In this part, we follow the same protocol: whether a supervised classifier can predict the outcome of an action sequence based on the initial scene.
Architectures We formulate the problem as a binary classification task, wherein a model predicts whether an action sequence will succeed. We consider three different general architectures: Global Fusion, Object Fusion, and Vision Fusion. Of note, this paradigm falls into the category of planning in advance. Each model takes as input the action sequence and the initial scene configuration and outputs the success probability. The Global Fusion model embeds the entire action sequence and all object states and fuses them together using multiple MLPs. The Object Fusion model embeds each action and symbolized object independently and fuses them, both using MLPs. The Vision Fusion model is similar to the Global Fusion model except that it takes the pre-trained ViT (Dosovitskiy et al., 2020) features of the initial scenes as extra inputs.
Learning We generate 50 successful and 50 failed action sequences for each game by random sampling. The generation iteration is regarded To simplify the action space, we discretize 15 seconds into 150 time steps. Models are trained on the basic split and are tested on the other three splits. The object features are the same the ones in Appx. C. The supervised agents are trained for 200 epochs with a batch size of 16, with the learning rate annealed from 1 x 10 3
approximate estimate of the difficulty of games.
10-6 using a cosine scheduler.
Results Tab. A1 tabulates the performance of each supervised learning model, measured by mean accuracy across the test games. Of note, a random classifier should reach about 50% accuracy due to an equal number of positive and negative samples. However, all supervised learning agents fail to show notably improved performance compared with random guess. Among the models, the Object Fusion model shows the best generalization compared to others due to fine-grain embeddings for actions and object states. Adding visual features from a pre-trained ViT does not necessarily improve the overall performance in the Vision Fusion model, though we do find enhanced accuracy when generalized to multiple balls.
Table A1: Performance of different supervised learning models on I-PHYRE measured by mean accuracy (%) ). Models are trained the basic split.
Agent
Bas.
Noi.
Comp.
Mul.
Global Fusion
87.5
59.8
57.0
56.7
Object Fusion
71.4
60.1
60.2
54.1
Vision Fusion
86.4
57.5
55.8
59.5
These experimental results demonstrate that training supervised learning models on naively sampled data cannot endow agents with interactive physical reasoning ability.
E MODEL-BASED AND OFFLINE RL LEARNERS
We apply the model-based World Model (Ha & Schmidhuber, 2018) and the offline Decision Transformer (DT) (Chen et al., 2021) to the I IPHHYRE.
Architectures In the model-based method, we pre-train an MDRNN to predict the next state given the current state and action. The predicted states are concatenated with the current states as guidance for learning policies. We use PPO. SAC, and A2C as controllers. In the offline method, we use GPT-2 as the backbone to learn the mapping from states to actions autoregressively.
Learning We used 50 successful and 50 failed actions per game to train the model-based and offline models. Models are trained on the basic split and tested on the other three splits. For the model-based learner, the prediction module of MDRNN is trained for 50 epochs with a learning rate of x 10-3 batch size of 128, and sequence length of 64. The prediction module is then fixed and served as an additional part of the observation. The offline RL learner is trained for 100 epochs with a batch size of 128, with the learning rate annealed from 3 3 to 1 10 using cosine scheduler.
Results As shown in Fig. A6, the results indicate that the model- based method doesn't 't improve the performance of PPO, SAC, and A2C, potentially due to the fact that RNN cannot learn an appropriate
A4

---

Published as a conference paper at ICLR 2024
physical dynamics model to accurately predict the next state. The offline method is also unsatisfactory; the Decision Transformer planning on-the-fly learns a conservative strategy that takes no action at all. Although the performance may improve by increasing the data size, argue that delicate modeling of physics is essential to learning better dynamics and emerging interactive physical reasoning.
Basic
Noisy
Compasitional
Multi ball
800
600
Reward
400
200
MB-PPO-I
MB-SAC-1
MB-A2C-I
DT-1
MB-PPO-O
MB-SAC-O
MB-A2C-O
DT-O
DT-C
Agent
Figure A6: Performance of model-based and offline RL learners.
F WITHIN-TEMPPLATE GENERALIZATION
To the generalization capabilities of current intelligent agents in games with similar structures, developed an unseen basic split consisting an additional 10 games. These games resemble the basic ones but differ object positions and angles. As with the previous setting, we evaluate A2C-I, PPO-I, A2C-C, PPO-C, A2C-O, and PPO-O that are trained on the basic split. The agents' performance in these modified games is detailed in Tab. A2. The performance nearly matches the basic split but exceeds the performance of the noisy, compositional, and multi-bai ball splits. These results indicate that current agents are adept at generalizing similar tasks but struggle with generalization across different game templates. Thus, we focus specifically on the challenge of generalizing to entirely new tasks in I-PHYRE.
Table A2: Performance of agents on 10 similar tasks to basic games, measured by average rewards.
Split
Random
A2C-I
PPO-I
A2C-C
PPO-C
A2C-O
PPO-O
Basic
360.33
862.47 862.25
861.56
765.26
862.78
461.2
663.91
Unseen basic
360.17
763.13
662.72
660.51
561.75
662.12
G LARGE LANGUAGE MODEL
In this interactive physical reasoning environment, GPT-4 is tasked not only to do physical reasoning but also to plan and take actions at specific times. This is quite challenge for GPT-4 with only the initial states as input configuration. We prompt GPT-4 with detailed game rules and object features. The results show that GPT-4 can successfully finish some preliminary tasks like support and noisy support but fail other tasks. The average rewards of different folds are in Tab. A3.
Table A3: Performance of GPT-4 I-PHYRE measured average rewards.
Agent
Bas.
Noi.
Comp.
Mul.
GPT-4
75.17
64.17
-29.10
77.49
The preliminary results suggest that the large language model cannot perform well in this interactive reasoning task, although they may be good at understanding physical concepts and utilizing physical heuristics. The interactive physical reasoning scenarios challenge GPT-4 reason both spatial information and precise action timing. Studies need to combine quantitative methods to further extend its capability to do physical reasoning, especially in terms of interactive reasoning.
A5

---

Published as a conference paper at ICLR 2024
H ANALYSIS ON FAILURE CASES
To gain more insight into the interactivity in physical reasoning, we delve into failure cases specifically related action order and timing. We assume that, to solve the game an oracle way, one should first decide the correct action order and then the precise action timing. The situations when actions in the wrong order can still solve the game are beyond our discussion. Thus, the failure may come from two sources: (i) wrong action order and (ii) wrong action timing with the correct order. We want to explore to what extent agents failed due to those two reasons respectively. We benchmark against established baselines: PPO, A2C, and SAC. We count three terms:
Success Number (SN): The number of scenes in split where agents could solve puzzles.
Right Order Number (RON): The number of scenes in split where agents aligned with the Oracle in terms of action order.
Success from Right Order Number (SRON): The number of scenes that the agent solved and also matched the Oracle's 's action order.
With total number of games Totol, the percentage in failure cases of wrong action timing with the correct order, denoted as P(T) F), can be calculated as:
RON SRON
P(T F)
(A1)
Total - SN
The percentage in failure cases with wrong action order, denoted as P(O) F), can be calculated as: P(O F) = = P(T F).
(A2)
The percentage of two failure sources in PPO, A2C, and SAC are shown in Fig. A7.
Failure Sources
Action Timing
Action Order
1.0
100.00%
100.00%
100.00%
100.00%
0.8
80.00%
80.00%
Percentage
68.00%
56.25%
0.4
64.445
78
0.2
14.20%
Comp. PPO
Comp.
Comp.
A2C
SAC
Figure A7: The percentage two failure sources I-PHYRE.
Our findings suggest that, of the cases solved, many of them executed with the optimal action order. For cases not solved, about half of them are due to correct action order but wrong action timing average (56.25% 47.62% and 68.00%), suggesting that both action order and action timing are important. When examining basic games, we find that learning action order is much easier than learning action timing since all of the failures come from wrong action timing. Additionally, for compositional games, the main challenge lies in executing actions in a more reasonable order, while in noisy games, the primary hurdle is enhancing the accuracy of action timing. In summary, our observations indicate that while action order significantly influences puzzle-solving success, the nuances of action timing prove particularly challenging for RL agents to get better performance. We hope this analysis can provide some of the future directions for model design from the temporal aspect.
I DETAILED DISCUSSION ABOUT PHYSICS MODELING
Physics modeling serves as a crucial tool for comprehending and engaging with the physical realm, and its integration is critical in the progression of machine learning agents (Zhu al., 2020; Duan
A6

---

Published as a conference paper at ICLR 2024
et al., 2022). The primary branches of physics modeling include physics- based simulation, physics- informed methodologies, and intuitive commonsense modeling.
The implementation of physics modeling (Battaglia ct al., 2013; Kubricht et al., 2016). Due to the recent developments in computational capabilities and efficiency, it is possible accurately simulate array of physical phenomena in real time. These physics engines offer a consistent and comprehensive platform where agents can employ these principles to foresee future states. Despite its practicality, physies-based simulation presents its own challenges, especially regarding the computational demands for precisely depicting complex physical systems and grasping latent variables in partially observed environments (Ludwin- Peery et al., 2021). Moreover, it necessitates expert knowledge to construct, and it's not capable of learning from experience.
be achieved through physics-based simulations
Another method is incorporating explicit physics constraints directly into the agent by employing ohysics-informed neural networks (Raissi al., 2019; Cuomo et al., 2022). Physical laws and principles are used as constraints or guides in the learning process. It can help in building more robust models that are less prone to overfitting and ensure that the model's s predictions adhere to established physical laws, making them more interpretable and reliable. This approach helps mitigate accumulated estimation errors. However, it might not generalize well to unseen scenarios due to its reliance on predefined physical laws.
Supplementing learning of physical dynamics with commonsense reasoning is another method. Agents learn not just how object states evolve, but also the commonsense intuitive understanding of how these objects behave under certain circumstances (Piloto et al., 2022; Kubricht et al., 2017; Dasgupta et al., 2021; Weihs al., 2022). This involves leveraging the vast amount of implicit knowledge humans possess about the world and encoding it into agents and could potentially lead to more robust and generalizable models.
However, each method has its challenges and limitations, and further research is required to develop more efficient and effective ways of incorporating physics modeling into agents.
COMPLETE RESULTS OF HUMANS AND RL AGENTS
We show the detailed game rewards from RL agents, human participants, and a random agent. For straightforward comparison, we average the rewards in 40 games; see Tab. A4 for details.
A7

---

Published as a conference paper at ICLR 2024
Table A4: All results of humans and different RL agents I-PHYRE benchmark, measured by rewards gameplay.
Game
Human
Random
DDPG-1
DQN-D
A2C-1
ACC-D
A2C-C
PP0-1
PPO-O
PPO-C
SAC-1
SACO
SAC-C
913.96
977.6
968
971.3
971.6
970.5
914.3
913.6
977.6
973
969.5
977.6
975.9
hinder
911.57
-45
-25
972.5
962.5
962.5
962.5
962.5
962.5
962.5
962.5
-45
972.5
direction
971.89
953.3
.15
947.3
958.
963.3
959.7
949
953.3
945.9
963.3
953.1
970
hole
966 06.08
-55
955
953.5
950
-55
-55
949.7
962.5
940
959.6
953.9
966
till
970.89
45
957.1
35
958.4
953.6
45
969.
436.36
35
35
35
25
35
augle
9529
impuls
pendulum
spring
mony suppot
977.09
947.9
mny
950.5
951.9
942.5
953.5
-75
938.6
940
nony hole
906.43
955.4
55
950.3
946
909.5
941.3
982.5
951.5
mony Jill
971.47
-55
45
-15
-45
-53
-55
-45
-55
-35
-55
-55.
neney seesow
455.68
-45
-25
-45
-45
-45
45
-45
-45
455
-15
mmny_angle Langle
565.84
-75
-55
-25
-75
-75
911
935.3
-75
D41.3
-75
-75
nonry impulse
968.36
45
35
35
956.7
43
987
956.2
962.3
956.9
45
35
45
many pendulum
972.59
-45
-15
-45
-45
-55
9614
-5
445
.45
45.1
nony
974.47
-45
.35
953.1
9608
-43
-5
9619
96066
963.3
966.1
-55
984.3
support hinder
961.54
-55
-25
946
-55
9468
-55
945.9
-35
supput direction
963.07
55
suppnt.ble hole
969.51
hinder_fil
954.2
impulse spring
inpulse pendalom
975.79
966.3
966
9734
9662
activated pendilum
spring_lickk
915.8
966
-45
958.6
963.9
975
seeuaw angle
409.22
-45
15
43
45
48
muli ball stock
612.59
45
909.9 9
95R.5
957.3
45.L
mult hull hinder
487.67
-45
-25
-45
-55
-35
-35
.45
mukkhalll Leeinett
919.63
964.7
-35
-15
961.
962.4
961.2
-45
964
962.4
955.5
964.5
-35
multibel Jole
673.51
45
-55
-45
.55
.55
.55
-35
45
45
mult ball fill
957.84
65
43
45
936.8
55.1
65
93811
45
65
65
multi Jull Jever
980.33
969.7
9908
9008
968.9
960.9
970.8
969.1
074.5
U70.7
990.8
976.9
9808
multi Langle
927.3
-55
-45
-55
-55
-55
-55
-55
969.7
.5
multi ball
896.18
45.1
624.17
average
528.10
377.34
success
27.50%
42.50%
A8


Reviews:
[{"review_id": "WJMxD7W7FR", "reviewer": "Reviewer_BYPY", "summary": "This paper introduces I-PHYRE, a benchmark for intuitive physical reasoning capabilities in decision-making agents. It consists of four different block-removal games and benchmarks three planning strategies against them, implemented with both supervised and reinforcement learning. I-PHYRE's design centers on three principles: physical reasoning, multi-step planning, and in-situ intervention. \n\nThe four games are \"basic\" (teaching basic principles of physics), \"noisy\" (minor perturbations), \"compositional\" (combining various structures to require multi-step reasoning), and \"multi-ball\" (multiple dynamic events occurring concurrently, motivating carefully timed in-situ intervention). The planning strategies are \"planning-in-advance\" (generating an entire plan with timings based only on the initial state), \"planning-on-the-fly\" (generating the next action at each timestep given the observation - standard RL-type setup), and \"combined\" (generating the entire plan, then updating it after executing the first action). \n\nExperiments include results from model-free deep RL agents as well as some other baselines in supplementary on the games. Everything is compared to a human baseline. The paper specifically discusses how agents perform when generalizing (without further training) from the basic split to other splits, as well as how the three training strategies differ.\n\nFinally, the paper gives analysis of sources of difficulty in I-PHYRE, performance by offline algorithms, and limitations/future work.", "questions": "- In a previous version of the paper, failure on I-PHYRE was ascribed to sparse action requirements and delayed reward. Now, those concepts are being pitched as being inherent to multi-step reasoning (delayed reward) and action timing (sparse action requirements), but it's not true that failing for those reasons means that if the agents were better at handling them, they would robustly learn to handle multi-step reasoning problems and in-situ intervention problems. Could you flesh this argument out more? \n- What exactly is the nature of the \"combined\" strategy, and could you say more on why it's effective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "#### Quality\n- Overall, a solid paper. Intuitively, if an agent did well on I-PHYRE, I would believe that it had robust intuitive physics capabilities in certain domains, which is what a benchmark should convince me of.\n- Appendix G is very valuable. It does rest on the assumption that all failures are either of bad order or bad timing, which eliminates more basic failures, but since it is a simple simulator, and since Appendix G clearly shows that all the failures in Basic are timing-based (i.e. all the failed plans have correct order), I am convinced that the Basic split teaches nontrivial principles and the other splits \n\n#### Clarity\n- Very well-written! Clear language\n- The text organization is really useful, especially in the results section. The subsections make sense and each paragraph follows a claim-warrant structure. In terms of readability, papers often fall apart in the results section; this one does not. \n\n#### Originality\nI-PHYRE seems to have an original design. However, it's not the first interactive intuitive physics benchmark. The paper should compare to [1] and [2], though I do think it serves different, valuable purposes.\n\n#### Significance\nThe tasks in I-PHYRE are distinct from other related work and test compelling aspects of intuitive physics reasoning. So, I think if the paper can prove its claims, it is significant. \n\n[1] Physion: Evaluating Physical Prediction from Vision in Humans and Machines. Bear, D. M. et al. arxiv preprint: arXiv:2106.08261. 2021. \n[2] Jain, A. et al. \"Generalization to new actions in reinforcement learning.\" ICML 2020.", "weaknesses": "#### Quality\n- The paper says that RL agents perform well on the noisy split, sort of justified by the fact that their noisy results \"correlate\" with their basic results. But that doesn't follow - the noisy results are clearly of lower magnitude even if they follow the same trends (across what factor of variation?), and we aren't given a correlation statistic to warrant this claim. The same applies to \"correlation diminishes in the compositional and multi-ball splits... inherent complexities of these tasks impact performance negatively.\" The correlation claim is similarly 1) nonobvious from looking at Fig 2, even if it does sort of look true, 2) not backed up by a number, 3) not clear why it matters - even if the performance on these two were correlated with perf on basic across whatever factor of variation, but lower in magnitude, I would accept that they are harder (and I of course do, just from looking at Fig 2). Then making a claim that this difficulty is due to the \"inherent complexities of these tasks\" is, while not unbelievable, strong. I'm not doubtful, just not convinced. \n- The discussion has a section titled \"why do current RL agents fail on I-PHYRE?\", which in my opinion is the most important aspect of a benchmark paper other than conceptual design and grounding in the environment. The paper claims three benefits: physics modeling being hard, multi-step interventions, and action timing - i.e. asserting that its design principles have effectively resulted in challenges for agents. However, this is all prose and little analysis of actual results - it would help to spell out for the reader which quantitative result comparison should lead to each conclusion. (Appendix G does a good job of this). \n\n#### Clarity\n- Figure 1 (repeated from a previous review I did of this paper, as the figure hasn't changed):\n  - Colors are hard to follow, maybe better to annotate split on box\n  - Since the boxes are much larger than the arrows, sort of look like two columns, and generally don't look like flowchart elements, it's confusing that the top-left box isn't the best one to start reading with. Bigger arrows and/or numbering would help.\n  - \"Wrong order leads to no overlap elimination timing for two balls\" - I don't understand this\n  - Compositional solution isn't that clear - annotation of key occurrences in each step might help\n- \"Combined\" planning strategy needs better explanation (a step-by-step could help) - it sounds like the whole plan is generated based on the initial state, then the first action is executed, then the plan is updated, and that's it. But I'm not totally sure if the entire plan is updated, if it's ever updated again later after subsequent actions, etc.\n- Figure 2 needs to be more organized and readable (in the previous version it was also hard to follow, but still more organized)\n  - Strategies should be put into separate subplots (with the same axes) or otherwise cued, especially since there are different numbers of each, making it hard to eyeball\n  - Baselines (especially the human comparison point) can be horizontal lines crossing the figure\n  - Not every algorithm gets every planning strategy, which doesn't seem to *only* be a function of the nature of the algorithm - so it's confusing to take all of this in just in the form of bars and text\n- Fig 3 would benefit from organization as well - e.g. group strategies by color and differentiate within them by line texture.\n\n#### Originality\n- The paper claims the three planning strategies as a contribution, but they aren't original - they are baselines just like the offline methods tested on I-PHYRE. I agree that the *results* are a contribution, but I would fold that into the third contribution bullet, or at least make it clear that the \"devised\" planning strategies themselves shouldn't be considered original. \n\nNits: \n- Different parts of the paper use different naming conventions for agents - e.g. \"SAC-I\" in one place, \"SAC Inadvance\" in another. Better to use the same thing throughout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yxihvLc5Mt", "reviewer": "Reviewer_yzoQ", "summary": "This paper proposes I-PHYRE, an interactive physical reasoning benchmark. While previous physical reasoning benchmarks mainly focus on reasoning happening in stationary scenes, I-PHYRE tests physical reasoning in an interactive format. This demands the agent to quickly understand the underlying physics, plan over multi-steps, and perform timely manipulation within a scene. The authors formulated I-PHYRE into four game splits to measure the agents' capability to learn and generalize about essential principles of physics, and\nconducted extensive evaluations of existing learning algorithms and human performance. They showed a significant gap between human and learning algorithms, and also analyzed what are the main factors of current learning methods fails.", "questions": "1. I wonder why the authors simply concatenated the predicted states with the current states when performing model based reinforcement learning. A more dominant approach to use would be \n     1. Along with the dynamics model, also train a reward model that predicts the reward given a state.\n     2. Given a reward model and a dynamics model, perform planning (e.g. CEM planning)\n\n2. There are also other recent model based RL methods the authors can try, such as [1]. \n\n\n\n[1] Temporal Difference Learning for Model Predictive Control, International Conference on Machine Learning, 2023, Hansen et al.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and novel design achieving the motivation.\n     - The authors clearly stated I-PHYRE’s contribution compared to other existing benchmarks. \n     - The proposed task of block elimination is well designed for the stated purpose, highlighting interactivity of physical reasoning. \n     - Additionally, dividing the tasks into 4 types for testing different types of generalization is well designed.\n2. Thorough experiments and analysis\n     - The experiments are very thorough, including multiple reinforcement learning baselines, multiple planning strategies, learning from offline data, integration with large language model(LLM), human evaluation, etc.\n     - The authors also tried to analyze what makes learning algorithms difficult to generalize, and came up with three plausible factors.      \n     - Additionally, the authors also quantitatively analyzed how significant action timing influences the performance of the algorithms.", "weaknesses": "1. The visualization of Figure 1. is slightly hard to understand at first glance. Although the authors explain the figure more thoroughly in page 3, it would be better to either separate the figure into several images or add another image that describes the task more briefly. \n2. Although the experiment was thorough, it can be that 40 games is a small number of games to anticipate strong generalization. It would be better if the authors have tested learning methods in a larger scale. For instance, the authors can try more diverse configuration given the same basic physics to model in the scene.\n3. The cited work [1] also contains interactivity in the benchmark, although it mainly tackles generalization to new actions as the authors mentioned.\n\n\n\n[1] Generalization to New Actions in Reinforcement Learning, International Conference on Machine Learning, 2020, Jain et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64QGA8W94P", "reviewer": "Reviewer_p9BE", "summary": "The paper presents a series of intricate tasks centered on physical reasoning and interaction. Additionally, it suggests three distinct approaches for task resolution using supervised or reinforcement learning methods. To conclude, an extensive user study is executed to gauge human proficiency across various learning techniques.", "questions": "Mentioned in weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study introduces a valuable benchmark for assessing model predictions concerning physical outcomes. The benchmark's interactive nature facilitates real-time planning, crucial for real-time physics interactions, particularly when timing plays a key role in the dynamics. Furthermore, it supports multi-step interventions, promoting long-term predictions over brief, single-step action forecasts.", "weaknesses": "1. Please add a table grid to Figure 2 for clearer comparisons.\n2. The author notes the absence of 3D interactive environments as a limitation. However, this is a significant point to address since many high-performing models should ideally transfer seamlessly to robotics. A 3D interactive environment would greatly facilitate this transition. Notably, papers like ComPhy feature 3D environments, as referenced in Table 1 by the author.\n3. I appreciate the author's use of this paper as a benchmark for various RL approaches, highlighting the need for more research in this domain to address physical reasoning tasks. What potential solutions or recommendations does the author suggest for this benchmark?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vc3g1LmXdg", "reviewer": "Reviewer_PPjJ", "summary": "The paper studies the problem of interactive physical reasoning with focus on intuitive physical reasoning (approximate capability to predict physical outcome), multi-step planning (execution of multi-step actions to complete the task), in-situ interventions (necessity for timely object manipulation to succeed). To study this problem authors propose a block elimination task where the task is to ensure all red balls fall into the hole by removing minimum number of blocks. The benchmark consists of 40 unique games segmented into 4 splits: Basic split for training and noisy, compositional and multi-ball splits for testing generalization of physical understanding of agents. Authors also propose 3 different planning strategies: planning in advance, planning on the fly, and combined strategy to solve I-PHYRE using both supervised and reinforcement learned agents. In addition, authors also benchmark human performance on the task and compare it with current learning algorithms.", "questions": "1. How is the combined strategy baseline implemented and what is the action space of this baseline? From the text in main paper it seems like this baseline predicts continuous values at the start of the episode and keeps updating the full vector after each timestep. Is this correct? \n2. Why are the combined strategy baseline trained only for < 100k step in Fig. 3 experiments? \n3. Have authors tried using a on-the-fly version of GPT-4 baseline presented in the appendix? How well does that perform? By on-the-fly version, I mean querying GPT-4 after each timestep to output the next action instead of using just the initial scene\n\nThe primary concern I have is around the small size of training dataset and the challenges around scaling it. I'd appreciate if authors can discuss issues and concerns around that question. I am open to updating my rating if authors answer my questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposed problem statement is interesting and relevant to advance physical understanding ability of learned agents. The proposed block elimination task captures the multi-step planning and necessity for timely actions which are novel aspects of the benchmark\n2. The experimental setup demonstrates performance of 3 strong baselines using different learning paradigms i.e. reinforcement and supervised learning. It also demonstrates effectively that proposed baselines significantly underperform and there’s a lot of room for improvement\n3. The paper benchmarks and establishes a human baseline for interactive physical reasoning\n4. The paper is well written and easy to follow", "weaknesses": "1. It is unclear if the learned agents can generalize to unseen games from the basic split. The benchmark tests generalization to noise, compositionality and multi-ball setup but doesn’t present results on unseen games with properties similar to basic split. It would be good if the authors can add a unseen basic split and present performance of trained agents on unseen basic split. The Noisy split seems like a substitute of basic split but as it is using same games from basic split + some noise I am worried it is not a good representative of generalize to unseen games with similar properties\n2. The training dataset of 20 games for basic split seems too small a dataset to lead to any meaningful generalization to complex splits on which generalization is being tested. Can authors describe why they chose a small dataset for training? Is it possible to procedurally scale the training data? If not, why?\n3. Results presented in fig. 2 and 3 only show comparison of different methods on average reward on evaluation split it doesn’t highlight what is the success rate of these baselines on full task. It would be nice to have a comparison on achieved success on the full task to get a better sense of the results.\n4. Poor performance on compositional and multi-ball splits for RL trained baselines look as expected with MLP policies and no large scale RL training. Can authors elaborate more on whether the I-PHYRE benchmark expects emergence of compositional generalization to such complex splits by training on just the basic split games? If yes, have authors tried using recurrent policies? Eventhough limited but recurrent networks have demonstrated some form of compositional generalization [1] in limited machine translation tasks which require “mix-and-match” strategies to solve the task (which is true for I-PHYRE compositional split) \n5. The proposed task operates in a simple 2D environment which is less realistic setup\n\n[1] Lake, B. M., & Baroni, M. (2018) Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)