BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
Authors: Zeyi Liu
Zhenjia Xu
Shuran Song
Venue: CoRL 2022 Poster (2022)
Keywords: Manipulation
Scene Understanding
Learning Environment

Abstract: We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions. Based on this environment, we introduce a learning framework, BusyBot, which allows an agent to jointly acquire three fundamental capabilities (interaction, reasoning, and planning) in an integrated and self-supervised manner. With the rich sensory feedback provided by BusyBoard, BusyBot first learns a policy to efficiently interact with the environment; then with data collected using the policy, BusyBot reasons the inter-object functional relations through a causal discovery network; and finally by combining the learned interaction policy and relation reasoning skill, the agent is able to perform goal-conditioned manipulation tasks. We evaluate BusyBot in both simulated and real-world environments, and validate its generalizability to unseen objects and relations.

BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
Shuran Song Columbia University, New York, NY, United States https: //tp:///bbubo.o/eee cs. co I lumbiuia a. edu/
Zeyi Liu
Zhenjia Xu
Abstract: We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions. Based this environment, we introduce a learning framework, BusyBot, which allows an agent to jointly acquire three fundamental capabilities (interaction, reasoning, and planning) in an integrated and supe-vsee manner. With the rich sensory feedback provided by BusyBoard, BusyBot first learns policy to efficiently interact with the environ- ment; then with data collected using the policy, BusyBot reasons the inter-object functional relations through causal discovery network; and finally by combining the learned interaction policy and relation reasoning skill, the agent is able to per- form goal-conditioned manipulation tasks. We evaluate BusyBot in both simulated and real-world environments, and validate its generalizability to unseen objects and relations.
Keywords: Manipulation, Learning Environment, Reasoning
1 Introduction
Learning through physical interactions plays a critical role in human cognitive development [1, 2, 3]. For instance, a well-designed toy like the "busyboard" (Fig. 1a) provide effective learning environment for children to develop fundamental manipulation and reasoning skills: the rich and amplified sensory feedback encourages children actively explore and interact; and the observed inter-object functional relations (e.g., switch turns on light) facilitate the development of reasoning and task solving skills.
In this paper. we aim to provide a similar learning environment for embodied artificial agents, the BusyBoard environment, where agents learn to discover the underlying relations of objects through informative interactions and plan for goal-conditioned tasks. While simple at the first glance, this relational environment provides integrated tool for learning and evaluating three critical capabilities of an embodied intelligent system:
The ability to infer action affordances from visual observations knowing where and how to manipulate object effectively change its state. Learning this skill through visual feedback is particularly hard for small disll-displacement objects (e.g., switches), whose appearance changes be subtle under effective actions.
Interact:
Reason: The ability to reason about inter-object functional relations (e.g.. pressing button turns on light). In particular, the agent should learn to infer the relations by observing and predicting future states of the environment, without using the ground-truth relations supervision.
Plan: The ability to use the learned manipulation and reasoning skills in goal-come condiiondt planning tasks, other words, generating sequence of actions transform the environment from a random initial state to a given goal state.
To learn these skills from the environment, we propose the BusyBot framework that acquires the above three capabilities through self-supervised interactions. To acquire the manipulation skill, the algorithm learns a visual affordance model that infers effective action candidates through visual feedback. To reason about inter-object functional relations, the algorithm infers functional scene
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

De (a) Busy Board for Kids
(b) BusyBoard for BusyBot
(c) Real-world
Interaction
ti Ampttfeed Effect
Figure 1: Busy Board Environments inspired by toys for children, an integrated tool for learning and evaluating a robot's capabilities in interaction, reasoning, and planning.
graph and predicts the future states through causal discovery network. Finally, to accomplish goal- conditioned manipulation tasks, the algorithm combines the learned action affordances, inter-object functional relationship, and dynamics to plan its actions with a model predictive control (MPC) framework. In summary, our contributions are two-fold:
We introduce a new learning environment for embodied agents, BusyBoard, which features a diverse set of articulated objects with typical- and small- displacement joints, and rich inter-object functional relations.
We propose BusyBot, an integrated learning framework which allows an embodied agent to acquire interaction, reasoning, and planning skills through self sulf-super learning.
2 Related Work
Simulation environments for robot learning. Simulation environments are crucial for advances in robot learning. However, most of the existing simulated environments are developed for specific tasks or capabilities, such as navigation [4, 5, 6, 7], manipulation [8, 9, 10, I. 12], causal reasoning in 2D [13, 14], high-level task planning [15]. Inspired by human toys, our BusyBoard is an integrated environment that is compact and relevant to real-world applications, where an embodied agent can jointly learn three critical capabilities: interaction, reasoning, and planning.
Learning interaction policy. The ability to interact with a diverse set of objects critical for many robotics tasks. Different methods have been proposed to learn interaction polices through human demonstrations [16, 17, 18] or self-guided explorations [19, 20, 21, 22). However, most prior works have been ignoring set of common but challenging objects: small- mmall-displacement objects. When interacting with these objects (c.g.. switches), the effectiveness of an action often cannot be observed from the object's 's own visual appearance. In this work, we address this challenge by taking advantage of Busy Board, which amplifies action effects through responder objects and enables learning by enriching the supervision signal.
Inferring inter-object functional relations. Perceiving and understanding objects individually is often not sufficient for lot of real-world applications that involve environments with multiple objects, and understanding inter-object relations [23, 24, 25) a erucial skill for efficient planning. In this work, we will tackle the problem of uncovering the inter-object functional relationship. as defined by Li al. [26]. One common approach to induce changes through interventions and iteratively construct functional scene graph [27]. More recently. Graph Neural Networks (GNNs) have been demonstrated to promising for extracting the underlying structural causal model (SCM) [28, 29, 30, 31, 32] and predicting future dynamics from motions [25]. In our work, further demonstrate that GNNs are able to infer inter-object functional relations from changes in visual appearances. We also show that the inferred inter-object functional relationship and scene dynamics can assist action planning for downstream coal-conndtioned manipulation tasks.
Hierarchical Planning in Dynamic Environment. Hierarchical planning has been proven effective for long-horizon planning dynamic environment by decomposing a plan into several high-level sub-goals and low-level actions [33, 34, 35]. Some prior works have proposed using high-level lasec-based planner with learned planning operators (POs) to encode preconditions, actions, and effects [36, 37] in RL setup. However, both works assume abstract action and state space (e.g., grid world). In contrast, our work learns physical actions to manipulate articulated objects from image observations, making inter-object relations implicitly from state transitions, while we explicitly learn a scene graph which helps better generalization to new tasks and environments.
work easier be deployed in the real world. In addition, prior works model

---

Single-link
Singlo-stage
Multi-stage
One-to-one
Visual
Relations
Multi-dincction
One-to-many effects
R
One-to-many objects
Multi-link
Modio
(a) Trigger
(b) Responder
(c) Relations
(d) Boards
Figure 2: BusyBoard Environment is procedurally generated using articulated objects (a.b) with randomly
sampled inter-object functional relation pairs between trigger and responder objects (c). (d) shows example boards and the underlying functional relations.
3 The BusyBoard Board Environment
Trigger and responder objects. As shown in Fig. 2, the BusyBoard is procedurally generated using object URDF models. For trigger objects, we select switch instances from the Partnet-Mobility dataset [12], including mmall-displacement instances (with small displacements upon interaction), multi-direction instances (contain one movable link that can be pushed to multiple directions), and multi-link instances (contain multiple movable links). We use other object categories (e.g., lamp, door, tracktoy) as responder objects, which can either single-stage or multi-stage. A single-stage responder has only two possible states defined by either appearance (e.g.. lamp state (e.g., door open or closed). A multi-stage responder has multiple possible states (e.g., multiple light colors of lamp responder multiple joint positions of tracktoy responder).
off) joint
Relations. We introduce three types of inter-object functional relations:
One-to-one: one trigger controls one single-stage responder.
One-to-many effects: one trigger controls multiple effects on one responder. The trigger is multi-direction object and the responder multi-stage object.
One-to-many objects: one trigger controls multiple responders. The trigger is multi-link object (a switch with multiple buttons), and each link controls one single-stage responder.
The inter-object functional relations enable an important property of BusyBoard: in addition to providing visual feedback on the interacted trigger object (c.g., appearance change on a button after being pressed), BusyBoard also amplifies the effect of an action with responder objects (e.g., a light turns on after the button is pressed). This especially useful for learning manipulation policies for objects with small displacements upon interaction, for which state changes often hard to observe.
Note that in goal-conditioned tasks, for both "one-to-many effects" and "one-to-mant objects" relations, the algorithm not only needs to know the trigger object to interact with, but also the direction and position
action to execute. this paper, we use "one-to-many" to refer to the super-set of both categories. We exclude many-to-one relations to eliminate possible ambiguities.
4 The BusyBot Framework
The goal of BusyBot is to meaningfully interact with the BusyBoard environment ($4.1), infer the inter-object functional relations and dynamics through these interactions ($4.2), and eventually perform con-itondi manipulation using the learned interaction policy, relations, and dynamics. ($4.3). We will discuss each module in detail below.
4.1 Interact: Learning to Interact with Amplified Effects
The goal of the interaction policy is to take top-down depth image o, W generate an action a, at each step !: x(o,) 4. The action is parameterized by an end effector (i.e., a suction-based gripper) position and moving direction a, = (a,e "adirr where af IR is the 3D coordinate and ddir e R ([ladirl = 1) a unit vector in 3D indicating the moving direction of end effector. The moving distance is incrementally assigned until reaching pre-defined limit. The
input and

---

Light
Interaction
Network
Postion
BBAK
Direction Newwrrk
Interaction
Dynamies Network
Pulicy
Binall State
Position Scoces
Direction Scores
Execution
Comting
Action Candidizes
Plani
Reasoning
Inference
Inforttio
Network
Narwork
Newwrk
Graph
Inierattion Suquinos
Relations Curman
Target
Compare belect Action
Figure 3: BusyBot Overview, [Interaction] infers a sequence of actions from visual input to efficiently interact with a given scene. [Reasoning] infers functional scene graph (i.c., inference network) and predicts future states (i.e., dynamies network). [Planning] uses the trained manipulation policy network (learned from multiple boards), relation inference and dynamies network (extracted from the specific board) plan actions for reaching the target state.
policy considered effective if can 1) successfully trigger changes in responder objects, and 2) interact with different objects to explore novel states of the board.
Interaction policy. The interaction policy is modeled by two neural networks: a position network and a direction network, which follows a similar formulation as UMP-Net [20] to jointly learn the action position and direction. The position network takes depth image input and outputs per-pixel position affordance P E [0, i|W*H which indicates the likelihood of an effective interact position. The direction network takes in the depth image and the selected action position (represented as a 2-D) Gaussian distribution centered around the corresponding pixel location of the 3-D action position) and outputs score for each direction candidate r(a,"i) E [0, L]. We uniformly sample directions in SO(3) as direction candidates. Since it often hard to identify the state of small-displacement objects from visual observations, the agent executes both the selected direction and its opposite direction.
Supervision. Unlike UMP-Net [20] that requires object state from simulation to compute reward, Bot uses simple self superviedd reward computed from image difference, which is enabled by the amplified effects in the BusyBoard environments:
{
8 if # W (oij. if TH 1ooj,
- if =d if of #
Fimg (a dir
((o,),,))=
where and of denote RGB image observations before and after the action execution. 8 is threshold specifies the minimum number different pixels. We use binary cross-entropy (BCH) loss between the inferred action score r(adir) E [0 and the ground-truth reward computed from image observations rimg (atir) E 1).
Exploration. At the early stage of training the position and direction inference network, we the epsilon-greedy method to encourage random exploration. Additionally, in order to prevent the model from only selecting the position that has the highest affordance score, apply the Upper Confidence Bound (UCB) Bandit algorithm on the inferred position affordance. Given the per-pixel position
affordance score P(i, j), the updated score is (t. = P(i.j) +cv " where c == 0.5, / is the number of past steps, and N is the numeber of times when the pixel (i. falls in the M x MM 10) window centered around each previously selected pixels.
4.2 Reason: Learning Discover Inter-object Relations by Predicting the Future
The reasoning module takes in RGB image sequences of the agent's 's interactions ($4.1), infers the inter-object relations, and predicts future dynamics, which would guide coal-conditioned planning ($4.3). To accomplish this goal, we adopt and modify the V-CDN model [25].
The inference network is implemented with three Graph Neural Networks (GNNs) extract functional relations as a scene graph. Each object O corresponds to node in the graph, with node input n/ where T the first frames an interaction sequence. Unlike the V-CDN model [25] that uses keypoint locations as node features, BusyBot uses both objects' visual features and locations, which allows the network to represent both motion change (e-g., door opens) and appearance change (e.g., light turns on) of the responders. The first GNN learns spatial node and edge embeddings at each step, which are concatenated with 256-dimensional embeddings of the executed actions !!! output from an MLP layer. The combined embeddings then aggregated over temporal dimension using

---

1-D convolution network and input to the second GNN which predicts probabilistic distribution over edge types e² =eejle 1,j= lij== (index indicates no relation and index indicates has relation). Conditioned on the edge types, the third GNN predicts 32-dimensional edge embeddings e" = (elllely E R 2 J which store history dynamics associated with each edge.
The dynamics network is a Graph Recurrent Network (GRN) that predicts the next state nt x/+1 given the current observation nt, the executed action a', and the edges E = {ee .e") }. from the inferred functional scene graph. The inference and dynamics network are jointly trained under the objective minimize the mean squared error (MSE) between predicted and ground object features. Sg denotes the
inference model parameterized by @, f denotes the dynamics model parameterized by w.
= min
[ Ao(n' (n' dt fo ()) @.V
(12
Data collection. The interaction dataset used in the reasoning module is collected the using learned interaction policy: at each step. positions with affordance scores above a threshold are grouped into clusters using the K-means clustering algorithm (with & being the maximum possible number of movable links on all busyboards), and positions with the highest score in each cluster are selected as position candidates. Conditioned on each position candidate, direction with the highest affordance score will be selected to form the final action candidates, from which a candidate will be randomly chosen to execute. For each board environment, RGB image observations of 30 interaction steps are generated. In addition, to prevent the model from overfitting on board appearances, we ensure that every 20 board environments share the same initial visual appearance but with different inter-object functional relations.
4.3 Plan: Goal-conditioned Manipulation with Relation Predictive Agent
Given an initial and target image of a BusyBoard, the task is to have BusyBot infer 1) which object(s) manipulate; 2) what action(s) execute in order successfully reach the target state.
Using the data collection method as discussed in the reasoning module, the agent infers the action candidates and generate interaction sequence of 30 images, which is imput to the inference network to obtain the functional scene graph. Then we consider three options to plan for goal-conditioned tasks: 1) Relation agent, each step, identify responder that needs to be changed, and find the corresponding trigger based on the functional scene graph. This method is similar to the idea of Li et al. [26]. However, the agent might have trouble handling one-to- one-to-many relations. To solve this issue, we propose 2) Predictive agent that uses the dynamics network from the reasoning module and choose the action that minimizes the L2 distance of the predicted next state and the target state. However, the predictive agent may have difficulty generalizing to novel object instances, due to the difficulty of predicting unseen dynamics. 3) Unlike piror works that only use either predicted dynamics or functional scene graph for planning, our final method Busy Bot combines the relation and predictive agent, where action candidates are first filtered based on the functional scene graph and then selected based on dynamic predictions. More discussions are provided in Sec. 5.
5 Evaluation
We evaluate BusyBot with both simulated (Fig. 4) and real-world busyboards (Fig. 6). In simulation experiments, we set up the following environments: a) Training Board: for training interaction and reasoning module. b) Novel Config: testing board with training object instances but in new configurations, which include new inter-object functional relations, position and orientation of objects, board color and texture; Novel Object: both object instances and board configurations are novel. In total, we generate 10,000 training boards, 2,000 boards with novel configurations, and 2,000 boards with novel object instances. We generate 30 interaction images for each board, where 23 images are reserved for relation inference and the rest for future predictions. As for object instances, we use 41 switches, 10 doors, 5 lamps, and 2 tracktoy objects, split into training testing with ratio: 32/9, 5/5, 3/2, 1/1. The setup for real-world evaluation is described in Sec 5.4
5.1 Interaction Module Evaluation
To evaluate the interaction policy network, we compute the average precision and recall of the inferred actions for the boards, where precision # successful actions / # total proposed actions, and recall # successfully interacted objects/ / # total interactable objects. We compare the following methods:
5

---

(a) Interaction
Steps
Action
Pred
(c) More Configurations
Board
Position
Direction
Novel Config
Novel Object
(b) Reasoning
1-to-1
I-to-many
I-to-many
28
Figure 4: Qualitative Results, (a) action affordances (b) interaction steps and corresponding reasoning results (c) More reasoning results - inferred inter-object functional relations. -:: ground truth.
Oracle (joint state supervision): Interaction policy supervised on joint states. This is considered as the oracle because changes in joint states are directly obtained from simulation.
RGB: A baseline that takes RGB images instead of depth images as input to the model.
w/o responder:
Interaction policy supervised on visual feedback but no responder effects. An ablated version of BusyBot without using UCB for exploration.
w/o exploration:
Results and Analysis. From Tab. 1. we can see that [w/o responder] achieves poor performence since vi- sual feedback of small-displacement objects alone insufficient for learning. In contrast, [BusyBoi] able to achieve comparable performance as the or- acle, which validates our hypothesis that triggered responder effects can assist the model in learning good interaction policy by amplifying the visual feed- back. We also demonstrate the effectiveness of our exploration method by showing the recall of [w/o exploration] drops by more than 45% than that of [BusyBot]. From the [RGB] baseline, find that color observations alone struggle to learn interaction policy for instances such as switches that are all-white. Moreover, the performance of [RGB] drops significantly for novel objects, implying that the model trained with RGB observations overfits on colors and lacks generalizability.
Novel Config Prec
Novel Object Prec
Recall
Recall
Oracle
91.8
82.4
79.5
90.2
RGB
62.3
63.1
9.50
17.0
w/o responder
0,71
0.65
0.24
0.49
w/o exploration
94.2
33.7
81.9
38.6
BusyBot
90.1
80.1
82.6
84.8
Table 1: Performance of Interaction Policy
5.2 Relation Reasoning Module Evaluation
The reasoning module is evaluated by the following metries: 1) Relation inference accuracy, measured by the precision (Edge-P) and recall (Edge-R) of the inferred functional relation pairs. 2) Future state prediction accuracy (Pred-A), measured by the percentage of correct future state predictions. We compare the following alternatives:
w/o inference: An ablated version of the model without the inference network. The dynamics network takes in history interaction data and directly predicts the next state.
w/o exploration: ablated version of our method, where the input of the reasoning network are data collected under an inferior interaction policy.
Training Board Edge-P Edge-R
Novel Config Edge-R
Novel Object Edge-R
Pred-A
Edge-P
Pred-A
Edge-P
Pred-A
w/o inference
79.2
36.2
7.04
w/o exploration
55.6
51.0
89.6
74.6
3.10
14.3
75.1
0.96
11.6
BusyBot 95.8
100
88.1
95.5
99.7
73.8
85.0
99.5
31.0
Table 2: Performance of Reasoning Module. For BusyBot, while the future state prediction accuracy (Pred-A) decreases unseen board appearances (novel config, novel object), the reasoning module is still able to reliably infer the inter-object functional relations (Edge-P, Edge-R) in novel scenarios.
Results and Analysis. Compared [wo inference], we see that without inferring the inter-object relations, the model overfits on the training data and generalizes poorly to novel boards. We also observe that with bad interactions [w/o exploration]. the reasoning model is not able to uncover the relations accurately and make correct future state predictions. In comparison, our model generalizes well to novel board configurations and achieves performance comparable to that of the training board. This demonstrates that good interaction policy helps the agent uncover the correct onter-object

---

functional relations, which then helps the agent to understand scene dynamics, For boards with novel object instances, even though the future state prediction accuracy drops by around 40% than the seen instances (which is expected since the object features are never seen by the dynamies model), the relation inference accuracy is still comparable. The performance on novel boards verifies that the model's 8 ability to infer inter-object functional relationship can transfer to new scenes and objects.
5.3 Goal-conditioned Manipulation
We generate 50 one-to-one tasks and 50 one-to-many tasks for each type of board (training, novel config, novel object). One-to-one tasks con- tain only two-state triggers, and thus only require the algorithm to iden- tify the correct trigger (similar task studied in IFRexplore [26]). One- to-many tasks contain both multi- direction and multi-link triggers that require the agent to not only identify the correct trigger, but also infer the correct action to manipulate the trig- ger (e.g., the correct button position or pushing direction).
Inirial
Goal
Relation
Predictive
Bacy Rot
Novel abject
direcion
1-to-many
gt edge
edge @ gi action @-iinferee action action__jaj tanget state
Figure 5: Goal-conditioned Manipulation. Compared to the pre- dictive agent, the relation agent generalizes better on novel objects, while struggles in handling one-to-many relations. Our method BusyBot combines the advantages of both agents.
Metries & Baselines. We measure object- level success rate on both one-to-one and one- to-many manipulation tasks for each type of board. The success rate defined object level at the end of an interaction sequence with a maximum of steps. Success rate # affectable responders in goal state / # total affectable responders. We compare the three agents discussed in 4.3, along with two learning-basee based agents using behavior cloning (BC) and proximal policy optimization (PPO). More details can be found in supp.
Training
Novel Config 1-to-m
Novel Object
1-to-1
1-to-m
I-to-1
1-to-1
1-to-m
PPO
91.2
88.5
62.3
54.3
59.9
55.7
BC
95.4
94.9
$7.0
51.8
63.8
54.8
Relation
98.3
61.1
93.7
60.0
92.0
62.8
Predictive
97.7
67.5
91.0
67.0
89.0
58.2
BusyBot
98.3
71.0
93.7
69.4
92.3
64.9
Table 3: Goal-conditioned Manipulation Result
Results and Analysis. All agents achieve good performances on one-to-one tasks. This means that both the relation and dynamics learned by the reasoning module can generalize to novel board configurations and objects. The [predictive] agent achieves better performance on one-to-many tasks with seen object instances by leveraging future predictions to select the correct action to apply on the trigger object. In contrast, the [relation] agent can only identify the trigger object but not the exact action (e.g., which link to interact with or which direction to push). On the other hand, the [relation] agent performs slightly better than the [predictive] agent on all one-to-one tasks and boards with novel objects, when the dynamics model sometimes fails to predict the correct next state. This shows that inter-object functional relationship can generalize to scenarios when future predictions are not reliable enough to assist planning. The result of [PPO] and [BC] indicate that RL-based agents fail to generalize to boards with novel configurations or objects, and it is thus critical to learn an explicit representation of inter-object functional relations.
5.4 Real-world Experiments
Setup. We test the trained model on a busyboard in real world with robot interactions (Fig. 6). The board consists of 3 trigger objects (switches) and 3 responder objects (LEDs). Objects outside the effective region are ignored. We manually modified the underlying inter-object functional relations of the board by rewiring the objects. We test with 6 different configurations including 4 one-to-one and 2 one-to-me -ee-to-many configurations. For each configuration, the robot interacts with the board for 30 steps and the rollout is grouped into 6 overlapping and continual sub-sequences, each of which has a length of 25. In total, we generate a real-world dataset of 36 sequences with 108 inter-object functional relation pairs for evaluating the reasoning module.
Results. Fig. 6 (c) shows that the algorithm able to refine inter-object functional relations (reducing additional edges) through interactions. The precision and recall of inferred relations are
7

---

Direction
Stepe ObstaActio
Prediction
(4) More Configurations
Prodiction
Ground Trurh
End-effector
Position
6
one to many
effective region
BusyBoard
(a) Setup
(b) Interaction
(c) Reasoning
Figure 6: Real-world Busyboard. We test the trained models on real-world busyboard with robot interactions
(a), and show that algorithm is able to discover the inter-object relations (c) through interactions (b),
93.9% and 100% % respectively. All inter-object functional relations can be discovered by our model, with only a few additional pairs predicted. The result shows that the relation reasoning ability of the model is transferable to real-world scenarios. More results can be found in supp.
5.5 Application in Simulation Home Environments
To demonstrate the learned skills can be applied beyond the BusyBoard en- vironment, we further test our reason- ing model in 2 kitchen scenes from AI2THOR [15]: a) a stove with mul- tiple controls and b) room with mul- tiple objects. Following the same evaluation protocol, we let the agent interact with the environments for few steps and use the trained rea- soning model to infer the functional scene graphs. We observe that the algorithm achieves similar performance to boards with novel objects (shown in 5.2): while the algorithm cannot perfectly predict the future states of the object (due to out-of-distributin object visual appearance), it is able to infer the correct edges through interactions, without the need of fine-tuning. The result demonstrates the generalization ability of our proposed environment and algorithm to new domains and applications.
Rrem okjic
Figure 7: Application in AI2THOR Home Environments. The figure shows the interactions (in yellow). corresponding state changes, ground truth inter-ohject functional relationships (in green), inferred relationships (in red).
5.6 Limitation and Future Work
While the BusyBoard environment is inspired by toys, it still lacks certain diversity and complexity in real-world toys. For example, real-world toys are often designed with multi-sensory feedback such as sound and tactile, while our environment focuses visual effects only. Several assumptions made by BusyBor could also be relaxed for more general applications. First, BusyBot assumes full observability of objects in single image. Future work may consider using 3D scene representation that integrates multi-view observations to handle larger-scale environments. In addition, the interaction module assumes trigger objects can reach all states through single-step actions. Future work could consider learning a more general manipulation policy [20] to accomodate objects that require sequence of actions manipulate. Finally, the relation reasoning module assumes objects are detected. Though the assumption is valid in our setup, it may not hold for cluttered scenes.
Conclusion
We propose a toy-inspired relational environment, BusyBoard, and a learning framework, BusyBot, for embodied Al agents acquire interaction, reasoning, and planning abilities. Our experiments demonstrate that the rich sensory feedback in BusyBoard helps the agent learn a policy to efficiently interact with the environment; using the data collected under this interaction policy, inter-object functional relations can be inferred through predicting future states; and by combining the ability to interact and reason, the agent is able to perform conl-conditioned manipulation tasks. We verify the effectiveness and generalizability of our method in both simulation and real-world setups.

---

Acknowledgments: The authors would like to thank Huy Ha, Cheng Chi, Samir Gadre, Neil Nie, and Zhanpeng He for their valuable feedback and support. This work was supported part by National Science Foundation under 2143601, 2037101, 2037101, and 2132519, and Microsoft Faculty Fellowship. We would like to thank Google for the URS robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] R. Held and A. Hein. Movement-produced stimulation in the development of visually guided behavior. Journal of comparative and physiological psychology, 56, 1963. doi:10. 1037/ h0040546.
[2] G. E. Roberson, M. Wallace, and J. A. Schirillo. The sensorimotor contingency of multisen- sory localization correlates with the conscious percept of spatial unity. Behavioral and Brain Sciences, 24(5), 2001.
405555055400 IX.
[3] R. Baillargeon. Infants' physical world. Current Directions in Psychological Science, 13(3). 2004. doi: 11.1111 (tt.j..66372221 000.0--2222.200000000
[4] D. Perille, A. Truong, X. Xiao, and P. Stone. Benchmarking metric ground navigation. In 2020 IEEE International Symposium on Safery, Security, and Rescue Robotics (SSRR), pages 116-121. IEEE, 2020.
[5] N. Tsoi, M. Hussein, O. Fugikawa, J. Zhao, and M. Vázquez. An approach to deploy interactive robotic simulators on the web for hri experiments: Results in social robot navigation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7528-7535. IEEE, 2021.
[6] Manolis Savva*, Abhishek Kadian Olcksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodicd AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[7] C. Li, F. Xia, R. Martín-Martin, M. Lingelbach. S. Srivastava, B. Shen, K. E. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In 5th Annual Conference Robot Learning, 2021. URL https: wpwppeeeeeiivveee/ee/ee net www.mrrrmm..mmmmmmmmmmmmooo..mmm ?id= JuGN5jNJRON
[8] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[9] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Roboties and Automation Letters, 5(2):3019-300 2020.
[10] O. Ahmed, F. Träuble, A. Goyal, A. Neitz, M. Wüthrich, Y. Bengio, B. Schölkopf, and S. Bauer. Causalworid: robotic manipulation benchmark for causal structure and transfer learning. In International Conference on Learning Representations, 2021.
[11] Y. Zhu, Wong, A. Mandlekar, and Martin-Martin robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint htt::20009.12999. 2020.
[12] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, Liu, M. Liu, Jiang, Y. Yuan, H. Wang, Yi. A. X. Chang, J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[13] A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.

---

[14] A. Jain. A. Szot, and J. Lim. Generalization to new actions in reinforcement learning. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 199 of Proceedings of Machine Learning Research, pages 4661 -4772. PMLR, 13-18 Jul 2020. URL https:tpps... //ppoceeo/e/eccooo ngs ml www.mmrr.rrsss.. /eress/vvii / a n200. html.
[15] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 2017.
[16] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8709-87188 8719, 2019.
[17] Nagarajan, C. Feichtenhofer, and K. Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688-8697, 2019.
[18] T. Nagarajan, Y. Li, C. Feichtenhofer, and K. Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 163-172, 2020.
[19] K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 68 13 63-3823, 2021.
[20] Z. Xu, H. Zhampeng. and S. Song. Umpnet: Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters, 2022.
[21] B. Eisner, H. Zhang, and D. Held. Flowbot3d: Learning 3d articulation flow manipulate articulated objects. RSS, 2022.
[22] S. Y. Gadre, K. Ehsani, and S. Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 15752-15761, 2021.
[23] Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[24] C. Mitash, A. Boularias, and K. Bekris. Physics- scene-level reasoning for object pose estimation in clutter. The International Journal of Robotics Research, page 0278364919846551, 2019.
[25] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020.
[26] Li,
K. Mo, Y. Yang, H. Zhao, and L. Guibas. IFR-Explore: Learning inter-object functional relationships in 3d indoor scenes. In International Conference on Learning Representations (ICLR), 2022.
[27] S. Nair, Y. Zhu, S. Savarese, and L. Fei-Fei. Causal induction from visual observations for goal directed tasks. arXiv preprint wrXvv:110..0151,, 2019.
[28] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017.
[29] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanzhez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:/806001
2018.
[30] M. Zéčević, D. S. Dhami, P. Veličković, and K. Kersting, Relating graph neural networks to structural causal models. arXiv preprint 210.0421., 2021.
10

---

[3]] W. Lin, H. Lan, and B. Li. Generative causal explanations for graph neural networks. In International Conference on Machine Learning. pages 1666-6679, PMLR, 2021.
[32] N. R. Ke, O. Bilaniuk, A. Goyal, S. Bauer, H. Larochelle, B. Schölkopf, M. C. Mozer, Pal, and Y. Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv:1910.01075.2000 2019.
[33] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical planning for long-horizon manipu- lation with geometrie and symbolic scene graphs. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 6541-6548. IEEE, 2021.
[34] D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.044 04114, 2022.
[35] Li, Tang, M. Tomizuka, and W. Zhan. Hierarchical planning through goal-conditioned offline reinforcement learning. arXiv preprint vrXiiv2205./17900 2022.
[36] A. Agostini, C. Torras, and F. Wörgötter. Efficient interactive decision-makine framework for robotic applications. Artificial Intelligence, 247:187-212, 2017. ISSN 0004-3702. doi:
URL https://t/////...tm //t:::///wwwwwww/ www.eccieicci..... www.rreet.... com/ occ.acee/aatccee///////00021133.
Special Issue on Al and Robotics.
[37] B. Quack, F. Wörgötter, and A. Agostini. Simultaneously learning at different levels of abstraction. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4600-4607, 2015. doi: 10.1 000//ROS..001.73544550


Reviews:
[{"review_id": "8pnyl87T_PL", "reviewer": "Reviewer_9A6a", "summary": "There are 2 main contributions of the paper.\n* The first is the BusyBoard environment which allows generations of many tasks. The environment places articulated objects and creates inter-object functional relationships between the objects and responder objets.\n* The second is a self-supervised learning framework called BusyBot that proposes to solve goal conditioned tasks in the above environment. The framework proposes a interaction policy using supervised learning, a reason module which collects interactions with the environment to create a scene graph using GNNs and a future dynamics model, and a planning agent which uses the above modules to perform goal conditioned tasks in the environment.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The environment is interesting because it allows exercising integrated skills for embodied agents. Interaction, reasoning, and planning fused into one environment.\n* The simplicity of the observations and action spaces (topdown depth and SE(3) pose) could be seen as a plus, to focus on the reasoning and planning parts of the problem.\n* The ability of the environment to generate new task configurations to test novel placements and novel objects.\n* The proposed BusyBot framework has a high success rate in sim and real and beats a few baselines proposed by the authors.\n* The solution for each reasoning/planning/interaction components seems well thought out.\n* Real world experiments.\n* The paper overall is well written and clear.\n\n\nWeaknesses\n* The environment does not contain many-to-one relationships, which would be a very interesting setup to test for embodied agents, requiring agents with memory and potentially longer sequences and interactions.\n* The observation is a top-down depth image. This modality may not be available in all cases for real world robotic systems. There are no ablations on this modality. It would be nice to see how the system compares with rgb observations for example, or a perspective view of the board.\n* No vanilla RL or BC baselines. It would be interesting to see the performance of even a simple BC sequence model, trained on a sim oracle or human dataset. It could help answer the question: how necessary are the separation of interaction/reasoning/planning models?\n* BusyBot is not evaluated on environments other than BusyBoard, so it’s unclear how generally useful this system could be. Is it “overfit” to this particular setup? A few details in particular that could be a concern for it’s usefulness:\n* As mentioned above, many-to-one interactions are not included and could be common in real world tasks. Unclear if the method would work for this case.\n* A few parts of the system, such as the k-means clustering hyperparam, could require tuning in different environment setups.\n* Baselines are not tested on the real world.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the paper is borderline and I would consider changing my recommendation with a few clarifications and/or results. The environment and proposed method are solid and the experimental results support the authors conclusions overall. The environment is a good contribution, as a single environment that evaluates several skills in embodied agents.\nThe BusyBot results could be made more convincing. I think testing the proposed method on more complex many-to-one tasks or with more realistic modalities, or against a more diverse set of baselines would be more convincing.\nI recommend addressing some of the weaknesses above to improve the paper.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* Clarify the issue of many-to-one tasks - were these tasks attempted or could the environment be modified to generate these sorts of tasks? Would the proposed method address these tasks?\n* Could the authors clarify the choice of observation space. Why was only topdown depth chosen?\n* It's hypothesized that the relation agent might have trouble handling 1-to-m relations (L196). Why is the 1-to-m result higher for novel objects for the relation agent than the predictive agent?\n* Could the authors clarify how BusyBot could be useful outside of the 1-to-1 or 1-to-m scenarios presented in the env?\n* Addressing or explaining limitations mentioned above in limitations section."}}, {"review_id": "75WuIw8v_Vh", "reviewer": "Reviewer_55uE", "summary": "This work proposes a framework for generating simple learning environments both in simulation and in the real world, dubbed BusyBoard. BusyBoard consists of sets of switches and objects, some of which are articulated, that can be interacted with and uses lights to amplify the effects of many of the small actions (e.g. pushing switches) making them more observable. BusyBoard instances are generated procedurally from URDF files making the environments amenable to many variations on the BusyBoard theme.\n\nThe authors also propose a baseline for the BusyBoard environment dubbed BusyBot. BusyBot learns action affordances from depth, learns inter-object interactions, and then creates and executes plans in the BusyBoard enviornment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The BusyBoard framework complements typical toy domains, such as gridworlds and block stacking, by providing a relatively rich set of inter-object interactions in a small and feasibly explorable space with relatively dense reward.\n- BusyBoard's procedural nature lends itself nicely to explorations of transfer learning and distribution shift.\n- BusyBot appears to be a strong baseline for this domain (with one of the biggest assumptions being the sufficiency of single-step actions to realize meaningful change in the world).\n- The paper contains an interesting hardware implementation of the both the BusyBoard domain and BusyBot.\n\nWeaknesses\n- By design the BusyBoard domain does not attempt to expose challenges such as partial-observability and sparse long-horizon rewards. While the authors are explicit about this design choice, these are two extremally fundamental difficulties in robotics.\n- The BusyBot baseline is not well explored in the evaluation. While the design appears reasonable, virtually no comparison was made with existing learning methods that may be applicable to this task. As a result, this paper's primary contribution centers on BusyBoard; it's hard for a reader to judge the effectiveness of BusyBot.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I advocate for acceptance of this work. I think the BusyBoard domain, while simple, has some value to the community as a platform for exploring learning in constrained settings. That being said, BusyBoard, by design, hides much of the complexity present in less structured environments. As a result, progress in the BusyBoard domain may not be a particularly good proxy for progress in real-world tasks.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Consider adding comparisons to existing agent-learning approaches for BusyBot. As is, it's difficult to evaluate the value of BusyBot and most of the paper's contribution centers around the BusyBoard domain."}}, {"review_id": "72NIGmus1Vq", "reviewer": "Reviewer_gcRk", "summary": "The paper describes, on the one hand, a scenario for robot learning, planning, and acting inspired in a toy for cognitive development for children, and, on the other hand, a learning and planning framework tailored to this specific scenario to let a robot learn to execute a given task. The authors claim that one of the main contributions resides in the characteristics of the environment that allows a robot to learn cause-effect relations through interaction and to use these relations to select a sequence of action to bring the scenario from an initial configuration to a target (goal) one. They also propose a set of algorithms for learning and representing these cause-effects in a robotic application.\n\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The tasks used for validation of learning and planning approaches are too simple and can be solved by traditional approaches. The integration of sense-plan-act mechanisms with learning through interaction is not novel.  The perception mechanisms using vision and deep learning may entail some novelty but this is not developed in the paper. A clear presentation of the theoretical background and (theoretical) improvements achieved with respect to the state of the art is still missing. \n\nThere are no interdependencies between consecutive actions in a plan. The search problem is simple and straightforward. \n\nOne potentially interesting contribution of this work might be to let the robot learn the primitives to manipulate objects that requires small displacement. Unfortunately, these methods are not properly described and contrasted with respect to the state of the art. The experimental evaluation does not allow to fully assess the validity of the method. The comparisons are carried out only with ablated versions of their system, but not against the state of the art.\n\nThe contributions are difficult to distil from the paper. The authors condense a large number of algorithms without providing enough details and justifications. The paper in its current state gives the impression that the proposed methods are rather ad-hoc, with arguable transferability to other problems and scenarios.\n\nThe supplementary video is well-edited and clear.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I suggest the authors to avoid presenting the BusyBoard scenario as a contribution and to elaborate on methods that potentially entail scientific contributions with respect to the state of the art.\n\nThe description of Sec. 4.1 is incomplete and confusing. It is not clear for me how the action policy is defined/learned. I encourage the authors to better describe this method: What actions are considered in the action space? What is the state/object space? How the interaction policy is initialized? The authors mention that the robot-object interaction points are learned from depth images. How this takes place? Is there any previous knowledge used to guide the learning? How the per-pixel affordance is learned/defined? I think these could entail appealing contributions but unfortunately it is not possible to assess their validity with the current state of the paper. \n\nThe word \"prediction\" already entails future events. I suggest to avoid using the phrase \"prediction of the future\". I encourage the authors to explore the large literature in task and motion planning and learning of prediction models and use the same terminology.\n\nThe description of the limitation is insufficient. I recommend the authors to better discuss scalability, transferability, and robustness.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Reduce the description of the scenario (e.g. Sec. 3) and use the gained space to elaborate on the scientific contributions.\n\nThe paper is missing a section where the terminology, problem definition, and basic tools are properly introduced. It will be helpful if the authors provide concrete examples of complete interact-learn-plan processes.\n\nThe reward is defined from differences in images. This will generate problems in real scenarios with changing light conditions and non-causal events taking place from external contingencies. How the authors are planning to address this problem? Would the proposed algorithms fail in these cases?\n\nThe explanation of the exploration strategy is unclear and insufficient.\n\nLine 200. It is not clear is BusyBot is just the planning approach or the framework comprising all the learning algorithms (interaction policy, cause-effect relations, etc.).\n\nJustify how the values of the parameters in the experimental set-up are defined (23 images, training test 32/9, etc. Why these arbitrary values?)\n\nLines 243 and 245. Typos.\n\nSome references are incomplete (volume, issues, pages,…)."}}, {"review_id": "3UEhpXZ7H-f", "reviewer": "Reviewer_yhwf", "summary": "Self-contained busyboard environment with articulated objects and functional relations.\nSelf-supervised strategy: 1- learn policy to interact with environment, 2- collect data with policy and learn inter-object relations with causal discovery model 3- combine policy and causal model to learn goal-conditioned manipulation model.\nmotivation: busyboards amplify sensory feedback and have causal relations and so are ideal self-learning env for toddlers and robots.\n8 sub-models: position network, direction network, 3 GNNs, 1 GRN, 2 agents, trained with thresholded image difference reward, MSE, and L2. Trained on 10k boards, eval’d on novel boards (2k) and objects (2k).\nResults: interaction module: on par with “oracle”, benefits from responders vision and exploration. Relation reasoning: high accuracy even for novel configs and objects. goal-conditioned manipulation: good for 1-1, less for 1-many. real: high scores for 3 triggers and 3 lights.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n- A self-contained diverse environment for learning interactions, reasoning and planning is good for robotics research.\n- publicly available code and simulation is nice (have not seen it available yet).\n- The proposed solution seems to yield good results.\n\nweaknesses:\n- thresholded image difference as reward function: how would this be deployed in the real world?\n- The learning method seems overly engineered and tuned for this specific problem, it seems unlikely that it would be useful for broad learning in the real world as is, a more unified model (as opposed to the 8 modules used here) or more end-to-end design. I understand that some engineering is ok for the sake of solving this problem, but it doesn’t seem that the proposed method is advancing research beyond this narrow setup.\n- The real setup is pretty narrow, I wonder how much tuning went into the real setup given its small size and how easy it is to obtain good results with tuning.\n- There is no mention or precise articulation / plan of how the techniques used here are transferable to the real world besides the analogy to how human infants learn from busyboards.\n- Although busyboard is a more relevant env to the real world than Atari games, it is not necessarily that much more convincing in terms of relevance to general real world robotics. Maybe authors can improve that front.\n- Future work discussion could benefit from a vision on how to expand the learnings of this work beyond the narrow setup of the busy board.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The science seems sound. The motivation of self-supervision with simplified toy-like setups with multiple abilities being tested (interaction, reasoning, planning) is good. While a new public benchmark is good as well, the issue I see with the paper is the proposed learning solution not being general enough to be of value beyond this narrow setup.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- minor: Lacking references for GNN papers?\n- more articulate vision on how the proposed approach is applicable beyond this setup.\n- code and sim available?\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)